search

cmhungsteve / TA3N

[ICCV 2019 (Oral)] Temporal Attentive Alignment for Large-Scale Video Domain Adaptation (PyTorch)
https://arxiv.org/abs/1907.12743
MIT License
259 stars 41 forks source link

Reproducing the results in the paper #3

Closed jinwchoi closed 3 years ago

jinwchoi commented 5 years ago

Hi @cmhungsteve,

Thank you for releasing the wonderful code. I am trying to reproduce the UCF-HMDB_{full} results from the ICCV19 paper. i.e. source only, target only, TA^2N, and TA^3N of Table 3 and Table 4 of the paper. But I cannot reproduce the TA^2N and TA^3N results. Can you share the hyperparameter settings of the TA^2N and TA^3N? If you can share the shell script file of those, it would be really helpful.

Thanks!

cmhungsteve commented 5 years ago

Hello @jinwchoi,

Thank you for your interest. You can use script_train_val.sh to reproduce the results. There could be small variations because of different machines.

jinwchoi commented 5 years ago

I am using that script, but getting 74.4% accuracy on UCF->HMDB setting. The reported accuracy in the paper is 78.33%. Can you tell me what exactly the input parameters should be?

I am posting my script_train_val.sh file here.

#!/bin/bash

#====== parameters ======#
dataset=hmdb_ucf # hmdb_ucf | hmdb_ucf_small | ucf_olympic
class_file='data/classInd_'$dataset'.txt'
training=true # true | false
testing=true # true | false
modality=RGB 
frame_type=feature # frame | feature
num_segments=5 # sample frame # of each video for training
test_segments=5
baseline_type=video
frame_aggregation=trn-m # method to integrate the frame-level features (avgpool | trn | trn-m | rnn | temconv)
add_fc=1
fc_dim=512
arch=resnet101
use_target=uSv # none | Sv | uSv
share_params=Y # Y | N

if [ "$use_target" == "none" ] 
then
    exp_DA_name=baseline
else
    exp_DA_name=DA
fi

#====== select dataset ======#
path_data_root=dataset/ # depend on users
path_exp_root=action-experiments/ # depend on users

if [ "$dataset" == "hmdb_ucf" ] || [ "$dataset" == "hmdb_ucf_small" ] ||[ "$dataset" == "ucf_olympic" ]
then
    dataset_source=ucf101 # depend on users
    dataset_target=hmdb51 # depend on users
    dataset_val=hmdb51 # depend on users
    num_source=1438 # number of training data (source) 
    num_target=840 # number of training data (target)

    path_data_source=$path_data_root$dataset_source'/'
    path_data_target=$path_data_root$dataset_target'/'
    path_data_val=$path_data_root$dataset_val'/'

    if [[ "$dataset_source" =~ "train" ]]
    then
        dataset_source=$dataset_source
    else
        dataset_source=$dataset_source'_train'
    fi

    if [[ "$dataset_target" =~ "train" ]]
    then
        dataset_target=$dataset_target
    else
        dataset_target=$dataset_target'_train'
    fi

    if [[ "$dataset_val" =~ "val" ]]
    then
        dataset_val=$dataset_val
    else
        dataset_val=$dataset_val'_val'
    fi

    train_source_list=$path_data_source'list_'$dataset_source'_'$dataset'-'$frame_type'.txt'
    train_target_list=$path_data_target'list_'$dataset_target'_'$dataset'-'$frame_type'.txt'
    val_list=$path_data_val'list_'$dataset_val'_'$dataset'-'$frame_type'.txt'

    path_exp=$path_exp_root'Testexp'
fi

pretrained=none

#====== parameters for algorithms ======#
# parameters for DA approaches
dis_DA=none # none | DAN | JAN
alpha=0 # depend on users

adv_pos_0=Y # Y | N (discriminator for relation features)
adv_DA=RevGrad # none | RevGrad
beta_0=0.75 # depend on users
beta_1=0.75 # depend on users
beta_2=0.5 # depend on users

use_attn=TransAttn # none | TransAttn | general
n_attn=1
use_attn_frame=none # none | TransAttn | general

use_bn=none # none | AdaBN | AutoDIAL
add_loss_DA=attentive_entropy # none | target_entropy | attentive_entropy
gamma=0.003 # depend on users

ens_DA=none # none | MCD
mu=0

# parameters for architectures
bS=128 # batch size
bS_2=$((bS * num_target / num_source ))
echo '('$bS', '$bS_2')'

lr=3e-2
# lr=1e-4
optimizer=SGD

if [ "$use_target" == "none" ] 
then
    dis_DA=none
    alpha=0
    adv_pos_0=N
    adv_DA=none
    beta_0=0
    beta_1=0
    beta_2=0
    use_attn=none
    use_attn_frame=none
    use_bn=none
    add_loss_DA=none
    gamma=0
    ens_DA=none
    mu=0
    j=0

    exp_path=$path_exp'-'$optimizer'-share_params_'$share_params'/'$dataset'-'$num_segments'seg_'$j'/'
else
    exp_path=$path_exp'-'$optimizer'-share_params_'$share_params'-lr_'$lr'-bS_'$bS'_'$bS_2'/'$dataset'-'$num_segments'seg-disDA_'$dis_DA'-alpha_'$alpha'-advDA_'$adv_DA'-beta_'$beta_0'_'$beta_1'_'$beta_2'-useBN_'$use_bn'-addlossDA_'$add_loss_DA'-gamma_'$gamma'-ensDA_'$ens_DA'-mu_'$mu'-useAttn_'$use_attn'-n_attn_'$n_attn'/'
fi

echo 'exp_path: '$exp_path

#====== select mode ======#
if ($training) 
then

    val_segments=$test_segments

    # parameters for optimization
    lr_decay=10
        lr_adaptive=dann # none | loss | dann
        lr_steps_1=10
        lr_steps_2=20
        epochs=30
    gd=20

    #------ main command ------#
    python main.py $class_file $modality $train_source_list $train_target_list $val_list --exp_path $exp_path \
    --arch $arch --pretrained $pretrained --baseline_type $baseline_type --frame_aggregation $frame_aggregation \
    --num_segments $num_segments --val_segments $val_segments --add_fc $add_fc --fc_dim $fc_dim --dropout_i 0.5 --dropout_v 0.5 \
    --use_target $use_target --share_params $share_params \
    --dis_DA $dis_DA --alpha $alpha --place_dis N Y N \
    --adv_DA $adv_DA --beta $beta_0 $beta_1 $beta_2 --place_adv $adv_pos_0 Y Y \
    --use_bn $use_bn --add_loss_DA $add_loss_DA --gamma $gamma \
    --ens_DA $ens_DA --mu $mu \
    --weighted_class_loss_DA N --pred_normalize N \
    --use_attn $use_attn --n_attn $n_attn --use_attn_frame $use_attn_frame \
    --gd $gd --lr $lr --lr_decay $lr_decay --lr_adaptive $lr_adaptive --lr_steps $lr_steps_1 $lr_steps_2 --epochs $epochs --optimizer $optimizer \
    --n_rnn 1 --rnn_cell LSTM --n_directions 1 --n_ts 5 \
    -b $bS $bS_2 $bS -j 4 -ef 1 -pf 50 -sf 50 --copy_list N N --save_model \

fi

if ($testing)
then
    model=model_best # checkpoint | model_best
    echo $model

    # testing on the validation set
    echo 'testing on the validation set'
    python test_models.py $class_file $modality \
    $val_list $exp_path$modality'/'$model'.pth.tar' \
    --arch $arch --test_segments $test_segments \
    --save_scores $exp_path$modality'/scores_'$dataset_target'-'$model'-'$test_segments'seg' --save_confusion $exp_path$modality'/confusion_matrix_'$dataset_target'-'$model'-'$test_segments'seg' \
    --n_rnn 1 --rnn_cell LSTM --n_directions 1 --n_ts 5 \
    --use_attn $use_attn --n_attn $n_attn --use_attn_frame $use_attn_frame --use_bn $use_bn --share_params $share_params \
    -j 4 --bS 512 --top 1 3 5 --add_fc 1 --fc_dim $fc_dim --baseline_type $baseline_type --frame_aggregation $frame_aggregation 

fi

# ----------------------------------------------------------------------------------
exit 0
cmhungsteve commented 5 years ago

hmmm...I use the same script. Do you use the same virtual environments as mine?

Also, what's your source only result on UCF->HMDB setting with your current environment? I ask this because actually the most important metric is the difference between DA results and source-only results.

jinwchoi commented 5 years ago

For the source only UCF->HMDB_{full} setting, I got 71.67% exactly the same as the reported accuracy in the paper. So I think the virtual environment should not be the problem? I am wondering maybe I have mistakes on setting the hyperparameters or input arguments to the python script.

Question: the difference between the source only baseline and TA3N should be the variable use_target, right? none is source only and uSv is TA3N and Sv is the fully supervised finetuning on the target. Is it correct?

cmhungsteve commented 5 years ago

I will run again with my machine and let you know the results. But that machine is not at my hand right now, so I may get back to you later.

While waiting for me, you can also try to use the same virtual environment like mine.

Your understanding for use_target is correct. Sv is just the option I may use in the future. I haven't really worked on it right now.

jinwchoi commented 5 years ago

Hi @cmhungsteve,

I set up a new virtual environment following your README.md. I used Python 3.6, PyTorch 0.4.1, CUDA 9.0, CUDNN 7.1.4. And I installed the requirements by pip install -r requirements.txt But I still got 74.44% accuracy from TA^3N which is far from the reported one in the paper (78.33%).

I totally don't know what causes this issue. I am attaching my environment and the shell script to run the experiment. Can you take a look and what could be the problem? That would be really helpful.

absl-py==0.7.1
appdirs==1.4.3
astor==0.8.0
atomicwrites==1.3.0
attrs==19.1.0
backcall==0.1.0
backports-abc==0.5
bleach==3.1.0
blessings==1.7
certifi==2019.3.9
chardet==3.0.4
cloudpickle==1.2.1
colorama==0.4.1
contextlib2==0.5.5
cycler==0.10.0
Cython==0.29.9
dask==1.2.2
decorator==4.4.0
defusedxml==0.6.0
entrypoints==0.3
enum34==1.1.6
funcsigs==1.0.2
future==0.17.1
gast==0.2.2
gpustat==0.5.0
grpcio==1.21.1
h5py==2.9.0
html5lib==1.0.1
idna==2.8
imageio==2.5.0
importlib-metadata==0.18
ipykernel==5.1.1
ipython==7.5.0
ipython-genutils==0.2.0
ipywidgets==7.4.2
jedi==0.13.3
Jinja2==2.10.1
joblib==0.13.2
jsonschema==3.0.1
jupyter==1.0.0
jupyter-client==5.2.4
jupyter-console==6.0.0
jupyter-core==4.4.0
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
kiwisolver==1.1.0
lxml==4.3.3
Markdown==3.1.1
MarkupSafe==1.1.1
matplotlib==3.1.0
mistune==0.8.4
more-itertools==7.0.0
nbconvert==5.5.0
nbformat==4.4.0
networkx==2.3
nose==1.3.7
notebook==5.7.8
numpy==1.16.4
nvidia-ml-py3==7.352.0
olefile==0.46
opencv-contrib-python==4.1.0.25
packaging==19.0
pandas==0.24.2
pandocfilters==1.4.2
parso==0.4.0
pexpect==4.7.0
pickleshare==0.7.5
Pillow==6.0.0
pluggy==0.12.0
prometheus-client==0.7.0
prompt-toolkit==2.0.9
protobuf==3.8.0
psutil==5.6.3
ptyprocess==0.6.0
putil==0.9.12
py==1.8.0
PyContracts==1.8.12
Pygments==2.4.2
pyparsing==2.4.0
pyrsistent==0.15.2
pytest==4.6.3
python-dateutil==2.8.0
pytz==2019.1
PyWavelets==1.0.3
PyYAML==5.1.1
pyzmq==18.0.1
qtconsole==4.5.1
requests==2.22.0
scikit-image==0.15.0
scikit-learn==0.21.2
scipy==1.3.0
Send2Trash==1.5.0
simplegeneric==0.8.1
six==1.12.0
tensorboardX==1.7
tensorflow-gpu==1.4.1
tensorflow-tensorboard==0.4.0
termcolor==1.1.0
terminado==0.8.2
testpath==0.4.2
toolz==0.9.0
torch==0.4.1
torchfile==0.1.0
torchnet==0.0.4
torchvision==0.2.1
tornado==6.0.2
tqdm==4.32.1
traitlets==4.3.2
urllib3==1.25.3
visdom==0.1.8.8
wcwidth==0.1.7
webencodings==0.5.1
websocket-client==0.56.0
Werkzeug==0.15.4
widgetsnbextension==3.4.2
youtube-dl==2019.6.8
zipp==0.5.1

script_train_val.sh

#!/bin/bash

#====== parameters ======#
dataset=hmdb_ucf # hmdb_ucf | hmdb_ucf_small | ucf_olympic
class_file='data/classInd_'$dataset'.txt'
training=true # true | false
testing=true # true | false
modality=RGB 
frame_type=feature # frame | feature
num_segments=5 # sample frame # of each video for training
test_segments=5
baseline_type=video
frame_aggregation=trn-m # method to integrate the frame-level features (avgpool | trn | trn-m | rnn | temconv)
add_fc=1
fc_dim=512
arch=resnet101
use_target=uSv # none | Sv | uSv
share_params=Y # Y | N

if [ "$use_target" == "none" ] 
then
    exp_DA_name=baseline
else
    exp_DA_name=DA
fi

#====== select dataset ======#
path_data_root=dataset/ # depend on users
path_exp_root=action-experiments/ # depend on users

if [ "$dataset" == "hmdb_ucf" ] || [ "$dataset" == "hmdb_ucf_small" ] ||[ "$dataset" == "ucf_olympic" ]
then
    dataset_source=ucf101 # depend on users
    dataset_target=hmdb51 # depend on users
    dataset_val=hmdb51 # depend on users
    num_source=1438 # number of training data (source) 
    num_target=840 # number of training data (target)

    path_data_source=$path_data_root$dataset_source'/'
    path_data_target=$path_data_root$dataset_target'/'
    path_data_val=$path_data_root$dataset_val'/'

    if [[ "$dataset_source" =~ "train" ]]
    then
        dataset_source=$dataset_source
    else
        dataset_source=$dataset_source'_train'
    fi

    if [[ "$dataset_target" =~ "train" ]]
    then
        dataset_target=$dataset_target
    else
        dataset_target=$dataset_target'_train'
    fi

    if [[ "$dataset_val" =~ "val" ]]
    then
        dataset_val=$dataset_val
    else
        dataset_val=$dataset_val'_val'
    fi

    train_source_list=$path_data_source'list_'$dataset_source'_'$dataset'-'$frame_type'.txt'
    train_target_list=$path_data_target'list_'$dataset_target'_'$dataset'-'$frame_type'.txt'
    val_list=$path_data_val'list_'$dataset_val'_'$dataset'-'$frame_type'.txt'

    path_exp=$path_exp_root'Testexp'
fi

pretrained=none

#====== parameters for algorithms ======#
# parameters for DA approaches
dis_DA=none # none | DAN | JAN
alpha=0 # depend on users

adv_pos_0=Y # Y | N (discriminator for relation features)
adv_DA=RevGrad # none | RevGrad
beta_0=0.75 # depend on users
beta_1=0.75 # depend on users
beta_2=0.5 # depend on users

use_attn=TransAttn # none | TransAttn | general
n_attn=1
use_attn_frame=none # none | TransAttn | general

use_bn=none # none | AdaBN | AutoDIAL
add_loss_DA=attentive_entropy # none | target_entropy | attentive_entropy
gamma=0.003 # depend on users

ens_DA=none # none | MCD
mu=0

# parameters for architectures
bS=128 # batch size
bS_2=$((bS * num_target / num_source ))
echo '('$bS', '$bS_2')'

lr=3e-2
optimizer=SGD

if [ "$use_target" == "none" ] 
then
    dis_DA=none
    alpha=0
    adv_pos_0=N
    adv_DA=none
    beta_0=0
    beta_1=0
    beta_2=0
    use_attn=none
    use_attn_frame=none
    use_bn=none
    add_loss_DA=none
    gamma=0
    ens_DA=none
    mu=0
    j=0

    exp_path=$path_exp'-'$optimizer'-share_params_'$share_params'/'$dataset'-'$num_segments'seg_'$j'/'
else
    exp_path=$path_exp'-'$optimizer'-share_params_'$share_params'-lr_'$lr'-bS_'$bS'_'$bS_2'/'$dataset'-'$num_segments'seg-disDA_'$dis_DA'-alpha_'$alpha'-advDA_'$adv_DA'-beta_'$beta_0'_'$beta_1'_'$beta_2'-useBN_'$use_bn'-addlossDA_'$add_loss_DA'-gamma_'$gamma'-ensDA_'$ens_DA'-mu_'$mu'-useAttn_'$use_attn'-n_attn_'$n_attn'/'
fi

echo 'exp_path: '$exp_path

#====== select mode ======#
if ($training) 
then

    val_segments=$test_segments

    # parameters for optimization
    lr_decay=10
        lr_adaptive=dann # none | loss | dann
        lr_steps_1=10
        lr_steps_2=20
        epochs=30
    gd=20

    #------ main command ------#
    python main.py $class_file $modality $train_source_list $train_target_list $val_list --exp_path $exp_path \
    --arch $arch --pretrained $pretrained --baseline_type $baseline_type --frame_aggregation $frame_aggregation \
    --num_segments $num_segments --val_segments $val_segments --add_fc $add_fc --fc_dim $fc_dim --dropout_i 0.5 --dropout_v 0.5 \
    --use_target $use_target --share_params $share_params \
    --dis_DA $dis_DA --alpha $alpha --place_dis N Y N \
    --adv_DA $adv_DA --beta $beta_0 $beta_1 $beta_2 --place_adv $adv_pos_0 Y Y \
    --use_bn $use_bn --add_loss_DA $add_loss_DA --gamma $gamma \
    --ens_DA $ens_DA --mu $mu \
    --weighted_class_loss_DA N --pred_normalize N \
    --use_attn $use_attn --n_attn $n_attn --use_attn_frame $use_attn_frame \
    --gd $gd --lr $lr --lr_decay $lr_decay --lr_adaptive $lr_adaptive --lr_steps $lr_steps_1 $lr_steps_2 --epochs $epochs --optimizer $optimizer \
    --n_rnn 1 --rnn_cell LSTM --n_directions 1 --n_ts 5 \
    -b $bS $bS_2 $bS -j 4 -ef 1 -pf 50 -sf 50 --copy_list N N --save_model \

fi

if ($testing)
then
    model=model_best # checkpoint | model_best
    echo $model

    # testing on the validation set
    echo 'testing on the validation set'
    python test_models.py $class_file $modality \
    $val_list $exp_path$modality'/'$model'.pth.tar' \
    --arch $arch --test_segments $test_segments \
    --save_scores $exp_path$modality'/scores_'$dataset_target'-'$model'-'$test_segments'seg' --save_confusion $exp_path$modality'/confusion_matrix_'$dataset_target'-'$model'-'$test_segments'seg' \
    --n_rnn 1 --rnn_cell LSTM --n_directions 1 --n_ts 5 \
    --use_attn $use_attn --n_attn $n_attn --use_attn_frame $use_attn_frame --use_bn $use_bn --share_params $share_params \
    -j 4 --bS 512 --top 1 3 5 --add_fc 1 --fc_dim $fc_dim --baseline_type $baseline_type --frame_aggregation $frame_aggregation 

fi

# ----------------------------------------------------------------------------------
exit 0
cmhungsteve commented 5 years ago

Hello @jinwchoi,

I checked on my machine and I think that the script I uploaded is not updated. I may need time to find the updated one in my machine. It may take time. I will update the script as soon as possible. Thank you for your patience. And I am really sorry for the inconvenience.

AliaksandrSiarohin commented 4 years ago

Any news on this?

cmhungsteve commented 4 years ago

Sorry for the late response. I was super busy in recent weeks that I am not able to find time working on this. I will try to update the script ASAP after the CVPR deadline. Hope you understand.

Before I update the script, you can also try to search those weights. I simply used grid-search without any fancy strategy.

Really sorry for the inconvenience.

xcpeng commented 4 years ago

Hi, I am wondering if there are any updates on this?

I also get around 74.9% performance with exactly the same hyper-parameter setting.

It will be great if the optimal hyper-parameters are available to public.

Thank you for the nice work and code!

cmhungsteve commented 4 years ago

Sorry for the late. I believe the hyper-parameters in the script were outdated. I did try to find the one that I used to generate the results but couldn't find it unfortunately.

I will definitely do the experiments again to find the hyper-parameters. But I cannot guarantee when I can finish that since I am currently swamped by some other work. Things always change much faster than I expected... I will try my best to find time to do so!

Really sorry for the inconvenience. Hope you understand.

xcpeng commented 4 years ago

No worries! Thank you for your timely response. The work is really cool! I spent some time finetuning the hyperparameters with grid search but no luck! One question is: Is it OK to report the results we got with the current hyperparameter setting in our paper?

cmhungsteve commented 4 years ago

I think it should be OK. It's fair to compare with the same hyperparameter setting. I have seen other people did that before. You just need a brief explanation in your paper.

stefanxinhong commented 4 years ago

Could the authors provide the script used for training and testing on Kinetics-Gameplay? Directly change dataset in script_train_val.sh cannot reproduce the results in Table 5 including baselines and TA3N. Reprocude results: Target Only (57.41%), Source only(15.03%), DANN(15.09%), JAN(22.16%), AdaBN(19.23%), TA3N(15.22%) Results in the Paper: Target Only (64.49%), Source only(17.22%), DANN(20.56%), JAN(18.16%), AdaBN(20.29%), TA3N(27.50%）

jlim13 commented 3 years ago

Hi all, I was able to get 77.5 with the default script by turning --weighted_class_loss and --weighted_class_loss_DA to both Y in opts.py.

This is for ucf->hmdb

wlin-at commented 3 years ago

Could the authors provide the script used for training and testing on Kinetics-Gameplay? Directly change dataset in script_train_val.sh cannot reproduce the results in Table 5 including baselines and TA3N. Reprocude results: Target Only (57.41%), Source only(15.03%), DANN(15.09%), JAN(22.16%), AdaBN(19.23%), TA3N(15.22%) Results in the Paper: Target Only (64.49%), Source only(17.22%), DANN(20.56%), JAN(18.16%), AdaBN(20.29%), TA3N(27.50%）

Hi, did you manage to reproduce the numbers on Kinetics-Gameplay on the paper? I set the batch size to 512 according to Sec 7.3.2 in the paper and kept the other hyperparamters. However could get the results of 21.09%(model_best) and 17.76%(check point) for TA3N, 18.42% (model_best) and 13.48%(check_point) for source only baseline. I would really appreciate it if you could share the experience. Regards.