search

tim-learn / SHOT

code released for our ICML 2020 paper "Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation"
MIT License
437 stars 78 forks source link

Can't reproduce the results on Office-Home for PDA #5

Closed Zysty closed 4 years ago

Zysty commented 4 years ago

Hi, thanks for your sharing! I run your code on Office-Home dataset for PDA when source domain is the Art. (i.e., A->C, A->P, A->R). But they can not reproduce the results only except A->C. I got the following results on the three different seeds {2019, 2020, 2021}: Source Only: image image image

SHOT-IM seed=2020: image image

Particularly, the performance on A->R gets worse as training time increases. image

These results should be image

I don't konw why it happens. Could you please tell me how to correctly reproduce results on PDA? Thanks.

Zysty commented 4 years ago

I can reproduce the results on office-home for PDA now!
These experiments above were conducted on torch=1.1.0, python=3.6.2, torchvision=0.2.2. Then I conduct corresponding experiments on the other server, on which torch=1.4.0,python=3.7.4, torchvision=0.5.0. And it works!

Thank for your great work. But I still don't know why it happens...

tim-learn commented 4 years ago

I can reproduce the results on office-home for PDA now! These experiments above were conducted on torch=1.1.0, python=3.6.2, torchvision=0.2.2. Then I conduct corresponding experiments on the other server, on which torch=1.4.0,python=3.7.4, torchvision=0.5.0. And it works!

Thank for your great work. But I still don't know why it happens...

Err, maybe the version of torchvision 0.2.2 is too old. It is really hard to explain why this happens....

ygjwd12345 commented 3 years ago

I can reproduce the results on office-home for PDA now! These experiments above were conducted on torch=1.1.0, python=3.6.2, torchvision=0.2.2. Then I conduct corresponding experiments on the other server, on which torch=1.4.0,python=3.7.4, torchvision=0.5.0. And it works!

Thank for your great work. But I still don't know why it happens...

but in the camera-ready the number increase to 57.1 78.1 81.5 68.0 78.2 78.1 67.4 54.9 82.2 73.3 58.8 84.3 71.8. Do you reproduce this result?

tim-learn commented 3 years ago

I can reproduce the results on office-home for PDA now! These experiments above were conducted on torch=1.1.0, python=3.6.2, torchvision=0.2.2. Then I conduct corresponding experiments on the other server, on which torch=1.4.0,python=3.7.4, torchvision=0.5.0. And it works! Thank for your great work. But I still don't know why it happens...

but in the camera-ready the number increase to 57.1 78.1 81.5 68.0 78.2 78.1 67.4 54.9 82.2 73.3 58.8 84.3 71.8. Do you reproduce this result?

Hi, the results you mentioned is SHOT for closed-set UDA.

roysubhankar commented 3 years ago

I can reproduce the results on office-home for PDA now! These experiments above were conducted on torch=1.1.0, python=3.6.2, torchvision=0.2.2. Then I conduct corresponding experiments on the other server, on which torch=1.4.0,python=3.7.4, torchvision=0.5.0. And it works!

Thank for your great work. But I still don't know why it happens...

Hi @Zysty can you please share with us a list of all the packages and their versions (pip list or conda list) which you used for running the experiments? I am still struggling to reproduce the numbers as reported in the paper. Thank you in advance.