search

DeLightCMU / RSC

This is the official implementation of Self-Challenging Improves Cross-Domain Generalization, ECCV2020
BSD 2-Clause "Simplified" License
160 stars 18 forks source link

Experimental results in paper #14

Open VuongLong opened 3 years ago

VuongLong commented 3 years ago

Dear authors, thank you for work, I run your implementation and got lower results than ones reported in the paper. I have some questions:

  1. Your results reported in the paper (average ~85.15) is selected by test-set?

  2. I got average results from your implementation:

    • selected by validation-set ~ 83.37
    • selected by test-set ~ 84.25

  3. when I turn off your seed configs and try 5 runs with random seed?

    • selected by validation-set ~ 82.63
    • selected by test-set ~ 83.81

Thank you for your contribution!

SirRob1997 commented 3 years ago

A common problem here, see #12 #13 #2 #5

Justinhzy commented 3 years ago

Hi, Thanks for your reported issue.

Indeed, closing the seed config will change the results. In experiments, however, this can change results so much that it is hard to select the parameters. So I fix the random seed. I am not sure if fixing random seed is common or not in DG task, but it is common in other tasks. If you want to close the seed configs, please see my updated notice which may be useful for you.

Yes, I reported the results selected by val-set. You can see my training logs in the testing section.

gshuangchun commented 3 years ago

Dear authors, I have the same problem. I have try 3 runs and got the average result 82.50. Besides, when I test the model you released, I got the same results you reported in art, sketch, photo. But my cartoon result is 76.75, and your report is 81.61. The environment and seed are the same of yours.@Justinhzy

Justinhzy commented 3 years ago

@gscahhh Hi, do you use the exact same environment? if you use a different type of GPU or seed, the performance will definitely change and you need to tune the hyper-parameter a little bit. Please see more details in the updated notice. I will check the cartoon resnet model.