search

DeLightCMU / RSC

This is the official implementation of Self-Challenging Improves Cross-Domain Generalization, ECCV2020
BSD 2-Clause "Simplified" License
160 stars 18 forks source link

Random seeds value #13

Closed hamanhbui closed 3 years ago

hamanhbui commented 3 years ago

Dear authors, thank you for your supporting code for DG, it could be a good baseline for us to expand a new idea. However, I have a question about your reported results. I am seeing your code have set "torch.manual_seed(0) torch.cuda.manual_seed(0)", this means that you fix seeds for your CUDA, but your papers said that you ran on 5 different times (I understand here is with different seeds for CUDA) then get the average. I am asking this question because if removing 2 of those lines, I can not reproduce your results, especially when tuning on validation set (only around 81.2% accuracy). Could you please explain this problem?

Looking forward to your answer, thanks.

Justinhzy commented 3 years ago

Hi, Thanks for your question. I fix the random seed and code wrt probabilities for PACS experiments for 5 runs. Do you mean changing 5 different random seeds and get the average?

hamanhbui commented 3 years ago

Yes, It is exactly what I mean.

SirRob1997 commented 3 years ago

@hamanhbui has been mentioned before, results seem to be non-reproducible see #12 #2 #5

hamanhbui commented 3 years ago

Many thanks for your suggestion @SirRob1997

Justinhzy commented 3 years ago

Hi, Sorry for the late reply. I've been a little busy lately. Please see my updated notice. Generally speaking, most DG datasets are small, you may need to turn the parameter for your environment, including random seed. Though it is not ideal, you may get a higher baseline to start with.