Closed GA-17a closed 3 years ago
How many times did you repeat on PACS?
The provided script runs the code 5 times
In the paper, I think I have run more than 5 times to have a smaller std
Would you mind sharing the standard deviation of your PACS performances?
Please see the paper
How did you select the model to report your results?
simply use the last-step checkpoint
another way is to use the median performance among the last, e.g., 10 steps, or use their average
Thanks for your reply. For the last question, I think the common way is to use the checkpoint which has the highest accuracy on validation dataset. And it usually has a relatively high variance. What do you think about it?
due to domain shift, a higher performance on the same-domain val set doesn't necessarily give a better result on unseen domains (e.g. overfitting), but it's not uncommon to follow the standard cross-val way for selecting checkpoints
as I mentioned, you could also simply use the last, converged checkpoint which might lead to a smaller variance
I'd assume there is no further issue regarding the code so I'm closing it now
Hi @KaiyangZhou , Thanks for sharing the code. I have the following questions:
Thanks!