Reproducibility issue on Stanford40-Actions

[jnhwkim]

I am sincerely thankful for this shared code. I've really enjoyed the work.

While I try to reproduce your reported table, I faced an issue with the reproducibility for Stanford40-Actions (though I've confirmed CUB200, Dog and Indoor). It gives me around 53% compared to the reported score of 63.08%. Let me check with few details: 1) how do you split the training data with what percent (I tried 20% and 25%)? 2) Is there any change for hyper-parameters? (Adam optimizer for meta-networks has 1e-3 or 1e-4 learning rate? Or, the weight decay is 0 or 1e-4? (Appendix B) 3) Any other comment?

Thanks in advance.

Jin-Hwa

[hankook]

Thanks for your interest. The details of the Stanford40 experiment are as follows.

1. In Stanford40, there are 4000 and 5532 images in training and test splits, respectively. We used 10% of training images for validation, i.e., we used 3600 images for training.
2. For Stanford40, we used meta_lr=1e-3 and meta_wd=1e-4. Other parameters are same as other experiments, e.g., beta=0.5 and T=2.

Could you check the split ratio and hyperparameter setting as mentioned above?

[jnhwkim]

I've successfully reproduced, though I got a bit higher variance. Thanks.