search

M-Nauta / ProtoTree

ProtoTrees: Neural Prototype Trees for Interpretable Fine-grained Image Recognition, published at CVPR2021
MIT License
87 stars 17 forks source link

Acc lower than paper claims #4

Closed huygens12 closed 2 years ago

huygens12 commented 2 years ago

Hi @M-Nauta , thank you for your very interesting paper and this very well-written repo. However, when I follow README to reproduce the results on CUB_200_2011, I got 72% and 78% accuracy in two runs. Did I miss anything? How should I fix this?

M-Nauta commented 2 years ago

Thanks for your interest in our work and I'm sorry to hear you get a lower accuracy. It is difficult to pinpoint the main reason but it could be that the training hyperparameters, data preprocessing or other settings were not the same as used in the paper. Some default arguments, such as the milestones for lr decay, need to be changed in order to follow the setup of the paper. All hyperparameters are reported in the appendix of the paper. Maybe also check the PyTorch version. Since CUB is a small dataset it could also be that you had a very bad seed.

huygens12 commented 2 years ago

The training hyperparameters are set strictly following the paper. I reran the experiments several times. So far, my acc distribution across these runs is [72.4, 78.6, 82.4, 82.4, 80.5]. Seed issue could be the reason. Did this happen to you? And will a larger batch size help?

huygens12 commented 2 years ago

Follow-up: I slightly adjusted the hyperparameters to "epochs=150 milestones=60,80,100,120,140" and conducted 5 runs, and the test accs are now [0.828, 0.825, 0.820, 0.831, 0.815]. Hopefully this will help other readers to reproduce the results. Thank @M-Nauta again for this interesting work.

The training hyperparameters are set strictly following the paper. I reran the experiments several times. So far, my acc distribution across these runs is [72.4, 78.6, 82.4, 82.4, 80.5]. Seed issue could be the reason. Did this happen to you? And will a larger batch size help?

M-Nauta commented 2 years ago

I was already looking into potential causes of the issue, but I am glad to hear that your results are now in line with the paper and that you shared your solution here.