search

facebookresearch / mae

PyTorch implementation of MAE https//arxiv.org/abs/2111.06377
Other
7.2k stars 1.2k forks source link

ImageNet-C numbers #104

Closed shoaibahmed closed 2 years ago

shoaibahmed commented 2 years ago

Hi.

Thank you for sharing a very clean and extensible repository. I am confused regarding the metric that you are reporting for ImageNet-C. Is it just the average of the per-corruption average error rate? Or is it the relative mCE (relative w.r.t. AlexNet as reported in the original paper)?

I have tried the average of the per-corruption average error rate and it is vastly different (mean error rate of 27.91% for ViT-H and 37.98% for ViT-B). Although these results are better in terms of robustness, I am assuming that you reported something else which I would like to reproduce. So just want to clarify that.

Thank you once again for putting in a lot of effort for this!

endernewton commented 2 years ago

Please use the AlexNet normalized values, the results in the paper are also verified by another group using the repo.