Open kobybibas opened 3 years ago
Unfortunately, we're not entirely certain why we can't seem to reproduce the original paper's results exactly as published. If you are able to discover the reason, please feel free to make a pull request.
On Thu, Dec 24, 2020, 4:43 AM Koby Bibas notifications@github.com wrote:
[This email originated from outside of OSU. Use caution with links and attachments.]
Hi, Thanks for this very helpful code.
I run it with the default arguments using the model from ./code/good_models/desnet300_cosine.pth
The best AUC that is printed is AUROC: 49.04454277777778 TNR@TPR95: 0.028000000000000025 supposedly best auc: 50.0 and tnr@tpr95 0.0 true best auc: 52.790187222222215
where the paper AUC is Imagenet(c) 98.7
Is there a way to reproduce the paper results?
Thanks again, Koby
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/guyera/Generalized-ODIN-Implementation/issues/3, or unsubscribe https://github.com/notifications/unsubscribe-auth/AINDF532I54OHSLBR3VEEMTSWMZOZANCNFSM4VIFIO5A .
I apologize, my inbox did not provide me with the entirety of your comment. It seems like your model did not converge. While we are unable to perfectly reproduce the published results, we are able to get very close. An AUROC of 0.5 is equivalent to a model which is randomly guessing; perhaps your learning rate was too large?
Hi, Thanks again for your time. I the following model from the repository: https://github.com/guyera/Generalized-ODIN-Implementation/blob/master/code/good_models/desnet300_cosine.pth
Is it ok? Is it pretrained?
Yes, but I'm not exactly certain what the experimental setup was for that pretrained model, and the code may have been changed since it was trained. Perhaps it was trained on a different dataset from the defaults, even. I'll remove the pretrained models from the repository; once the code is finalized a bit more, we'll pretrain some new models. In the meantime, it'd be best to train your own models.
The pretrained models have been removed. When you get a chance, please try training a model from scratch. If there are still issues (e.g. 0.5 AUROC), let us know. The issue will remain open in the meantime.
Hi, After training got the following: AUROC: 27.57753055555555 TNR@TPR95: 0.01319999999999999 supposedly best auc: 95.79497777777777 and tnr@tpr95 0.7751 true best auc: 98.12951444444444
Mush better than 0.5 :-)
Hi, Thanks for this very helpful code.
I run it with the default arguments using the model from ./code/good_models/desnet300_cosine.pth
The best AUC that is printed is AUROC: 49.04454277777778 TNR@TPR95: 0.028000000000000025 supposedly best auc: 50.0 and tnr@tpr95 0.0 true best auc: 52.790187222222215
where the paper AUC is Imagenet(c) 98.7
Is there a way to reproduce the paper results?
Thanks again, Koby