search

guyera / Generalized-ODIN-Implementation

24 stars 3 forks source link

Comparing to paper results #3

Open kobybibas opened 3 years ago

kobybibas commented 3 years ago

Hi, Thanks for this very helpful code.

I run it with the default arguments using the model from ./code/good_models/desnet300_cosine.pth

The best AUC that is printed is AUROC: 49.04454277777778 TNR@TPR95: 0.028000000000000025 supposedly best auc: 50.0 and tnr@tpr95 0.0 true best auc: 52.790187222222215

where the paper AUC is Imagenet(c) 98.7

Is there a way to reproduce the paper results?

Thanks again, Koby

guyera commented 3 years ago

Unfortunately, we're not entirely certain why we can't seem to reproduce the original paper's results exactly as published. If you are able to discover the reason, please feel free to make a pull request.

On Thu, Dec 24, 2020, 4:43 AM Koby Bibas notifications@github.com wrote:

[This email originated from outside of OSU. Use caution with links and attachments.]

Hi, Thanks for this very helpful code.

I run it with the default arguments using the model from ./code/good_models/desnet300_cosine.pth

The best AUC that is printed is AUROC: 49.04454277777778 TNR@TPR95: 0.028000000000000025 supposedly best auc: 50.0 and tnr@tpr95 0.0 true best auc: 52.790187222222215

where the paper AUC is Imagenet(c) 98.7

Is there a way to reproduce the paper results?

Thanks again, Koby

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/guyera/Generalized-ODIN-Implementation/issues/3, or unsubscribe https://github.com/notifications/unsubscribe-auth/AINDF532I54OHSLBR3VEEMTSWMZOZANCNFSM4VIFIO5A .

guyera commented 3 years ago

I apologize, my inbox did not provide me with the entirety of your comment. It seems like your model did not converge. While we are unable to perfectly reproduce the published results, we are able to get very close. An AUROC of 0.5 is equivalent to a model which is randomly guessing; perhaps your learning rate was too large?

kobybibas commented 3 years ago

Hi, Thanks again for your time. I the following model from the repository: https://github.com/guyera/Generalized-ODIN-Implementation/blob/master/code/good_models/desnet300_cosine.pth

Is it ok? Is it pretrained?

alexanderguyer commented 3 years ago

Yes, but I'm not exactly certain what the experimental setup was for that pretrained model, and the code may have been changed since it was trained. Perhaps it was trained on a different dataset from the defaults, even. I'll remove the pretrained models from the repository; once the code is finalized a bit more, we'll pretrain some new models. In the meantime, it'd be best to train your own models.

guyera commented 3 years ago

The pretrained models have been removed. When you get a chance, please try training a model from scratch. If there are still issues (e.g. 0.5 AUROC), let us know. The issue will remain open in the meantime.

kobybibas commented 3 years ago

Hi, After training got the following: AUROC: 27.57753055555555 TNR@TPR95: 0.01319999999999999 supposedly best auc: 95.79497777777777 and tnr@tpr95 0.7751 true best auc: 98.12951444444444

Mush better than 0.5 :-)