search

hongxin001 / logitnorm_ood

Official code for ICML 2022: Mitigating Neural Network Overconfidence with Logit Normalization
141 stars 14 forks source link

CIFAR-100 & Resnet34 Results #2

Open puningYang opened 2 years ago

puningYang commented 2 years ago

I reproduce the work and discover that it does not work well on the CIFAR-100 when I choose the Resnet-34 as the backbone, results have been shown below hyperparameter tau is 0.04, as same as CIFAR-10 setting. fpr95/auroc/aupr/indistribution_acc On SVHN:49.64/91.78/93.31/76.94 On Textures:85.53/73.73/83.18/76.94 On LSUN_C:13.78/97.65/97.78/76.94 On LSUN_R:94.25/69.71/74.86/76.94 On Places365:79.42/78.34/79.22/76.94 On iSUN:94.07/67.81/74.56/76.94 AVE:69.45/79.84/83.82/76.94

Is there anything wrong?

hongxin001 commented 1 year ago

Thanks for your interest. The poor performance is easy to understand because you use an improper hyperparameter. As shown in the paper, the lower bound of the loss is related to the \tau and class num k. Therefore, if we hope to achieve a small enough loss on the training set, the value of the \tau should be adjusted according to the class num k. Generally speaking, I recommend using 0.0085 on the CIFAR-100 dataset (but it may vary due to the arch.)

hongxin001 commented 1 year ago

The more class number in the dataset, the smaller the \tau should be.

puningYang commented 1 year ago

Thanks for your reply, I will adjust the hyperparameter, actually I have tried to set the /tau as 0.001 in CIFAR-100 and it does not work. Maybe it is too small? 0.0085 is not mentioned in your paper.

puningYang commented 1 year ago

0.0085 does not work in Resnet-34 CIFAR-100, results are even worse

hongxin001 commented 1 year ago

In the original paper, we do not conduct experiments on CIFAR100 with ResNet34. The above analysis suggests that we need to search for the best value of \tau at a finer granularity level.

wjtan99 commented 6 months ago

I coded it up LogitNorm in my own code. I can repeat the WRN results with standard CE and LogitNorm, and OOD metrics. However I have trouble with Resnet34.
@puningYang How did you train your Resnet34? On standard CE on Cifar100 ID classification, I cannot get accuracy higher than 70%. I notice you have 76%. Can you share your training hyperparameters, like batch, starting LR, scheduler?

wjtan99 commented 6 months ago

Never mind. I coded up a different Resnet34 instead of using Huggingface Resnet and got CIFAR100 classification accuracy 77%.