Open puningYang opened 2 years ago
Thanks for your interest. The poor performance is easy to understand because you use an improper hyperparameter. As shown in the paper, the lower bound of the loss is related to the \tau and class num k. Therefore, if we hope to achieve a small enough loss on the training set, the value of the \tau should be adjusted according to the class num k. Generally speaking, I recommend using 0.0085 on the CIFAR-100 dataset (but it may vary due to the arch.)
The more class number in the dataset, the smaller the \tau should be.
Thanks for your reply, I will adjust the hyperparameter, actually I have tried to set the /tau as 0.001 in CIFAR-100 and it does not work. Maybe it is too small? 0.0085 is not mentioned in your paper.
0.0085 does not work in Resnet-34 CIFAR-100, results are even worse
In the original paper, we do not conduct experiments on CIFAR100 with ResNet34. The above analysis suggests that we need to search for the best value of \tau at a finer granularity level.
I coded it up LogitNorm in my own code. I can repeat the WRN results with standard CE and LogitNorm, and OOD metrics.
However I have trouble with Resnet34.
@puningYang How did you train your Resnet34? On standard CE on Cifar100 ID classification, I cannot get accuracy higher than 70%. I notice you have 76%. Can you share your training hyperparameters, like batch, starting LR, scheduler?
Never mind. I coded up a different Resnet34 instead of using Huggingface Resnet and got CIFAR100 classification accuracy 77%.
I reproduce the work and discover that it does not work well on the CIFAR-100 when I choose the Resnet-34 as the backbone, results have been shown below hyperparameter tau is 0.04, as same as CIFAR-10 setting. fpr95/auroc/aupr/indistribution_acc On SVHN:49.64/91.78/93.31/76.94 On Textures:85.53/73.73/83.18/76.94 On LSUN_C:13.78/97.65/97.78/76.94 On LSUN_R:94.25/69.71/74.86/76.94 On Places365:79.42/78.34/79.22/76.94 On iSUN:94.07/67.81/74.56/76.94 AVE:69.45/79.84/83.82/76.94
Is there anything wrong?