ResNet50 get 0% (not ~2% as reported in the paper)

[giangnguyen2412]

Hey,

Thanks for your such great work. I am trying to run the eval.py to and use ResNet50 pretrained to get your result in Figure 2. However, I am getting 0% accuracy for ResNet50.

Can you give me your advice? Giang

[giangnguyen2412]

Other models' results in Table 1 ImageNet-A are reproducible but not ResNet50. @hendrycks Can you check it?

[hendrycks]

torchvision's ResNet-50 gets 0%. A different ResNet-50 trained from scratch gets ~2%.

[giangnguyen2412]

@hendrycks Hello. Thanks for your kind reply. Could you point me to the ResNet-50 trained from scratches? In your code, you still use torchvision's ResNet50.

[hendrycks]

We did not upload a ResNet-50 trained from scratch. https://github.com/clovaai/CutMix-PyTorch has a ResNet-50 (though I think it was trained for 300 epochs).

[giangnguyen2412]

Thank you @hendrycks

[expectopatronum]

Did anyone try if it works with the pretrained ResNet-50 from https://github.com/clovaai/CutMix-PyTorch?

[expectopatronum]

torchvision's ResNet-50 gets 0%. A different ResNet-50 trained from scratch gets ~2%.

Can this be explained by the fact that ResNet-50 was used for selecting images?

Thereafter we delete the images that fixed ResNet-50 [24] classifiers correctly predict.

If it is the exact same model and all correctly classified examples have been removed, it is no surprise that it has 0% accuracy.