search

rgeirhos / texture-vs-shape

Pre-trained models, data, code & materials from the paper "ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness" (ICLR 2019 Oral)
https://openreview.net/forum?id=Bygh9j09KX
Other
785 stars 101 forks source link

Different shape bias values for torchvision models #7

Closed eminorhan closed 4 years ago

eminorhan commented 5 years ago

I just wanted to mention that the alexnet and vgg16 implementations in torchvision.models (v 0.3) seem to produce different shape bias values in the cue conflict experiment than the caffe implementations reported in the paper. For alexnet, I get 25.4% shape-based decisions; for vgg16, I get 9.2% (both numbers are quite a bit smaller than the ones reported in the paper for the caffe implementations). I am able to reproduce the resnet50 numbers exactly (22.1%); I believe you already use the torchvision.models implementation for resnet50. I'm not quite sure about the source of this discrepancy between different implementations, but it is something to watch out for when trying to reproduce the results.

rgeirhos commented 5 years ago

Thanks @eminorhan for pointing this out. I can confirm that I too get different shape bias values for the torchvision implementation: AlexNet: 25.3 % VGG-16: 9.2 % ResNet-50: 22.1% (which was indeed already computed with torchvision) I have added a note in the project README (section data-analysis) with a recommendation to use the torchvision implementation per default.

rgeirhos commented 4 years ago

Added to README in commit 367a2444e6350a1ef883edd6f8100d140fdd8c76.