search

VectorInstitute / gram-ood-detection

46 stars 6 forks source link

Sensitivity to image preprocessing #1

Open zjysteven opened 3 years ago

zjysteven commented 3 years ago

Hi,

This is not a bug but rather just an observation I got. It seems that Gram Matrices-based method can be pretty sensitive to image preprocessing. I trained a WRN40-2 on CIFAR-10 and directly used the provided notebook to evaluate detection performance. When using LSUN_resized released by ODIN's authors, the AUROC is 99.8 (MSP gives 91.7). However, if one uses the original LSUN data and applies transformation during the loading procedure like below, the AUROC drops to 79.1 (MSP gives 88.1). This is likely due to the artifacts that exist in the LSUN_resized data, which has been pointed out in the paper "CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances" (Appendix I Figure 5, 6). 


transform_test = transforms.Compose([
    transforms.Resize(32),
    transforms.CenterCrop(32),
    transforms.ToTensor(),
    normalize
])
chandramouli-sastry commented 3 years ago

Hi,

That's interesting! The drastic drop is probably because the algo uses "hard" max and min for detecting deviations -- but, might need a closer examination to pin-point what's happening.

If you can share the weights of the model and your code (on google colab or google drive, for example), it'll be helpful for others (and myself) to easily reproduce your observation and inspect what's happening.

Thanks!

zjysteven commented 3 years ago

I just uploaded the model weights and the code here. And the assumed structure in the data folder is as follows, where the original LSUN data can be downloaded according to this repo. Let me know if you find any problems with the code.

data
├── LSUN
│   ├── test
├── LSUN_resize
│   ├── test

BTW, I also observed the performance drop in Mahalanobis.

chandramouli-sastry commented 3 years ago

Thank you for sharing the code and datasets!

Curious to understand what's going on. Interesting to know that Mahalanobis performance also drops.