search

RobustBench / robustbench

RobustBench: a standardized adversarial robustness benchmark [NeurIPS 2021 Benchmarks and Datasets Track]
https://robustbench.github.io
Other
669 stars 99 forks source link

Cannot load XCiT CIFAR-100 model (Debenedetti et. al, 2022) #108

Closed ngoctnq closed 2 years ago

ngoctnq commented 2 years ago

Using RobustBench's load_model gives a model that predicts with a random-like accuracy. Meanwhile, manually loading the checkpoint works. MWE: https://colab.research.google.com/drive/1l4RhImKkAvEOTzIhrZjeGA0xdh1B4M4u?usp=sharing

dedeswim commented 2 years ago

Hi, this is indeed weird! Thanks a lot for opening the issue and for providing an MWE. I'll look into it ASAP

dedeswim commented 2 years ago

I found and fixed the issue, thanks again for finding it! Just a small note about your MWE: here you shouldn't be calling model.model(x), but just model(x). If you call the former, you won't apply normalization to the inputs, which will lead to inaccurate results

ngoctnq commented 2 years ago

Sorry for the mistake, hope it didn’t lead you astray while bugfinding! I probably messed up the code cleaning when creating the MWE somehow.

On Nov 9, 2022, at 23:55, Edoardo Debenedetti @.***> wrote:

 I found and fixed the issue, thanks again for finding it! Just a small note about your MWE: here you shouldn't be calling model.model(x), but just model(x). If you call the former, you won't apply normalization to the inputs, which will lead to inaccurate results

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.

dedeswim commented 2 years ago

No worries, I noticed it right away! I hope this bug didn't cause big issues for your project 😊