search

RobustBench / robustbench

RobustBench: a standardized adversarial robustness benchmark [NeurIPS 2021 Benchmarks and Datasets Track]
https://robustbench.github.io
Other
672 stars 99 forks source link

Why does ImageNet-c uses only 5k examples rather than 50k images? #73

Closed Eric-mingjie closed 2 years ago

Eric-mingjie commented 2 years ago

I see that the current load_imagenetc function only loads 5k test images, why not load the whole 50k images? https://github.com/RobustBench/robustbench/blob/master/robustbench/data/imagenet_test_image_ids.txt

max-andr commented 2 years ago

this subset of 5k images was mainly introduced to make Linf/L2 robustness evaluation fast. and then, for the sake of consistency, we decided to use the same subset of images for ImageNet-C as well. we thought that 5k images (and times 5 severity levels times 15 corruption types) should be sufficient to get good estimates of accuracy.

best, maksym

Eric-mingjie commented 2 years ago

Thanks for the quick response. So to use all 50k images, can we just create a file imagenet_test_image_ids to include all 50k images instead of 5k now?

max-andr commented 2 years ago

yes, totally! that should work.