search

MadryLab / robustness

A library for experimenting with, training and evaluating neural networks, with a focus on adversarial robustness.
MIT License
916 stars 181 forks source link

Robust Accuracy on Cifar10 #34

Closed SolidShen closed 4 years ago

SolidShen commented 4 years ago

Hi, I am trying to reproduce the experiment results of one paper from your lab 'Adversarial examples are not bugs, they are features' with this repo. In the paper (Table 7), the author mentioned the robust accuracy on cifar10 for the standard training model is 4.49% (l2_bound = 0.25 (Is it the bound on (0,1) pixel space?)). But I can't achieve this accuracy with the same setting(7 steps with a step size of eps/5). (I can only get 11.3%). Do you have any suggestions to help me reproduce the result? Thanks!

dtsip commented 4 years ago

In general, the robustness of standard models can vary to some extent. Depending on the exact architecture and hyperparameters used, you might encounter slightly different results. Using the PyTorch pre-trained ResNet50 with a 100-step PGD described in the README, we get an accuracy of 7.34%.

(Yes, the bound is in [0, 1] pixel space.)