fra31
commented
4 years ago Hi,
I'm glad you find our work useful!
There is no preclusion to models from papers aiming at different aspects of adversarial robustness, like faster training or compressed models (e.g. we report the results from (Zhang et al., 2019a), (Wong et al., 2020), (Shafahi et al., 2019)). I think that it makes sense to evaluate also those models with strong attacks, but I agree that it should be clear which is the goal of the proposed method/defense.
To better track this information, I created here a template to get new models added to the list, so that it should be easier for everyone interested to find out more about the evaluated model (I'd add a link to such form in the list). I invite you to submit a model from https://arxiv.org/pdf/2002.10509.pdf through this procedure! In particular, it'd be very useful if you could include a script like this which runs the evaluation.
The other option would be, since you have code and pretrained models available, that I evaluate them myself, and then add a new entry to the list, but in this way it'd be more difficult to add comments about the nature of your models.
Let me know if this sounds reasonable, works or you have other suggestions!
VSehwag
siawpeng
Thanks for releasing such a rigorous evaluation of existing works on adversarial defenses. It is immensely helpful to get more clarity on this topic. I wonder what is the criterion to add new models to the existing list of defenses?
In particular, I am wondering whether papers (such as https://arxiv.org/pdf/2002.10509.pdf), which study adversarial training (in particular the SOTA approach from Carmon et al., 2019) in a new setting qualifies for it? In particular, the aforementioned paper revisits the question "whether high capacity is necessary for adversarial robustness" from Madry et al., 2018 and shows that high robustness can be achieved even after removing up to 99% of the parameters.
In general, it could be nice addition in the repo to have an evaluation of works that directly do not aim to improve robustness, but try to preserve it in the presence of other challenges (such as label noise, pruning etc).
Thanks.