Do not use number of iterations of attack required as a measure of robustness

[anishathalye]

We don't say that number of iterations of an attack required is not a measure of robustness. This is implied by Section 4.8, but not stated explicitly in the text or included as a checklist item.

I think the obfuscated gradients paper put it nicely: "It is not meaningful to restrict the computational power of an adversary artificially (e.g., to fewer than several thousand attack iterations). If two defenses are equally robust but generating adversarial examples on one takes one second and another takes ten seconds, the robustness has not increased." (but it seems that his has not become an accepted fact, given recent papers/preprints)

We already have Section 4.8, which talks about running attacks to convergence. Is that strong enough, or should we explicitly add text and/or a checklist item saying that number of iterations of PGD required to break a model is not a measure of robustness?

[ftramer]

Another recommendation along these lines that I've found useful is the following: To make sure an attack has converged, look at the average loss rather than (or in addition to) the success rate.

The paper currently recommends:

• "Verify that doubling the number of iterations does not increase attack success rate."
• "Plot attack effectiveness versus the number of iterations."

I've had this happen recently with a model, where going from 10 to 100 iterations of PGD resulted in no change in attack success rate. However, the loss was increasing steadily. By moving to 1000 iterations, the accuracy then dropped significantly. Just looking at the accuracy would have hidden the fact that the attack was actually still making progress, and needed just a bit more work to push most of the examples over the decision boundary.

[wielandbrendel]

@ftramer I think this is a very important point that we should clarify in the manuscript. @anishathalye That's also a good point, maybe we should add an explicit sentence in section 4.8?