ryderling
commented
5 years ago It is a basic observation that when given strictly more power, the adversary should never do worse. However, in Table VII the paper reports that MNIST adversarial examples with their l_infinity norm constrained to be less than 0.2 are harder to detect than when constrained to be within 0.5. The reason this table shows this effect is that FGSM, a single-step method, is used to generate these adversarial examples. The table should be re-generated with an optimization-based attack (that actually targets the defense; not a transfer attack) to give meaningful numbers.
Again, we made the observation that there is no clear relationship between the magnitude of perturbation of AEs and the detection AUC in non-adaptive scenarios, where detection does not need to know which type of the attack belongs to (gradient-based attacks or optimization-based attacks). From the experiments with FGSM, we at least demonstrated that there is no always strictly positive correlation between them.
carlini
It is a basic observation that when given strictly more power, the adversary should never do worse. However, in Table VII the paper reports that MNIST adversarial examples with their l_infinity norm constrained to be less than 0.2 are harder to detect than when constrained to be within 0.5. The reason this table shows this effect is that FGSM, a single-step method, is used to generate these adversarial examples. The table should be re-generated with an optimization-based attack (that actually targets the defense; not a transfer attack) to give meaningful numbers.