[BUG] PGDL2 very low attack success rate

[rikonaka]

✨ Short description of the bug [tl;dr]

Clean acc: 92.19%

PGDL2 attack result acc:

💬 Detailed code and results

Nothing.

[Framartin]

I will try to have a look. @rikonaka could you kindly share the full code in an executable format (full notebook or python code)?

[rikonaka]

I will try to have a look. @rikonaka could you kindly share the full code in an executable format (full notebook or python code)?

Yes, of course. 😋

https://colab.research.google.com/drive/1E9DQY8m_9-tENRF2d8bCYSMI70NnU7ND?usp=sharing

[Framartin]

I did not run the code, but the problem is likely the value of epsilon. 8/255 is a classical value for the Linf norm, but not for the L2 norm. One downside of the L2 is norm is that the value of epsilon should be adapted to the input size (number of pixels). Therefore, values should be different from one dataset to another. You can look, for example, at RobustBench for classical values of epsilon. On CIFAR-10, they use 0.5. This is also a value that I used for CIFAR-10 in a paper of mine on transferability (and I used epsilon=3 on ImageNet).

Could you try with a higher value of epsilon? And, the step-size alpha should be changed to keep it proportional to epsilon, for example alpha=epsilon/10.

[rikonaka]

I did not run the code, but the problem is likely the value of epsilon. 8/255 is a classical value for the Linf norm, but not for the L2 norm. One downside of the L2 is norm is that the value of epsilon should be adapted to the input size (number of pixels). Therefore, values should be different from one dataset to another. You can look, for example, at RobustBench for classical values of epsilon. On CIFAR-10, they use 0.5. This is also a value that I used for CIFAR-10 in a paper of mine on transferability (and I used epsilon=3 on ImageNet).

Could you try with a higher value of epsilon? And, the step-size alpha should be changed to keep it proportional to epsilon, for example alpha=epsilon/10.

Wow, it works now. 😱😱😱

Are there suggested parameters for different datasets? I mean is there a recommended perturbation parameter for different datasets and different attack methods? Or any website documentation or something like that. 😉😉😉

[Framartin]

My generic advice is to use the same experimental settings of another paper when you try to reproduce its experiments.

Here:

• it is true on every dataset that the max norm of the perturbation epsilon needs to be changed when changing the type of Lp norm
• when changing from one dataset to another, epsilon needs to be changed, if the input dimension changed (number of pixels)
• setting the step-size alpha = epsilon/10 should work on every dataset. Almost everytime, it is best to have more iterations than 10 (probably around 50 or 100). See the hyperparameters of the PGD attacks that are included in AutoAttack.
• obviously, epsilon should be the same for all attacks, when comparing them (on the same dataset and Lp norm). Since the success rate increases with epsilon (more visible perturbations)
• overall, it is key to understand the algorithm of an attack to adapt its hyperparameters. Therefore, starting from good set of hyperparameters is at utmost importance to know what changed.

[rikonaka]

My generic advice is to use the same experimental settings of another paper when you try to reproduce its experiments.

Here:

* it is true on every dataset that the max norm of the perturbation epsilon needs to be changed when changing the type of Lp norm

* when changing from one dataset to another, epsilon needs to be changed, if the input dimension changed (number of pixels)

* setting the step-size alpha = epsilon/10 should work on every dataset. Almost everytime, it is best to have more iterations than 10 (probably around 50 or 100). See the hyperparameters of the PGD attacks that are included in AutoAttack.

* obviously, epsilon should be the same for all attacks, when comparing them (on the same dataset and Lp norm). Since the success rate increases with epsilon (more visible perturbations)

* overall, it is key to understand the algorithm of an attack to adapt its hyperparameters. Therefore, starting from good set of hyperparameters is at utmost importance to know what changed.

Thank you very much for your patient reply, I have benefited a lot, but I have another question. 😋😋😋

obviously, epsilon should be the same for all attacks, when comparing them (on the same dataset and Lp norm). Since the success rate increases with epsilon (more visible perturbations)

When I use PGD in this way,

atk = PGD(model, eps=8/255, alpha=2/255, steps=10, random_start=True)

I can get a very high attack success rate, but when I put the same hyperparameters on PGDL2,

atk = PGDL2(model, eps=8/255, alpha=2/255, steps=10, random_start=True)

I can not get the same attack success rate, why? 🫣🫣🫣 In other words, when the same perturbation produces different attack success rates, should I keep the best perturbation method (like PGD), or increase the all perturbations to the same level so that all attacks conform to a basic attack success rate (this will cause PGD to have obvious perturbations).

[Framartin]

Again, the value of the max norm of the perturbation (epsilon) cannot be compared between Lp norms (it is not the same perturbation strength, 8/255 might be big for Linf and tiny for L2). You cannot compare directly the success rates of Linf and L2 attacks. You should read more papers to see how they compare attacks (Linf and L2 attacks are considered two distinct experimental settings).

obviously, epsilon should be the same for all attacks, when comparing them (on the same dataset and Lp norm). Since the success rate increases with epsilon (more visible perturbations)

I meant that the success rate of different attacks (PGD vs. AutoAttack for example) should be compared at constant epsilon and constant dataset (see for example how the AutoAttack paper compares attacks, or how the RobustBench paper compares defenses).

Let's kindly close the discussion on this GitHub issue, as it seems to have moved away from the original topic

[rikonaka]

Thank you very much!!! 😘😘😘