Replication of the Result in Madry's Cifar10 Challenge

[yguooo]

In your experiment for Cifar10, the l_inf perturbed model has a perturbation size of 4/255 = 0.0157. The achieved robust accuracy achieved in their paper (https://arxiv.org/abs/1805.12152) is 57.79% as in table 4, whereas your experiment achieves a robust accuracy of 63.8%

Could you please give some explanation about this difference.

[ylsung]

Hi,

Sorry for the late replying.

In this code, when testing, the target label used in creating the adversarial example (untarget attack) is the most confident prediction of the model, not the ground truth. In my view, it is a practical method since the attacker should not know the true label in the testing mode.

However, in Madry's Cifar10 Challenge, they use the true label as the target label. I think this cause the difference. In my setting, the adversarial example, created by the image which is wrongly classified at first , might be correctly classified.

[JonathanCMitchell]

Did you try testing using their method? If you are using the true class label does your results match theirs?

I also checked that your model as 36,489,240 trainable parameters while the one over at https://github.com/MadryLab/cifar10_challenge has 45,901,914 trainable parameters.

[ylsung]

Hi, @JonathanCMitchell, thank you for pointing out these issues!

Did you try testing using their method? If you are using the true class label does your results match theirs?

Thanks for the suggestion. I will try to test using their method. However, I have already deleted all my checkpoints, so it will take a few days.

I also checked that your model as 36,489,240 trainable parameters while the one over at https://github.com/MadryLab/cifar10_challenge has 45,901,914 trainable parameters.

There is something strange here, since I just see that the trainable parameters are 48,823,962 in my model (WideResNet-34). I add count_parameters(model) in src/utils/utils.py, and test them in the last line in src/model/model.py. You can have a check.

However, the number is still different from which of the model in MadryLab. Finally, I find the reason. In Resnet block, sometimes the the size of residual cut is mismatch with the main stream and most implementations use conv1*1 to resize the residual cut while they use padding in their codes (Madry_model line 134 ~ 140). If I comment out the line 76 and 77 in my codes, two models will have same numbers of parameters (45,901,914).

I will use their model to re-run.

[JonathanCMitchell]

Hello, what is the status of this? Did you actually match their benchmark? Is there any chance you could update your model to match theirs in terms of param count? Is your code currently modified so that it is testing the same way madry's code is? Using the ground truth label??

What is the exact command that you use to train your model because I see two of then inside train.sh?

[ylsung]

Hi, I update all the things to the README, please referring to it. The linf results become closer to which of Madry, but not the same. I guess that the number will be more convincing if one repeat the experiments more times.

Using the ground truth label?

Yes.

What is the exact command that you use to train your model because I see two of then inside train.sh?

Please refer to new train.sh in the repo.

[JonathanCMitchell]

Thanks for the note. What's the new accuracy (the latest that matches madry)? It shouldn't be 61% it should be ~44%.

[ylsung]

I have no idea where these two numbers, 61% and 44%, come from. Do they appear in the ReadMe or Madry's paper?

[JonathanCMitchell]

44% is the accuracy of the white box attack using LINF 8/255 on the adversarially trained model using Madry's wideresnet. (from their paper)

[ylsung]

It seems that it is 42% instead of 44% in their paper, but this typo might not that important here.

I used L_{inf} 4/255 as the bound of the attacks in this repo and got robust accuracy as 55.97%. The result of the same method is 57.79% in Madry's paper. There is a small gap, but you might repeat the training more times to get convincing results.

The new reproduced accuracies are in the row that start from "Madry's Model Standard Accuracy (train/test)" in the table.

[Harry-Up]

@louis2889184 Sorry, I am confused about the result. The result of Madry's wideresnet using LINF 8/255 and 10 steps on the adversarially trained model (the offered checkpoint in the cifar-10 challenge) is 47%. And the robust accuracy under linf pgd attack with 8/255 in the paper (https://arxiv.org/abs/1805.12152) is 41.93%. Any suggestions for such a gap?

[ylsung]

Hi @Harry-Up. Do you mean that the robust accuracies are higher for models with more channels? It seems an interesting finding.

wideresnet*1 (channels [16, 16, 32, 64]) in the other paper.

Could you refer to some papers for this? Thank you.

[Harry-Up]

Hi, @louis2889184 In the original paper (https://arxiv.org/abs/1706.06083), the model with more capacity is more robust. So, it makes sense that models with more channels are of higher robust accuracies.

However, I am still confused about the performance gap between this implementation and https://github.com/MadryLab/cifar10_challenge. Any suggestions? Thank you!

[ylsung]

@Harry-up. Sorry for the late reply. I found that there is a mismatch between this repo and https://github.com/MadryLab/cifar10_challenge.

1. In the setting of 20-step PGD on the cross-entropy loss (which gets the 47.04 % accuracy), it seems that they use PGD with 7 steps of size 2 to train the model (reference) while using 20 steps of size 1 to attack it (reference). On the other side, I use PGD with 10 steps of size 2 to both train and test the model.

The gap might come from the difference. I will update the codes as soon as possible. Thank you for reporting the issues.

[Harry-Up]

@louis2889184 Thank you very much. It helped me a lot. I was depressed because I couldn't get the expected result using your repo code and standard parameters. Namely, I use PGD algorithm in the linf norm with 8 / 255 eps, 2 / 255 step size, and 10 steps to train and attack the WRN-34 * 10 model, but I can only get about 42% robust accuracy. In almost published papers, the Madry's model with 47% robust accuracy was directly claimed. This made me wonder whether I had done something wrong to get such a bad result. Thank you again for your advice.

[Harry-Up]

@louis2889184 I find that the real reason for this performance gap is the learning rate. I use eps=8, steps=10, and step size=2 to train and test the model with Madry's code. When I set the initial learning rate to 0.01 like this repo setting, the robust accuracy is 43%. As I set the initial learning rate to 0.1, the robust accuracy rate returned to 47%. That‘s it! Another difference to mention is that the ReLu function actually uses LeakyReLu in Madry's code. Thanks for your attention!

[ylsung]

@Harry-Up. Thank you for your effort in finding this.

Does your experiment use their training settings (PGD with 7 steps of size 2 to train the model (reference) while using 20 steps of size 1 to attack it)?

I also found a difference is optimizers (they use SGD, I use Adam).

I will try your setting and update the results soon. Thank you.

[Harry-Up]

@louis2889184. I just adopt the same training and test settings, which is the PGD attack with 10 steps of size 2. Sorry, I do not find the difference in the optimizer, since I use the SGD optimizer directly in my adaptation. Thank you for sharing the code, it is a good start for beginners.

[ylsung]

Activation, learning, and optimizer are updated. The experiment uses use PGD with 20 steps of size 1 to attack the model trained on PGD with 7 steps of size 2 is about 48%, which is pretty close to the madry_lab's result. Thank you for your huge help.