Adversarial Training not working correctly.

[giorgiopiras]

Hi @ByungKwanLee and thanks for your work on this repo. I was trying to reproduce some experiments, and if I understood correctly, the MAD pipeline works as follows: 1-Clean pre-training of the model with main_pretrain.py. 2-Robust training of the model with main_robustobj_pretrain.py (either trades, at or mart). 3-Saliency computation with compute_saliency.py 4-Pruning (and fine-tuning as well) with main_mad_pretrain.py.

Problem

While step-1 works just fine, I reckon step-2 simply doesn't. With any kind of robust objective you choose, when using the PGD attack, the training process won't reduce the loss. As a matter of fact, accuracy gets stuck at around 10%. It looks like the attack is just too powerful, except it's not, it's tuned with your default PGD parameters.

When changing the attack to (say) FGSM, the training will work, meaning that the evaluation throughout training looks fine... but the network will then learn only on the attack samples and won't generalize to clean ones. Also, alternating two epochs (1 clean, 1 adv) won't solve the issue.

Question

I was wondering whether you faced the same issues, and how did you make your code work for the paper's experiments. I am not sure if this is a problem with torchattacks, honestly. Thanks

Log from: Adversarial Training with PGD on ResNet-18

The training gets stuck as such:

==> Loading Adv checkpoint..
checkpoint/pretrain/cifar10/cifar10_resnet18_best.t7
Low PGD training
=> Using a preset learning rate schedule:
{0: 0.1, 30: 0.010000000000000002, 45: 0.001}

Epoch: 0
[Train/LR=0.1] Loss: 2.352 | Acc: 9.814% (4907/50000): 100%|██████████████████████████████████████████| 391/391 [02:08<00:00,  3.03it/s]
[Test/LR=0.1] Loss: 2.304 | Acc: 10.000% (1000/10000): 100%|████████████████████████████████████████████| 40/40 [00:22<00:00,  1.80it/s]
Saving..
./checkpoint/pretrain/cifar10/cifar10_adv_resnet18_best.t7

Epoch: 1
[Train/LR=0.1] Loss: 2.305 | Acc: 10.054% (5027/50000): 100%|█████████████████████████████████████████| 391/391 [02:10<00:00,  3.00it/s]
[Test/LR=0.1] Loss: 2.304 | Acc: 10.000% (1000/10000): 100%|████████████████████████████████████████████| 40/40 [00:22<00:00,  1.79it/s]

Epoch: 2
[Train/LR=0.1] Loss: 2.305 | Acc: 9.852% (4926/50000): 100%|██████████████████████████████████████████| 391/391 [02:06<00:00,  3.10it/s]
[Test/LR=0.1] Loss: 2.304 | Acc: 10.000% (1000/10000): 100%|████████████████████████████████████████████| 40/40 [00:22<00:00,  1.80it/s]

[billy1902]

hi,I wonder if you met question like this : when I do the step of "python compute_saliency.py": Traceback (most recent call last): File "compute_saliency.py", line 129, in onehot_dict = optimizing_mask() File "compute_saliency.py", line 89, in optimizing_mask adv_x = attack(inputs, targets) File "/root/miniconda3/envs/torch18/lib/python3.8/site-packages/torchattacks/attack.py", line 511, in call adv_inputs = self.forward(inputs, labels, *args, kwargs) File "/root/miniconda3/envs/torch18/lib/python3.8/site-packages/torchattacks/attacks/pgd.py", line 63, in forward outputs = self.get_logits(adv_images) File "/root/miniconda3/envs/torch18/lib/python3.8/site-packages/torchattacks/attack.py", line 79, in get_logits logits = self.model(inputs) File "/root/miniconda3/envs/torch18/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, *kwargs) File "/mnt/bn/hlsyh/mlx/users/sunyouheng/code/school/Masking-Adversarial-Damage-master/models/vgg_mask.py", line 56, in forward x = self.feature(x) File "/root/miniconda3/envs/torch18/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(input, kwargs) File "/root/miniconda3/envs/torch18/lib/python3.8/site-packages/torch/nn/modules/container.py", line 119, in forward input = module(input) File "/root/miniconda3/envs/torch18/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl result = self.forward(*input, **kwargs) File "/root/miniconda3/envs/torch18/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 399, in forward return self._conv_forward(input, self.weight, self.bias) TypeError: _conv_forward() takes 3 positional arguments but 4 were given.

I guess it supposed to be the problem of pytorch version or something ,but I am not sure.If convenient ,could you show your visual vironment?Thank you for your help!!!!

[giorgiopiras]

Hi @billy1902. Unfortunately, I do not remember if I have encountered this error. I managed to run the code, but my problem was that the loss wouldn't go down with the specified hyperparameters...and even standard fixes such as a smaller learning rate wouldn't solve the problem. If you want to have access to more pruned model you can also check my new repo at https://github.com/pralab/AdversarialPruningBenchmark Here the idea is to store and benchmark pruned models. However, since I could not fit the models properly, the MAD models are not there.

Best, Giorgio