Adversarial training on cifar10?

[Originofamonia]

Hello,

I really appreciate your work! When I tried to perform the adversarial training on the CIFAR10 dataset by modifying the code of tutorial_train_mnist.py. I changed the get_mnist_train_loader and get_mnist_test_loader functions to get_cifar10_train_loader and get_cifar10_test_loader, and the LeNet5 model's input dimensions accordingly. But the problem is the loss doesn't decrease and clean acc and adv acc is always at 10%. I also tried to use a larger model like Resnet. But the problem is the same. So any ideas on why the loss doesn't decrease for cifar10?

`Train Epoch: 1 [0/50000 (0%)] Loss: 2.420085 Train Epoch: 1 [20000/50000 (40%)] Loss: 2.302042 Train Epoch: 1 [40000/50000 (80%)] Loss: 2.303400

Test set: avg cln loss: 2.3025, cln acc: 1000/10000 (10%)

Test set: avg adv loss: 2.3035, adv acc: 1000/10000 (10%)

Train Epoch: 2 [0/50000 (0%)] Loss: 2.300893 Train Epoch: 2 [20000/50000 (40%)] Loss: 2.303464 Train Epoch: 2 [40000/50000 (80%)] Loss: 2.303012

Test set: avg cln loss: 2.3026, cln acc: 1000/10000 (10%)

Test set: avg adv loss: 2.3027, adv acc: 1000/10000 (10%)

Train Epoch: 3 [0/50000 (0%)] Loss: 2.301586 Train Epoch: 3 [20000/50000 (40%)] Loss: 2.301844 Train Epoch: 3 [40000/50000 (80%)] Loss: 2.303260

Test set: avg cln loss: 2.3025, cln acc: 1000/10000 (10%)

Test set: avg adv loss: 2.3031, adv acc: 999/10000 (10%)

Train Epoch: 4 [0/50000 (0%)] Loss: 2.303174 Train Epoch: 4 [20000/50000 (40%)] Loss: 2.302358 Train Epoch: 4 [40000/50000 (80%)] Loss: 2.302135

Test set: avg cln loss: 2.3025, cln acc: 1008/10000 (10%)

Test set: avg adv loss: 2.3029, adv acc: 1000/10000 (10%)

Train Epoch: 5 [0/50000 (0%)] Loss: 2.303104 Train Epoch: 5 [20000/50000 (40%)] Loss: 2.303405 Train Epoch: 5 [40000/50000 (80%)] Loss: 2.301460

Test set: avg cln loss: 2.3023, cln acc: 1000/10000 (10%)

Test set: avg adv loss: 2.3032, adv acc: 1000/10000 (10%)

Train Epoch: 6 [0/50000 (0%)] Loss: 2.303206 Train Epoch: 6 [20000/50000 (40%)] Loss: 2.300870 Train Epoch: 6 [40000/50000 (80%)] Loss: 2.303452

Test set: avg cln loss: 2.3025, cln acc: 1000/10000 (10%)

Test set: avg adv loss: 2.3028, adv acc: 1000/10000 (10%)

Train Epoch: 7 [0/50000 (0%)] Loss: 2.302966 Train Epoch: 7 [20000/50000 (40%)] Loss: 2.302667 Train Epoch: 7 [40000/50000 (80%)] Loss: 2.303157

Test set: avg cln loss: 2.3025, cln acc: 1238/10000 (12%)

Test set: avg adv loss: 2.3028, adv acc: 724/10000 (7%)

Train Epoch: 8 [0/50000 (0%)] Loss: 2.302794 Train Epoch: 8 [20000/50000 (40%)] Loss: 2.302416 Train Epoch: 8 [40000/50000 (80%)] Loss: 2.302629

Test set: avg cln loss: 2.3025, cln acc: 1000/10000 (10%)

Test set: avg adv loss: 2.3027, adv acc: 1000/10000 (10%)

Train Epoch: 9 [0/50000 (0%)] Loss: 2.302886 Train Epoch: 9 [20000/50000 (40%)] Loss: 2.302330 Train Epoch: 9 [40000/50000 (80%)] Loss: 2.302263 `

Thanks!

[gwding]

@Originofamonia It's a bit hard for me to understand what's going on without looking at the specifics. Hopefully I'll put up a tested CIFAR10 adversarial training script later this week, and maybe that'll be helpful. Will reply here once that's up.

[loveWhitney]

I have the same result, but the training by using L1 is much better.

[csdongxian]

The common setting in CIFAR-10 under L-inf threat model is: eps=0.031, nb_iter=10 or 7, eps_iter=0.007. The original setting in the tutorial is for MNIST, which is too difficult to defend in CIFAR-10.