search

locuslab / fast_adversarial

[ICLR 2020] A repository for extremely fast adversarial training using FGSM
434 stars 92 forks source link

Include python/pytorch version for MNIST reproducibility #25

Closed latorrefabian closed 2 years ago

latorrefabian commented 2 years ago

Hi! I am having a hard time reproducing the results (on MNIST, for example) and I have found that they differ when I change the pytorch version. I observe the following:

pytorch 1.12: when training with MNIST, training accuracy of 0.98 is achieved, but robust test accuracy is zero pytorch 1.4: when training with MNIST, training accuracy of 0.95 is achieved, robust test accuracy is 0.88

I think the code was originally run with pytorch 1.0, I am trying to find out what is breaking the code in pytorch 1.12. It would be great to make it more clear which versions to use to reproduce the results

leslierice1 commented 2 years ago

I'm not sure why the results would be different with different pytorch versions, however I'm guessing that the model is catastrophically overfitting in the first case, resulting in zero robust test accuracy. I'd try checking during training whether this is occurring by checking the PGD accuracy on the first training batch of each epoch (i.e. see how we do this in the CIFAR10 training code here). I'd also try reducing the alpha parameter to avoid catastrophic overfitting if this is indeed what is occurring.