search

tml-epfl / understanding-fast-adv-training

Understanding and Improving Fast Adversarial Training [NeurIPS 2020]
https://arxiv.org/abs/2007.02617
94 stars 12 forks source link

free AT doesn‘t work in SVHN #2

Closed ctm-hxds closed 3 years ago

ctm-hxds commented 3 years ago

I run svhn with "python train.py --dataset=svhn --attack=fgsm --eps=8 --attack_init=zero --fgsm_alpha=1.0 --minibatch_replay=8 --epochs=45 --lr_max=0.01 --eval_iter_freq=200 --batch_size_eval=1024 --half_prec --eval_early_stopped_model --n_final_eval=1000 --seed=0 " But it didn't work... the accuracy is only 23% so....what's wrong ????

max-andr commented 3 years ago

Hi,

Thanks for the interest in our paper.

As far as I can see, you want to train a model using the approach from "Adversarial training for free". As shown in FIg. 8 (left), we didn't experience any problem training it even when replicated over 5 random seeds.

Did you modify the provided code in some way? Based on my experience, the warmup of the epsilon is quite important on SVHN to prevent the convergence to the most popular class (which leads to 23% accuracy or so): https://github.com/tml-epfl/understanding-fast-adv-training/blob/master/train.py#L132 Do you use it? Maybe as a sanity check, you can also try to increase it a bit, say, to 10 epochs instead of 5 and see whether it helps. Although in my experiments, it was working well even with 5 warmup epochs.

Best, Maksym

max-andr commented 3 years ago

Closing it for now. In case you have further questions, feel free to reopen.