pgd results

[xiaoxue1117]

Hi,

Congrats to your great work! I am wondering why the L2-PGD implementation is different from the results in your paper (Fig.1 a) and if I should adjust any parameters. Your paper shows WER =25% with SNR=20 and WER=10% with SNR=30 (refer to the yellow line Trf LAS in Fig 1 a), while the implementation results are WER=115 with SNR=20 and WER=86.78 with SNR=30. You may refer to the following results and commands. I did not change any code.

python evaluate.py attack_configs/LibriSpeech/pgd/trf_5000bpe.yaml --root=./lib --snr=30

Results for TEST- test loss: 1.71e-01, test adv loss: 15.71, test CER: 1.24, test WER: 3.39, test adv CER: 60.28, test adv WER: 86.78

• test Adversarial SNR: {'average': 30.879302978515625, 'min_score': 21.0, 'max_score': 40.0}

python evaluate.py attack_configs/LibriSpeech/pgd/trf_5000bpe.yaml --root=./lib --snr=20

Results for TEST - test loss: 1.72e-01, test adv loss: 25.74, test CER: 1.24, test WER: 3.39, test adv CER: 82.17, test adv WER: 1.15e+02

• test Adversarial SNR: {'average': 22.512958526611328, 'min_score': 12.0, 'max_score': 31.0}

Look forward to your reply!

[RaphaelOlivier]

Hi, It seems that you are correct and the values in the paper are not up to date anymore. Several things have changed since that paper:

• We have changed the default values of our attacks to improve results - notably the learning rates/step sizes.
• The speechbrain models have also been retrained with different hyperparameters. The HuggingFace hub paths have remained the same but the models are different, so we can't access the former ones anymore.

Because of this, it is hard to pinpoint what exactly is the root cause of this change. We intend to add all current default attack results to the README in the near future.

Thanks for pointing this out!