RaphaelOlivier / robust_speech

Apache License 2.0
40 stars 14 forks source link

pgd results #15

Open xiaoxue1117 opened 1 year ago

xiaoxue1117 commented 1 year ago

Hi,

Congrats to your great work! I am wondering why the L2-PGD implementation is different from the results in your paper (Fig.1 a) and if I should adjust any parameters. Your paper shows WER =25% with SNR=20 and WER=10% with SNR=30 (refer to the yellow line Trf LAS in Fig 1 a), while the implementation results are WER=115 with SNR=20 and WER=86.78 with SNR=30. You may refer to the following results and commands. I did not change any code.

python evaluate.py attack_configs/LibriSpeech/pgd/trf_5000bpe.yaml --root=./lib --snr=30

Results for TEST- test loss: 1.71e-01, test adv loss: 15.71, test CER: 1.24, test WER: 3.39, test adv CER: 60.28, test adv WER: 86.78

python evaluate.py attack_configs/LibriSpeech/pgd/trf_5000bpe.yaml --root=./lib --snr=20

Results for TEST - test loss: 1.72e-01, test adv loss: 25.74, test CER: 1.24, test WER: 3.39, test adv CER: 82.17, test adv WER: 1.15e+02

Look forward to your reply!

RaphaelOlivier commented 1 year ago

Hi, It seems that you are correct and the values in the paper are not up to date anymore. Several things have changed since that paper:

Because of this, it is hard to pinpoint what exactly is the root cause of this change. We intend to add all current default attack results to the README in the near future.

Thanks for pointing this out!