search

RockyLzy / TextDefender

codes for "Searching for an Effective Defender:Benchmarking Defense against Adversarial Word Substitution"
MIT License
31 stars 2 forks source link

Reproducibility Issue #7

Open Han8931 opened 2 years ago

Han8931 commented 2 years ago

Could you upload your dataset, please? I can't run your code because I have no datasets. I want to run experiments with AGNews and IMDb. I tested your FreeLB++ code on IMDb by importing it via Huggingface dataset library and I used your parameter settings, but I couldn't reproduce the number although it showed some improvements. Also, SAFER and InfoBERT were not really good in IMDb experiments. I wish I can try your full code as soon as possible.

dangne commented 2 years ago

I also have the same issue with the code. I got ~3% accuracy under attack for FreeLB++ on IMDB against BERT-Attack.

Han8931 commented 2 years ago

I also have the same issue with the code. I got ~3% accuracy under attack for FreeLB++ on IMDB against BERT-Attack.

I got around 70~80% attack success rate with TextFooler attack against FreeLB++. I set a max modification rate at 0.3 and 1.0, so I tested both rates. I tested with Huggingface IMDb dataset and RoBERTa-base model. I also didn't use their custom attack implementation. Instead, I used the official TextAttack library with modified parameter settings to reproduce their experiments.

ColinLu50 commented 2 years ago

I also have the same issue with IMDB dataset. I ran FreeLB++ and got around 20% accuracy under attack (TextFooler) on IMDB. I set the step size to 10 as mentioned in the paper. And I used the modified TextFooler attack from this project. Could the author share the hyperparameter on IMDB?

BTW, I also cannot reproduce DNE results on IMDB. The training accuracy (clean) is about 80%, which is lower by 10% as mentioned in paper. Is there something wrong?

Han8931 commented 2 years ago

I checked other adversarial attack papers from the lab of the authors and it seems 20% acc with FreeLB++ is the right one. I am not 100% sure though. Also, I couldn't reproduce SAFER's performance too.

ColinLu50 commented 2 years ago

ther adversarial attack papers from the lab of the authors and it seems 20% acc with FreeLB++ is the right one. I am not 100% s

Thansk for your response! I think the shared codes have some problems. Maybe running the codes provided by the original paper is better choice.

Han8931 commented 2 years ago

I agree with you. Actually, I checked the original SAFER code too, but it seems the original code has another issue too.

ColinLu50 commented 2 years ago

Yeah, nowadays a lot of codes have problems, even the works published on top conferences.

Han8931 commented 2 years ago

I know what you mean but I want to be cautious and this is a repository of their paper loool

MrigankRaman commented 2 years ago

I checked other adversarial attack papers from the lab of the authors and it seems 20% acc with FreeLB++ is the right one. I am not 100% sure though. Also, I couldn't reproduce SAFER's performance too.

Can you tell me the paper where they have 20% for FreeLB++?