Question about the results of the attack

jind11 / TextFooler

A Model for Natural Language Attack on Text Classification and Inference

MIT License

485 stars 79 forks source link

Question about the results of the attack #37

Open milumilum opened 3 years ago

milumilum commented 3 years ago

hi， when I experimented with the 1000 samples you provided from the AG dataset and the bert model, the results are: For dataset data/ag: For target model bert: original accuracy: 94.200%, adv accuracy: 29.500%, avg changed rate: 25.076%, num of queries: 481.7 But the adv accuracy in your paper is 11.5%. I'm not quite sure what led to this result. So did you use the same 1000 examples in your paper?

Thanks.

jind11 commented 3 years ago

I am also not sure why there is such a huge change. Did yo use the same BERT model parameters that I provided?

milumilum commented 3 years ago

Yes, the model parameters and data are all provided by you.All the other attacks are close to the results given in your paper, only this one deviates a lot, so I wonder whether it is due to the data.

Youoo1 commented 2 years ago

Where is the emdding.npz file, please? Or how is it generated? b9069123768ea397299dc7ed1419901 7a678cd5f2a8398b7980d8aaa9d5aec