search

jind11 / TextFooler

A Model for Natural Language Attack on Text Classification and Inference
MIT License
485 stars 79 forks source link

Cannot reproduce results with FAKE #60

Open AntoineGourru opened 1 year ago

AntoineGourru commented 1 year ago

Hi Di Jin and collaborators,

First of all, thank you for this great repo, this is highly valuable to have reproducible code/data/models of this quality.

I am currently trying to reproduce the scores from your paper, everything works well, I manage to observe the same accuracy than what's pointed in your article for imdb and yelp.

Nevertheless, the FAKE bert pretrained model that I took from your drive produces random results (I get 50% accuracy while it should be 96.7)

Are you sure that you uploaded the good model on drive ?

Best regards,

Antoine

jind11 commented 1 year ago

I think the pre-trained model should be the correct one. Could you double check whether the model parameters are initialized correctly?