search

jind11 / TextFooler

A Model for Natural Language Attack on Text Classification and Inference
MIT License
485 stars 79 forks source link

Low Accuracy of BERT Pretrained Model on Fake News #15

Closed RishabhMaheshwary closed 4 years ago

RishabhMaheshwary commented 4 years ago

The orig_failures count is 408 on fake news BERT. Is the pretrained model you uploaded the correct one?

jind11 commented 4 years ago

May I ask what is the "max_seq_length" you have set?

RishabhMaheshwary commented 4 years ago

512

jind11 commented 4 years ago

I checked my training code for the fake dataset and found that I used 256 as the "max_seq_length". I also double checked that the pre-trained model parameters can achieve 99.9% accuracy using this length setting.

RishabhMaheshwary commented 4 years ago

For Infersent, I trained their model from scratch. But when I load the pretrained model, I get the following error:

RuntimeError: Error(s) in loading state_dict for NLINet: Missing key(s) in state_dict: "classifier.4.weight", "classifier.4.bias", "classifier.7.weight", "classifier.7.bias". Unexpected key(s) in state_dict: "classifier.0.weight", "classifier.0.bias", "classifier.2.weight", "classifier.2.bias". size mismatch for classifier.1.weight: copying a param with shape torch.Size([512, 512]) from checkpoint, the shape in current model is torch.Size([512, 16384]).

Edit: Solved by changing 'nonlinear_fc' to 0 in config.

jind11 commented 4 years ago

Glad to hear that you have solved the error. I will close this issue, but if you have further questions, feel free to open it.