Closed AliHadizadeh closed 3 years ago
Hello! This may be because of labels being switched around for the MNLI task. See this thread https://github.com/huggingface/transformers/pull/10203 for more context.
Hello, Many thanks for your response. Yes, that seems to be the source of my issue, and now I can get the accuracy. Thanks!
I think there is also a specific problem in huggingface/distilbert-base-uncased-finetuned-mnli
: its labels seem wrongly coded. Using them specifically and evaluating gives me 34% accuracy.
Yes. But other models seem to work with the modification that I made https://github.com/huggingface/transformers/pull/10203#discussion_r582971857
@sgugger Can someone fix this, or remove the model from the model hub? This is a serious gotcha and cost me a couple weeks of confusion!
The model has been fixed a year ago, in this commit
Thank you for clarifying @sgugger! I think we had an old copy
Environment info
transformers
version: 4.4.dev / 4.3.3 / 4.3.2Who can help
@patil-suraj , @sgugger, @LysandreJik ## Information Model I am using (Bert, XLNet ...): huggingface/distilbert-base-uncased-finetuned-mnli - microsoft/deberta-v2-xxlarge-mnli - roberta-large-mnli - squeezebert/squeezebert-mnli - BERT-Base-MNLI.... The problem arises when using: * [x] the official example scripts: (give details below) * [ ] my own modified scripts: (give details below) The tasks I am working on is: * [x] an official GLUE/SQUaD task: (give the name) * [ ] my own task or dataset: (give details below) I use run_glue.py on fine-tuned models to reproduce the evaluation result (only `--do_eval`). But the accuracy is about 7%. Other tasks like MRPC or STS-B are ok when I use their fine-tuned models. ## To reproduce Steps to reproduce the behavior: 1. Run `python run_glue.py --model_name_or_path huggingface/distilbert-base-uncased-finetuned-mnli --task_name mnli --do_eval --max_seq_length 128 --output_dir temp/distill` or any other MNLI fine-tuned model. I even tried a model that I fine-tuned myself using V2.10.0 and that again results in 6%-7% accuracy.Expected behavior
It seems all the weights are loaded in the correct place, but the accuracy is below 10% which should be above 80%.