XLM-RoBERTa model for QA seems not properly work

antoniolanza1996 commented 4 years ago

Environment info

transformers version: 3.3.1 (and I've also tried installing Transformers from master, see details below)
Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
Python version: 3.6.9
PyTorch version (GPU?): 1.6.0+cu101 (False)
Tensorflow version (GPU?): 2.3.0 (False)
Using GPU in script?: Yes
Using distributed or parallel set-up in script?: No

Who can help

albert, bert, GPT2, XLM: @LysandreJik

Information

Model I am using (Bert, XLNet ...): deepset/xlm-roberta-large-squad2

The problem arises when using:

[x] the official example scripts: run_squad.py
[ ] my own modified scripts: (give details below)

The tasks I am working on is:

[x] an official GLUE/SQUaD task: SQuAD 2.0 dev set evaluation
[ ] my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

! wget https://raw.githubusercontent.com/rajpurkar/SQuAD-explorer/master/dataset/dev-v2.0.json
! python transformers/examples/question-answering/run_squad.py \
  --model_type xlm-roberta \
  --model_name_or_path 'deepset/xlm-roberta-large-squad2' \
  --do_eval \
  --do_lower_case \
  --predict_file 'dev-v2.0.json' \
  --output_dir 'output' \
  --overwrite_output_dir \
  --version_2_with_negative

Expected behavior

There are some values mismatch between:

values reported in the model card here
values obtained when Transformers is installed using pip install transformers
values obtained when Transformers is installed from master

In particular:

Reported metrics in the model card: "exact": 79.45759285774446, "f1": 83.79259828925511
Transformers installed from pip: 'exact': 64.67615598416576, 'f1': 77.27580544355429
Transformers installed from master: 'exact': 60.11959090373114, 'f1': 76.13129575803934

LysandreJik commented 4 years ago

Thanks for reporting, will investigate.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Forutanrad commented 2 years ago

hello, this command " --model_type xlm-roberta \" not work for me, can you help me? please

huggingface / transformers