search

deepset-ai / FARM

:house_with_garden: Fast & easy transfer learning for NLP. Harvesting language models for the industry. Focus on Question Answering.
https://farm.deepset.ai
Apache License 2.0
1.73k stars 247 forks source link

reproducing results in huggingface #840

Closed shairoz-deci closed 2 years ago

shairoz-deci commented 2 years ago

Question I've train roberta base on squad v2 using 

and got an f1 validation score of 83.2.

I ran the the model in huggingface using the following command:

python -u transformers/examples/pytorch/question-answering/run_qa.py --model_name_or_path FARM/saved_models/bert-english-qa-tutorial/ --train_file squad2/train-v2.0.json --validation_file squad2/dev-v2.0.json --do_eval --per_device_train_batch_size 4 --max_seq_length 384 --doc_stride 128 --output_dir /tmp/debug_squad/ --tokenizer_name FARM/saved_models/bert-english-qa-tutorial --version_2_with_negative --overwrite_output_dir



and got an f1 validation score of 5.8 . Can you please provide instructions on how to migrate the model to huggingface?

**Additional context**
I am able to load and evaluate the model from https://huggingface.co/deepset/roberta-base-squad2 and reach the published f1.

Thanks in advanced
shairoz-deci commented 2 years ago

Found the conversion example, closing