Problem with evaluate part on the colab notebook

[haythemtellili]

I am not able to get prediction.json file ! please any help will be appreciated

[husseinmozannar]

Will look at this soon, I might be releasing a new colab that fixes all the issues and makes it easier.

[husseinmozannar]

I made some changes to the evaluation script, it should have been able to run smoothly before although, try running the updated Colab now and let me know exactly what is happening.

[haythemtellili]

Yes it worked thank u but why the f1,em are very low !

[husseinmozannar]

For how long did you train (the number in "arabic_squad_train/model.ckpt-XXXX")? and what results are you getting? Ballpark values of F1 should be around 50, check the paper https://arxiv.org/pdf/1906.05394.pdf for full result breakdowns. Some recent work have been able to do better and get up to 70 F1 but not sure if their code is yet available.

[haythemtellili]

I just used your colab notebook, i think you have been using the pertained Bert multilingual model , so where can i find arabic_squad_train checkpoint and have i retrain the model ?

[husseinmozannar]

If you follow the notebook exactly, this is what you should get (gpu instance on Colab takes maybe 1-2hours): {"exact_match": 20.21505376344086, "f1": 52.098255315053166, "exact_sentence": 91.11111111111111}

Now to improve on this, one can train on training set of ARCD for one more epoch for example, or use Arabic trained Bert.

[husseinmozannar]

There is also an available to download trained Bert model . This is linked on the main readme page.

[haythemtellili]

Thank you but i don't know why when i try to run your colab, it ran just for 5 min and then i get too low result (6 as f1 score) !

[husseinmozannar]

Please make sure to have the Colab run with a GPU and make sure the training epochs is set tot 2

[husseinmozannar]

Closing this issue for now, GPU required to run the colab and results are consistent.