Closed PC09 closed 3 years ago
I am experiencing the same issue. The worst scores are replicated for NLI and QA.
The runs where training diverges are the ones where you get 33% accuracy. Try running with different hyperparameters (seeds, batch_size, learning rate).
For QA, there's no test set right ? So the evaluation is done on the dev set itself. I'm able to converge the training but printing out the F1 score locally (over ~70) is far from the score derived via the Pull Request(which is around 23)? I'm not able to figure out the inconsistency here? Can you please help clarify ?
The test set for both QA and NLI are small, so fluctuations in the score are very easily possible. Getting good numbers on the train set alone won't be enough. I will suggest that you try running with smaller batch size (2 or 4) and smaller learning rate (1e-5 or 5e-6) and see what you get. We had to follow a similar procedure to get the results we reported.
Thanks @Genius1237 , I was trying with the default hyper parameters. Will try tuning it as mentioned above.
Hi!
I was trying to replicate the results for NLI task using multilingual BERT model. The GLUECoS paper says, mBERT gives 61.09 or 57.74 (from the leaderboard). When I am running the sample NLI script here with default parameters, my test accuracy is coming out to be very low ~33. Could anyone confirm that the numbers in the paper are for baseline or not? Also, I see the data was updated, are these numbers for older version of data?
Thanks!