Getting 0 for eval set metrics

nyu-dl / dl4marco-bert

BSD 3-Clause "New" or "Revised" License

476 stars 87 forks source link

Getting 0 for eval set metrics #48

Closed candeveci13 closed 3 years ago

candeveci13 commented 3 years ago

Hi,

When I follow the instruction for MS Marco, I obtain the expected results for the dev set but not for eval set. All the evaluation metrics are 0 after evaluating each 1k samples. I also tried the code with the TFRecord files provided but the same results for that. What am I missing?

rodrigonogueira4 commented 3 years ago

That is because the ground-truth of the eval set is not public. The only way to get the MRR@10 for the eval set is to submit your run to the MS MARCO leaderboard.