Submission 20220112-BAIDU

microsoft / MSMARCO-Passage-Ranking-Submissions

Submission archive for the MS MARCO passage ranking leaderboard

https://microsoft.github.io/MSMARCO-Passage-Ranking-Submissions/leaderboard

MIT License

12 stars 30 forks source link

Submission 20220112-BAIDU #28

Closed Deriq-Qian-Dong closed 2 years ago

ghost commented 2 years ago

All CLA requirements met.

msmarco-bot commented 2 years ago

Hi there, I'm getting the follow warnings:

Sanity checking run ./submissions/20220112-BAIDU/dev.txt.bz2
Run has 69628 lines, 6980 unique queries.
Warning, 6980000 lines expected (1000 hits per query), instead 69628 lines found!

Sanity checking run ./submissions/20220112-BAIDU/eval.txt.bz2
Run has 68203 lines, 6837 unique queries.
Warning, 6837000 lines expected (1000 hits per query), instead 68203 lines found!

We would like submissions with 1000 hits per query... would you mind please fixing? Thanks!

Deriq-Qian-Dong commented 2 years ago

Hi, Top-10 is enough for MRR@10. Is this necessary for 1000 passages per query?This guidance notes that only the top 10 results could be accepted.

msmarco-bot commented 2 years ago

Yes, the official evaluation metric is MRR@10, so only top 10 hits are necessary for computing the metric. However, we (as organizers) perform additional meta-evaluations (for example, recall-based analyses) that require top 1000 hits.

Thanks for pointing out the link - that guidance appears to be outdated. We'll fix.

Deriq-Qian-Dong commented 2 years ago

Thank you for your answer~ Additionally, the official BM25 just retrieves 6668967 and 6515736 on DEV and EVAL query set respectively. Therefore, the reranking task cannot meet the requirement of "top 1000" based on the official BM25 retrieval results.

msmarco-bot commented 2 years ago

Correct, if you do reranking, please rerank the entire BM25 results.

Deriq-Qian-Dong commented 2 years ago

Hi there, I'm getting the follow warnings:

Sanity checking run ./submissions/20220112-BAIDU/dev.txt.bz2
Run has 69628 lines, 6980 unique queries.
Warning, 6980000 lines expected (1000 hits per query), instead 69628 lines found!

Sanity checking run ./submissions/20220112-BAIDU/eval.txt.bz2
Run has 68203 lines, 6837 unique queries.
Warning, 6837000 lines expected (1000 hits per query), instead 68203 lines found!

We would like submissions with 1000 hits per query... would you mind please fixing? Thanks!

This has been fixed.

msmarco-bot commented 2 years ago

Hi there, thanks for your submission!

I've gone ahead and run the evaluation... here are your results:

# Processing submission 20220112-BAIDU

Private key found!
Unpacking submission 20220112-BAIDU
Decrypting key...
Decrypting metadata...
Decrypting submission tarball...
Unpacking tarball...
Done!

Submission directory ./submissions/20220112-BAIDU
Verified: submission directory exists!
Verified: expected files appear in the submission directory!

Sanity checking run ./submissions/20220112-BAIDU/dev.txt.bz2
Run has 6668967 lines, 6980 unique queries.
Warning, 6980000 lines expected (1000 hits per query), instead 6668967 lines found!

Sanity checking run ./submissions/20220112-BAIDU/eval.txt.bz2
Run has 6515736 lines, 6837 unique queries.
Warning, 6837000 lines expected (1000 hits per query), instead 6515736 lines found!

Proceeding to evaluate:

Dev run MRR@10: 0.3939803065447765
Eval run MRR@10: 0.37732064792178943