ghost
commented
2 years ago
All CLA requirements met.
Closed autoliuweijie closed 2 years ago
ghost
commented
2 years ago
All CLA requirements met.
msmarco-bot
commented
2 years ago Hi there, this submission does accurately follow the submission instructions - doesn't contain the right files. Can you please fix? Thanks!
autoliuweijie
commented
2 years ago Hi, we have resubmitted our dev/eval files, is it right?
Thanks!
msmarco-bot
commented
2 years ago Hi there, thanks for your submission!
I've gone ahead and run the evaluation... here are your results:
# Processing submission 20220330-openplm_ltrbert
Private key found!
Unpacking submission 20220330-openplm_ltrbert
Decrypting key...
Decrypting metadata...
Decrypting submission tarball...
Unpacking tarball...
Done!
Submission directory ./submissions/20220330-openplm_ltrbert
Verified: submission directory exists!
Verified: expected files appear in the submission directory!
Sanity checking run ./submissions/20220330-openplm_ltrbert/dev.txt.bz2
Run has 6668967 lines, 6980 unique queries.
Warning, 6980000 lines expected (1000 hits per query), instead 6668967 lines found!
Sanity checking run ./submissions/20220330-openplm_ltrbert/eval.txt.bz2
Run has 6515736 lines, 6837 unique queries.
Warning, 6837000 lines expected (1000 hits per query), instead 6515736 lines found!
Proceeding to evaluate:
Dev run MRR@10: 0.366035498248967
Eval run MRR@10: 0.35649604973869947
This is our submission for the reranking task.