msmarco-bot
commented
2 years ago Hi there, thanks for your submission!
I've gone ahead and run the evaluation... here are your results:
# Processing submission 20220217-kolab
Private key found!
Unpacking submission 20220217-kolab
Decrypting key...
Decrypting metadata...
Decrypting submission tarball...
Unpacking tarball...
Done!
Submission directory ./submissions/20220217-kolab
Verified: submission directory exists!
Verified: expected files appear in the submission directory!
Sanity checking run ./submissions/20220217-kolab/dev.txt.bz2
Run has 6668967 lines, 6980 unique queries.
Warning, 6980000 lines expected (1000 hits per query), instead 6668967 lines found!
Sanity checking run ./submissions/20220217-kolab/eval.txt.bz2
Run has 6515736 lines, 6837 unique queries.
Warning, 6837000 lines expected (1000 hits per query), instead 6515736 lines found!
Proceeding to evaluate:
Dev run MRR@10: 0.3931489061718302
Eval run MRR@10: 0.3787587844849799
Hi there, please resubmit your results with the top 1000 hits; in your case, for reranking, since some queries have less than 1000 hits, rerank all the hits.