Submission 20211020-MPB

[yiyaxiaozhi]

This submission for ms marco passage reranking

[msmarco-bot]

Hi there, thanks for your submission!

Unfortunately, your submission currently does not conform to our submission guidelines. The team name "SCNU_FOUR" is not sufficiently descriptive - see additional details at: https://github.com/microsoft/MSMARCO-Document-Ranking-Submissions/#metadata-updates

Please include a clearly identifiable group/organization or the names of the individuals responsible for the submission; ideally, both.

(Sorry, we have yet to update the submission instructions in this repo, but our guidelines are the same across doc and passage submissions).

Can you please fix?

[msmarco-bot]

hi @yiyaxiaozhi We're having trouble reading your submission. The pack script generates a new key each time you run it, so the old key is unable to decrypt the new JSON. This issue can be fixed by repacking the entire submission again. Can you give that a try?

[msmarco-bot]

Hi there, thanks for your submission!

I've gone ahead and run the evaluation... here are your results:

# Processing submission 20211020-MPB

Private key found!
Unpacking submission 20211020-MPB
Decrypting key...
Decrypting metadata...
Decrypting submission tarball...
Unpacking tarball...
Done!

Submission directory ./submissions/20211020-MPB
Verified: submission directory exists!
Verified: expected files appear in the submission directory!

Sanity checking run ./submissions/20211020-MPB/dev.txt.bz2
Run has 6974598 lines, 6980 unique queries.
Warning, 6980000 lines expected (1000 hits per query), instead 6974598 lines found!

Sanity checking run ./submissions/20211020-MPB/eval.txt.bz2
Run has 6832196 lines, 6837 unique queries.
Warning, 6837000 lines expected (1000 hits per query), instead 6832196 lines found!

Proceeding to evaluate:

Dev run MRR@10: 0.3621322258607354
Eval run MRR@10: 0.35480601581961824