Submission 20220919-cotmae

[ghost]

All CLA requirements met.

[1024er]

@msmarco-bot

[msmarco-bot]

Hi @1024er - Sorry for the delay on this. I just looked at your submission, and it only has 200 hits per query. For consistency and fair comparisons to the other submission, we would like the submissions to have 1000 hits per query.

Thanks!

[ma787639046]

@msmarco-bot - Hi, I just submitted a new commit that holds 1000 hits per query. Would you please consider reevaluating our submission? Thanks so much.

[msmarco-bot]

Hi there, thanks for your submission!

I've gone ahead and run the evaluation... here are your results:

# Processing submission 20220919-cotmae

Private key found!
Unpacking submission 20220919-cotmae
Decrypting key...
Decrypting metadata...
Decrypting submission tarball...
Unpacking tarball...
Done!

Submission directory ./submissions/20220919-cotmae
Verified: submission directory exists!
Verified: expected files appear in the submission directory!

Sanity checking run ./submissions/20220919-cotmae/dev.txt.bz2
Run has 6980000 lines, 6980 unique queries.

Sanity checking run ./submissions/20220919-cotmae/eval.txt.bz2
Run has 6837000 lines, 6837 unique queries.

Proceeding to evaluate:

Dev run MRR@10: 0.45569529722108715
Eval run MRR@10: 0.43789163073937193

[ma787639046]

Hi, sorry to bother you again. The URLs for COT-MAE were redirected to the wrong sites by an unexpected URL prefix (https://microsoft.github.io/MSMARCO-Passage-Ranking-Submissions/leaderboard/). It seems that this prefix was added by your system, and we don't have it in our submission. Could you please fix them by removing the URL prefixes? Thanks again. (Change 'https://microsoft.github.io/MSMARCO-Passage-Ranking-Submissions/leaderboard/arxiv.org/abs/2208.07670' to 'arxiv.org/abs/2208.07670'; 'https://microsoft.github.io/MSMARCO-Passage-Ranking-Submissions/leaderboard/github.com/caskcsg/CoT-MAE' to 'github.com/caskcsg/CoT-MAE')

[ma787639046]

@msmarco-bot

[msmarco-bot]

Ah, the issue is that you were missing the http:// prefix, so our system interpreted it as a relative URL.

As a matter of policy, we don't modify submission ourselves.

Can you please change the metadata to the full URL, e.g.,http://arxiv.org/abs/2208.07670 - and send a PR with an updated metadata file. You'll need to make sure you use the same encryption key.

[ma787639046]

@msmarco-bot - Sorry to bother you again. I didn't reserve my encryption key file ''20220919-cotmae.key.bin". Could you please send this file to my email? My email is ma787639046@outlook.com, thank you very much!