Submission 20220302-zbertv2

[ghost]

All CLA requirements met.

[msmarco-bot]

Hi there, we request that you submit 1000 his per query - can you please update your submission accordingly? Thanks!

[nullzyz]

Hi there, we request that you submit 1000 his per query - can you please update your submission accordingly? Thanks!

Okay, I have updated the top1000 results now

[msmarco-bot]

Hi there, I'm having issues processing your files:

This file should not be part of the PR: submissions/20220302-zbertv2.key.bin

Can you check to see if you've packed the files correctly per our instructions?

[nullzyz]

Hi there, I'm having issues processing your files:

This file should not be part of the PR: submissions/20220302-zbertv2.key.bin

Can you check to see if you've packed the files correctly per our instructions?

Sorry for that, I have deleted it now.

[msmarco-bot]

Per submission guidelines: https://github.com/microsoft/MSMARCO-Passage-Ranking-Submissions#additional-submission-guidelines

It is generally expected that the team description in the metadata file will include the name of the organization (e.g., university or company). In many cases, submissions explicitly list the contributors of the run. It is not permissible to submit a run under an alias (or a generic, nondescript team) to first determine "how you did", and then ask for a metadata change only after you've been shown to "do well". We will reject metadata change requests in these circumstances. Thus, you're advised to make the team description as specific as possible, so that you can claim "credit" for doing well. We further request that your team description unambiguously identify who you are (for example, your identify should be fairly clear given a web search). Submissions with metadata containing ambiguous team identifies may be rejected.

We would request an organizational affiliation associated with your run. Kindly adjust the metadata accordingly please.

[nullzyz]

Per submission guidelines: https://github.com/microsoft/MSMARCO-Passage-Ranking-Submissions#additional-submission-guidelines

It is generally expected that the team description in the metadata file will include the name of the organization (e.g., university or company). In many cases, submissions explicitly list the contributors of the run. It is not permissible to submit a run under an alias (or a generic, nondescript team) to first determine "how you did", and then ask for a metadata change only after you've been shown to "do well". We will reject metadata change requests in these circumstances. Thus, you're advised to make the team description as specific as possible, so that you can claim "credit" for doing well. We further request that your team description unambiguously identify who you are (for example, your identify should be fairly clear given a web search). Submissions with metadata containing ambiguous team identifies may be rejected.

We would request an organizational affiliation associated with your run. Kindly adjust the metadata accordingly please.

Thanks for the reminder, we have now updated the team information now.

[msmarco-bot]

Hi there, thanks for your submission!

I've gone ahead and run the evaluation... here are your results:

# Processing submission 20220302-zbertv2

Private key found!
Unpacking submission 20220302-zbertv2
Decrypting key...
Decrypting metadata...
Decrypting submission tarball...
Unpacking tarball...
Done!

Submission directory ./submissions/20220302-zbertv2
Verified: submission directory exists!
Verified: expected files appear in the submission directory!

Sanity checking run ./submissions/20220302-zbertv2/dev.txt.bz2
Run has 6980000 lines, 6980 unique queries.

Sanity checking run ./submissions/20220302-zbertv2/eval.txt.bz2
Run has 6837000 lines, 6837 unique queries.

Proceeding to evaluate:

Dev run MRR@10: 0.452815299949972
Eval run MRR@10: 0.43917886105249054