Availability of previous NIST Labels

[Muennighoff]

Hi,

I was wondering if the NIST Labels from the previous years are available? / Is it still possible to evaluate models on those labels?

Also I was wondering what those labels are? (Are they manual rankings (1, 2, 3, ..) of e.g. the top 10 documents? Were the documents for labelling pre-selected or did labellers go through the entire document corpus?)

Thanks for the great work! :)

[bmitra-msft]

Yes, the NIST labels are linked under the dataset section on the following pages:

• https://microsoft.github.io/msmarco/TREC-Deep-Learning-2019
• https://microsoft.github.io/msmarco/TREC-Deep-Learning-2020

You can refer to the TREC DL Overview papers for details of the evaluation:

• https://arxiv.org/pdf/2003.07820.pdf
• https://arxiv.org/pdf/2102.07662.pdf

Best regards, Bhaskar

[Muennighoff]

Yes, the NIST labels are linked under the dataset section on the following pages:

• https://microsoft.github.io/msmarco/TREC-Deep-Learning-2019
• https://microsoft.github.io/msmarco/TREC-Deep-Learning-2020

You can refer to the TREC DL Overview papers for details of the evaluation:

• https://arxiv.org/pdf/2003.07820.pdf
• https://arxiv.org/pdf/2102.07662.pdf

Best regards, Bhaskar

Awesome, thanks. For 2021, the NIST labels for passage ranking havn't been made available yet, right?

[bmitra-msft]

No, not yet but it should be a few months down the line.

Best regards, Bhaskar