TREC Deep Learning 2022

[seanmacavaney]

Dataset Information:

"The Deep Learning track focuses on IR tasks where a large training set is available, allowing us to compare a variety of retrieval approaches including deep neural networks and strong non-neural approaches, to see what works best in a large-data regime."

Links to Resources:

• https://microsoft.github.io/msmarco/TREC-Deep-Learning

Dataset ID(s) & supported entities:

• TBD, but probably under msmarco-passage-v2 and msmarco-document-v2

Checklist

Mark each task once completed. All should be checked prior to merging a new dataset.

• [ ] Dataset definition (in ir_datasets/datasets/[topid].py)
• [ ] Tests (in tests/integration/[topid].py)
• [ ] Metadata generated (using ir_datasets generate_metadata command, should appear in ir_datasets/etc/metadata.json)
• [ ] Documentation (in ir_datasets/etc/[topid].yaml)
  • [ ] Documentation generated in https://github.com/seanmacavaney/ir-datasets.com/
• [ ] Downloadable content (in ir_datasets/etc/downloads.json)
  • [ ] Download verification action (in .github/workflows/verify_downloads.yml). Only one needed per topid.
  • [ ] Any small public files from NIST (or other potentially troublesome files) mirrored in https://github.com/seanmacavaney/irds-mirror/. Mirrored status properly reflected in downloads.json.

Additional comments/concerns/ideas/etc.

Little information is available yet.

[cadurosar]

Hi Sean, we are integrating the splade repository with ir_datasets and one of the tests we want to add is TREC-DL22 (and potentially the queries from NEUCLIR23 and DL23). Anything that I can help in adding this data (as the issue already exists)?

[seanmacavaney]

Hey @cadurosar! Thanks for bumping this. It should be easy to add, and I'll try to within the next couple of days.

[seanmacavaney]

@cadurosar I've finished the updates for TREC DL 2022, including the /judged subsets, and pushed to pypi (version 0.5.5).

I'll try to get to NeuCLIR later today, but I have a busy schedule today so it may not get done until early next week.

[seanmacavaney]

An important note: the publicly-released qrels propagate the relevance labels to the duplicates, so that's what I use here. So the results you get when using them won't align with the numbers in the notebook papers.

[cadurosar]

@cadurosar I've finished the updates for TREC DL 2022, including the /judged subsets, and pushed to pypi (version 0.5.5).

I'll try to get to NeuCLIR later today, but I have a busy schedule today so it may not get done until early next week.

Thanks a lot Sean! I completely understand and if you find that you are too busy I can try adding them next week, just let me know