Add test dataset for trec tip of the tongue dataset

allenai / ir_datasets

Provides a common interface to many IR ranking datasets.

https://ir-datasets.com/

Apache License 2.0

306 stars 40 forks source link

Add test dataset for trec tip of the tongue dataset #249

Open mam10eks opened 10 months ago

mam10eks commented 10 months ago

The test queries for the trec tot task are now available, and this pull request adds them in a similar way to how they are added for dev and train (no qrels are available yet, we can also wait to merge this until the qrels are available).

seanmacavaney commented 10 months ago

Thanks! A few points:

Can we rename it to trec-tot/2023/test? This is more aligned with the typical naming convention.
Are the sentence_annotations of a standard structure? If so, they should be represented as NamedTuples instead of dicts.
Can you add a metadata record and documentation for this dataset? (This is why the tests failed.)