search

allenai / ir_datasets

Provides a common interface to many IR ranking datasets.
https://ir-datasets.com/
Apache License 2.0
316 stars 42 forks source link

BEIR cqadupstack #221

Closed jobergum closed 1 year ago

jobergum commented 1 year ago

I'm wondering about the BEIR https://ir-datasets.com/beir.html#beir/cqadupstack/* versus https://github.com/beir-cellar/beir dataset? Is there one ir dataset that has all the docs, queries, and qrels for cqadupstack ?

seanmacavaney commented 1 year ago

Hi @jobergum -- cqadupstack works differently than the other datasets in BEIR. Their convention for the dataset is to evaluate on each of the domains independently and average the nDCG@10 score for the final cqadupstack score.

From an ir-datasets perspective, that's 12 separate datasets, each with its own corpus, queries, and qrels. A concatenated cqadupstack dataset might be useful for some experiments, but it doesn't align with their task definition. As such, I'd be hesitant to add it, since it could be confused with BEIR's official way of reporting cqadupstack.

There are more details in this issue: https://github.com/beir-cellar/beir/issues/9#issuecomment-842147129

Hope this helps!

jobergum commented 1 year ago

Wow, thanks for the quick reply @seanmacavaney! I did not realize that it was 12 datasets, then I think it makes a lot of sense. Closing this!