search

beir-cellar / beir

A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.
http://beir.ai
Apache License 2.0
1.55k stars 186 forks source link

End-Task Labels #73

Closed anthonywchen closed 2 years ago

anthonywchen commented 2 years ago

Thank you for the great repository!

I noticed that for datasets where there is an end-task (e.g. classification for QQP or QA for Natural Questions), that the data files do not have these end-task annotations (e.g. the answer for NQ instances). Is there any easy way to recover these end-task labels for the different datasets?

thakur-nandan commented 2 years ago

Hi @anthonywchen,

Sadly BEIR was focused on IR, hence we do not have these end-task annotations. For a few datasets, we might have some additional detail on their end-task in the metadata fields. However, I believe this is not present for NQ.

Kind Regards, Nandan Thakur

anthonywchen commented 2 years ago

Thank you for the help! closing the issue.