search

beir-cellar / beir

A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.
http://beir.ai
Apache License 2.0
1.54k stars 182 forks source link

Question about MS MARCO dataset in BEIR #80

Closed jordane95 closed 2 years ago

jordane95 commented 2 years ago

Hello,

I see that you processed the raw MS MARCO dataset to include it in your unified BEIR-style dataset. After downloading and checking it, I have several questions.

  1. Where does the data in test.tsv come from? Given that the test set of MS MARCO passage retrieval task is not publicly available.
  2. What does the scores in test.tsv stand for? I see there is only score 1 in train.tsv and dev.tsv, while there are 0, 1, 2, 3 in test.tsv.
thakur-nandan commented 2 years ago

Hi @jordane95, thank you for your questions!

  1. The data in the test.tsv comes from the passage ranking task in TREC-DL 2019 (https://microsoft.github.io/msmarco/TREC-Deep-Learning-2019.html#passage-ranking-dataset).
  2. Yes, the scores in train.tsv only will be binary and 1 as we only care about relevant passages (i.e. with score = 1) while training. However in evaluation for a few datasets, we may have multiple levels of relevancy, i.e. a score of 3 denotes a higher similarity between the query and passage in contrast to a score of 1.

Hope it helps!

Kind Regards, Nandan Thakur

jordane95 commented 2 years ago

Thank you for your answer! That helps a lot!