search

beir-cellar / beir

A Heterogeneous Benchmark for Information Retrieval. Easy to use, evaluate your models across 15+ diverse IR datasets.
http://beir.ai
Apache License 2.0
1.55k stars 186 forks source link

Fix sparse search #62

Closed maximedb closed 2 years ago

maximedb commented 2 years ago

Hi,

This pull request fixes two issues with the sparse search:

  1. Because of the step in trange, some queries are never evaluated (only the first one of each batch is evaluated) as reported in #60.
  2. Dense search skips some results (see here), this was not implemented in the sparse search. It impacts the results on ArguAna and Quora (#37)

Let me know if you have any questions.

Maxime.

thakur-nandan commented 2 years ago

Hi @maximedb and @jin530,

Thanks for the pull request. Indeed as mentioned in #60, this is an error that sadly I wasn't able to find earlier myself. Nevertheless, the scores for SPARTA reported in the BEIR paper are accurate. It used an old script that did not have the issue. I will soon have a look and merge the PR.

Kind Regards, Nandan Thakur