I noticed there was a problem in the way we handled queries that exist in the retrieval corpus. By default we have ignore_identical_ids=True which pops these duplicated queries from the results. Which means some queries would have top_k retrieved documents, while others have top_k-1 retrieved documents.
Fixing this behaviour gives a noticeable change in scores. Here's the difference in scores noticed for "intfloat/e5-large" on ArguAna evaluated using MTEB:
I noticed there was a problem in the way we handled queries that exist in the retrieval corpus. By default we have
ignore_identical_ids=True
which pops these duplicated queries from theresults
. Which means some queries would havetop_k
retrieved documents, while others havetop_k-1
retrieved documents.Fixing this behaviour gives a noticeable change in scores. Here's the difference in scores noticed for
"intfloat/e5-large"
on ArguAna evaluated using MTEB:Scores before fix:
Scores after fix:
cc @thakur-nandan