Closed iknoorjobs closed 2 years ago
Hi @iknoorjobs,
From my understanding, the evaluation should be based on the length of qrels
.
For example, for a few queries in lexical search with Elasticsearch, each query is unable to return any top-k documents. In such cases, the results
dictionary will not include this query. Here we must make sure the retriever is penalized by assigning MRR@k score equal to 0 for these queries present in qrels.
We always take qrels
to be the ground truth and do evaluations based on it.
Kind Regards, Nandan Thakur
Hi @thakur-nandan,
Thanks for your response. Yes, that makes sense. In this case, I'm using a custom dataset for evaluation and my qrels
are combined for the test and train set queries, that's probably what's causing the wrong results.
Regards,
Iknoor
Hi @thakur-nandan, Great work! Keep it up! 🍻 Just noticed one thing in the code for the custom metric (here), the sum of MRR@k scores is divided by len(qrels). However, this should actually be divided by the number of queries for which MRR is computed (in your code this can be len(results)). Please let me know if I'm wrong. If not, I am happy to create a PR :)
Best, Iknoor Singh