Open DeepaliP98 opened 4 months ago
When evaluating multiple query relevance (qrel) sets with pytrec_eval, incorrect scores are generated when one or more qrel sets are empty. For example:
qrel = { 'q1': { 'd1': 0, 'd2': 1, 'd3': 0, }, 'q2': { }, 'q3': { 'd2': 1, 'd3': 1, }, }
evaluator = pytrec_eval.RelevanceEvaluator(qrel, {'map', 'ndcg'}) evaluator.evaluate(some_valid_b)
In this case, the evaluator provides erroneous scores for 'q2' and relations after q2 i.e q3, without raising any warning, which could mislead users.
When evaluating multiple query relevance (qrel) sets with pytrec_eval, incorrect scores are generated when one or more qrel sets are empty. For example:
qrel = { 'q1': { 'd1': 0, 'd2': 1, 'd3': 0, }, 'q2': { }, 'q3': { 'd2': 1, 'd3': 1, }, }
evaluator = pytrec_eval.RelevanceEvaluator(qrel, {'map', 'ndcg'}) evaluator.evaluate(some_valid_b)
In this case, the evaluator provides erroneous scores for 'q2' and relations after q2 i.e q3, without raising any warning, which could mislead users.