search

cvangysel / pytrec_eval

pytrec_eval is an Information Retrieval evaluation tool for Python, based on the popular trec_eval.
http://ilps.science.uva.nl/
MIT License
282 stars 32 forks source link

Qrel Relevance scores between 1 to 5 #4

Closed amirj closed 5 years ago

amirj commented 5 years ago

Is it right to set qrel relevance scores to sometimg between 1 to 5? 1 means less relevant and 5 means high relevant. I’m going to use ndcg measure

cvangysel commented 5 years ago

Yep, that's generally how it works.

The value of the relevance level gets used in NDCG as the input to an exponential function (https://en.wikipedia.org/wiki/Discounted_cumulative_gain#Normalized_DCG). Be sure to take that into account when designing your experiments.