question about calculating NDCG score

[Eku0194]

Your code shows : class NDCG(Metric): def init(self, k=10, gain_type='exp2'): super(NDCG, self).init() self.k = k self.gain_type = gain_type self._dcg = DCG(k=k, gain_type=gain_type) self._ideals = {}

def evaluate(self, qid, targets):
    return (self._dcg.evaluate(qid, targets) /
            max(_EPS, self._get_ideal(qid, targets)))

Question:

1) Is evaluate method used to calculate the NDCG score of the model? If yes, then what is the format of targets ? Is it the array of labels for each query-document pair? If No, then how do we calculate NDCG score of the selected model?

[jma127]

Hello! targets is an ordered list of relevance scores, where targets[0] is the top of the result set. And the evaluate() method is used to calculate the NDCG score of each individual result set. You could sum and divide by n to get the average NDCG across the dataset.

[Eku0194]

Thank you, Jerry. So, the relevance score is the result of the ranking function? In order to calculate average NDCG for multiple queries at once - do I have to sum all the NDCG score of each individual result set and then divide by n or I need to calculate average NDCG per query_id and then divide it by the total no. of queries?

[jma127]

There's a one-to-one correspondence between a query and a result set (i.e. if n denotes the number of result sets, then n is equal to the total number of queries).

[Eku0194]

Thank you so much, Jerry!