search

usnistgov / trec_eval

Evaluation software used in the Text Retrieval Conference
232 stars 49 forks source link

ndcg different from ndcg_cut_20 for 20 results ? #11

Open gaurav5430 opened 7 years ago

gaurav5430 commented 7 years ago

I have not set the relevance gains level for ndcg from the command line. So, everything is at default, and i have 20 results per query, then why is ndcg_cut_20 different from ndcg ? (although i can see that they are same after ndcg_cut_100) , am i missing something?

gaurav5430 commented 7 years ago

It seems that for multiple queries, it is taking the maximum of num_rel across these queries and then calculating ndcg@max(num_rel) for all of them.

For example :

Q1 has 20 num_ret and 30 num_rel Q2 has 20 num_ret and 35 num_rel Q3 has 20 num_ret and 28 num_rel

then ndcg returns me a score which is same as ndcg_cut_35 while i expect it to give me ndcg@20, which is the number of results for each query.

gaurav5430 commented 7 years ago

for Q1 : ndcg = ndcg_cut_30 Q2 : ndcg = ndcg_cut_35 Q3 : ndcg = ndcg_cut_28

for all ndcg = ndcg_cut_35