search

usnistgov / trec_eval

Evaluation software used in the Text Retrieval Conference
232 stars 49 forks source link

Exposes DCG and ideal DCG (constituents of NDCG) #1

Closed cvangysel closed 8 years ago

cvangysel commented 8 years ago

Analysis of the constituents of NDCG on a per-query basis can give insight in for which particular queries retrieval can be improved, or if comparing across methods, which method performs better for queries with high or low ideal DCG.

isoboroff commented 8 years ago

Can you give a good example of why we want to expose the NDCG components? We don't do this for any other measure.

cvangysel commented 8 years ago

It is useful to investigate whether a method performs well for topics with high or low ideal DCG scores (i.e. whether lots of positive relevance information is available or not).

A good example can be found in the paper "Dynamic Query Modeling for Related Content Finding" by Odijk et al. published at SIGIR 2015 (see Figure 5): https://staff.fnwi.uva.nl/m.derijke/wp-content/papercite-data/pdf/odijk-dynamic-2015.pdf

Personally I find these measures useful for similar reasons, although these results haven't been published.

isoboroff commented 8 years ago

I created a new version-10.0-dev branch. Could you re-do your pull request against that branch, and I'll pull it in there?

cvangysel commented 8 years ago

Done, https://github.com/usnistgov/trec_eval/pull/3