Custom k for cut metrics

cvangysel / pytrec_eval

pytrec_eval is an Information Retrieval evaluation tool for Python, based on the popular trec_eval.

http://ilps.science.uva.nl/

MIT License

282 stars 32 forks source link

Custom k for cut metrics #12

Open ghost opened 5 years ago

ghost commented 5 years ago

Hi,

Is there any way to specify custom k values for map_cut, ndcg_cut etc., instead of default ones ?

It is definitely possible with trec_eval, but I don't see how to do it with the current interface (or do I miss something ?)

Thanks

Thibault

seanmacavaney commented 4 years ago

@tformal I added support for custom thresholds using the same convention as in trec_eval. E.g., {'ndcg_cut.1,15,100'} will give ndcg_cut_1, ndcg_cut_15, and ndcg_cut_100 results.

If this sounds good, I'll go ahead and make a PR.

ghost commented 4 years ago

@seanmacavaney this sounds perfect !

seanmacavaney commented 4 years ago

Do we also want to support a more natural way of requesting specific thresholds? I.e., {'ndcg_cut_1', 'ndcg_cut_15', 'ndcg_cut_100'} instead of {'ndcg_cut.1,15,100'}? This could be done in a python wrapper that converts the former into the latter.

Inspired by https://twitter.com/lintool/status/1232311794116501504