search

cvangysel / pytrec_eval

pytrec_eval is an Information Retrieval evaluation tool for Python, based on the popular trec_eval.
http://ilps.science.uva.nl/
MIT License
282 stars 32 forks source link

How to handle duplicate document IDs? #50

Open mrdrozdov opened 3 months ago

mrdrozdov commented 3 months ago

What if I am predicting a ranked list with the same document ID multiple times in different positions. How can I evaluate nDCG for this using pytrec_eval, given that scores are represented as dictionaries?

seanmacavaney commented 3 months ago

Hey @mrdrozdov -- trec_eval itself checks for duplicate documents and raises an error if it finds any. So I'm not sure diverging from this behavior in the python wrapper would make sense.

Even so, many measures are not well-defined in the presence of duplicate documents. E.g., you could get an ndcg score > 1 when duplicates are present. So you'd have to think carefully about what measures are potentially suitable.