Evaluator does not detect when a run retrieves too few topics when no qrels are given.

tira-io / ir-experiment-platform

28 stars 3 forks source link

Evaluator does not detect when a run retrieves too few topics when no qrels are given. #2

Open mam10eks opened 1 year ago

mam10eks commented 1 year ago

We found the following case that is not handled well: a system retrieves results for only 1 topic while there are 50 topics, and the evaluation script still says that the run is valid.