I feel pretty strongly that TREC's -c mode (calculate over the set of queries in the qrels, not the intersection of the qrels and runs) should be the default. The current calc_aggregate operates without this mode.
Does this need to be conditional? If so, where? As an argument of the measure or of calc_aggregate?
I think I'd rather have it non-conditional. If you want to score over a different set of queries, you should provide qrels without those queries to score.
cmacdonald
commented
3 years ago
I feel pretty strongly that TREC's
-cmode (calculate over the set of queries in the qrels, not the intersection of the qrels and runs) should be the default. The current calc_aggregate operates without this mode.Does this need to be conditional? If so, where? As an argument of the measure or of calc_aggregate?
I think I'd rather have it non-conditional. If you want to score over a different set of queries, you should provide qrels without those queries to score.