question: relevance scores in Qrels

[PaulLerner]

Why is the relevance score mandatory in Qrels?

I don’t see where you use it to compute the metrics https://github.com/AmenRa/ranx/blob/master/ranx/metrics.py

Is there any way to make it optional? If not, would a filler value like 0 for all documents be appropriate?

[AmenRa]

Hi, if you only have a list of relevant document ids but no relevance judgment, you should use 1 as relevance judgment score.

0 relevance judgment scores are removed before computing metric scores to prevent miscomputations.

Among the implemented metrics only NDCG uses the relevance judgment scores. You can see it here.

By definition, all the other metrics consider relevance judgment scores to be binary even if they are not.

[PaulLerner]

Got it, thanks !