[Feature Request] relevance_level parameter

[AIIRLab]

I was wondering if similar to trec_eval that we can specify the relevance_level, with -l parameter, this feature exists in ranx. If not that would be a useful feature for evaluation

[AmenRa]

Dear Behrooz,

Thanks again for your interest in ranx! :)

I added the requested feature in v0.3.3. I added two different ways to set relevance levels.

Qrels-wise Qrels now provides the set_relevance_level function: qrels.set_relevance_level(rel_lvl). It changes the document relevance judgments stored in qrels.

Metric-wise The relevance level can now be appended to metric names: map@100-l2, ndcg-l3. This way, the original relevance judgments stored in qrels are preserved.

I checked some metric scores against trec_eval after setting the relevance level: they seem to work as intended. I noticed that setting the relevance level for ndcg does not work on trec_eval. I will open an issue on its repo to check whether this works as intended.

Let me know if you have further feature requests.

Best,

Elias