Closed mpetri closed 1 year ago
Hi, thanks for pointing it out.
I suspect I assumed it was a binary relevance metric because the example in the original paper does not account for graded relevance.
I will fix it in the coming days.
Looking at other implementations of RBP I also saw an option to normalize back to [0,1] by dividing by the max relevance level in the qrls.
So if max relevance level RBP in the limit goes to 2. So we divide by 2 to make it go back to [0,1]. Not sure if this is useful to implement or not.
Fixed in v0.3.15
.
RBP
now uses graded relevance scores from qrels
instead considering them binary.
Do you think I should leave an option for the old behaviour?
Describe the bug
Given the formula of RBP of
where r_i is the reward/utility, RBP should support multiple relevance levels similar to DCG such that if max relevance level is 2 max rbp value should be 2