search

dlukes / rbo

Python implementation of the rank-biased overlap list similarity measure.
70 stars 17 forks source link

RBOext greater than one #4

Open vpodpecan opened 4 years ago

vpodpecan commented 4 years ago

Looks like there are some issues with your compuation of RBO. For example:

a = [1, 2, 3]
b = [1]
p = 0.4
print(rbo(a, b, p))

returns

RBO(min=0.766238435648986, res=0.24568078217550698, ext=1.0453333333333334)

Here, RBOext is > 1 which is not correct.

Also, RBOmin crashes when p=1.0 because of math.log(1 - p)

dlukes commented 4 years ago

Thanks for taking the time to report this! I'll try and figure out what's happening if/when I have a bit of free time :)

RBOext is > 1 which is not correct.

As mentioned in the README, I somewhat deviated from the definition in the paper precisely because I observed it yielded RBO.ext > 1 in some cases. However, with your test case, the original version correctly yields 1.0, while mine yields 1.045, as you've observed, which is ironic, since this is what my modifications were meant to fix.

If you test the example in the README, then it's the other way round: the master branch yields RBO.ext < 1, and the original-algorithm branch yields RBO.ext > 1.

Not quite sure what's happening here, it's been a while since I read the paper and implemented this, so I may be misremembering. I do remember that some parts of the specification of the RBO algorithm were not entirely clear to me -- it's possible they were ambiguously worded, or maybe I just misunderstood them when translating into code. If you have time to look into the code and provide some additional insights, I'd love to hear them!

RBOmin crashes when p=1.0 because of math.log(1 - p)

Yep :) I don't think RBO makes sense for p = 1.0, or does it? Or maybe you meant that the function should perform bounds checking and throw a more helpful error than ValueError: math domain error?