fix issue 39: Score is 0 when a token is in exactly 50% of the documents.

dorianbrown / rank_bm25

A Collection of BM25 Algorithms in Python

Apache License 2.0

983 stars 83 forks source link

fix issue 39: Score is 0 when a token is in exactly 50% of the documents. #40

Open danerlt opened 3 months ago

danerlt commented 3 months ago

@dorianbrown Can you help me review the code, please? Thank you.

dorianbrown commented 3 months ago

@danerlt Sure thing, I'll have time to take a look sometime in the next week. Looking forward to it 👍

dorianbrown commented 3 months ago

I think this is a problem that needs addressing, and from what I can tell there are two ways to go (see this PR for some of the thoughts I just shared), so either fix the epsilon approach and keep the IDF as is, or remove that negativity checking, and go for this "smoothed" IDF function.

I'm not working on this topic as intensively as before, but do you know if there's a reason to prefer one of the two from a ranking perspective? Computationally the changed IDF function seems preferable.