lucaong / minisearch

Tiny and powerful JavaScript full-text search engine for browser and Node
https://lucaong.github.io/minisearch/
MIT License
4.81k stars 137 forks source link

Support BM25 parameters customization #163

Closed rayhsieh closed 1 year ago

rayhsieh commented 2 years ago

Would you consider to support customization of BM25 parameters? It would be very helpful for optimizing search relevance.

var k = 1.2; // Term frequency saturation point. Recommended values are between 1.2 and 2.
var b = 1.2; // Length normalization impact. Recommended values are around 0.75.
var d = 0.5; // BM25+ frequency normalization lower bound. Recommended values are between 0.5 and 1.
lucaong commented 2 years ago

Hi @rayhsieh , it would definitely be possible to support customization of the BM25 parameters, and it might be offered as a feature in a upcoming release. That said, beware that tweaking these parameters tends to be tricky, and should almost never be necessary. Users needing to fine-tune results can usually rely on boosting, which has a more predictable effect.

See the relevant discussion, where @rolftimmermans correctly notes:

I agree, but to be honest most users should not attempt to tune this. The effect is fairly subtle and requires a thorough understanding of the BM25 scoring model. Having such a knob, people will probably want to tune it. While I think in practice a field boost is a much easier way to tune a custom dataset, if the underlying scoring model is solid enough (which I hope BM25 is).

That said, there is no hard technical reason why this cannot be done. It just would need to be properly documented, also explaining that these kind of tweaks are normally discouraged. I tend to prefer keeping the API surface as small as it can be, but if there is a use case for this, it can be done.

rayhsieh commented 2 years ago

Hi @lucaong , I totally agree that tweaking these parameters is tricky, and in most cases not necessary. But in my experience, the default parameters might not be the best choice when dealing with special dataset. I'm actually working on a limited Chinese dataset and I figured out that raising the impact of the length of terms by tweaking b do make the search experience way much better than the default set.

lucaong commented 2 years ago

That sounds like an interesting case. I personally would like to know more about how MiniSearch performs on non-latin scripts, and I am very interested in hearing feedback on what could be added or changed to improve the experience in those cases. MiniSearch follows the principle of not including any language-specific utility (such as stemming or normalization), but making it easily possible to plug those in whenever necessary.

I think I can soon prepare an update that makes BM25 parameters configurable, possibly initially as a beta feature.

rolftimmermans commented 2 years ago

Here are some of my notes that may help in documenting the parameters, if they're exposed. May need a bit of a rewrite :)

This article is also helpful for understanding k and b: https://opensourceconnections.com/blog/2015/10/16/bm25-the-next-generation-of-lucene-relevation/

k is the BM25 term frequency saturation point.

b is the BM25 length normalization impact.

d (actually δ) is the BM25+ frequency normalization lower bound.

rayhsieh commented 2 years ago

@lucaong Thank you for planning this request. While working on the search function on my dataset, it is very flexible for me the add language-specific tokenizer. I don't have other recommendation at this point since it is already met what I need.

lucaong commented 1 year ago

@rayhsieh this feature is now released along other features as part of v6.0.0-beta.1 (soon to be released as v6.0.0).