lucaong
commented
2 years ago Hi @akvadrako , I integrated stemmers in custom applications using MiniSearch in the past, I do not recall the specific one but I can look it up.
That said, I want to share a word of caution: stemmers can help increasing recall, but they often make results confusing for the user, so I found myself relying more and more on fuzzy match instead. Here’s a few reasons why:
- As you can see in several issues opened on Lunr (this one for example), search results when using a stemmers are often confusing.
- Relevance calculation is also negatively affected: if
lookinggets stemmed tolook, searching forlookwould give equal relevance (all else being equal) to documents containing the termlookorlooking. Fuzzy search would instead assign more relevance to the exact match. - Stemming does not combine well with prefix search: the confusing result would be that searching (prefix search) for
lo,loo,lookwould find documents containinglooking, but searching forlooki,lookinwould find no result, thenlookingwould again find results (because it gets correctly stemmed). - Stemmers are language dependent, but many documents mix languages (like mixing English terms of common use in text written in other languages)
- Fuzzy match covers many of the cases handled by stemming, and ranks results better by edit distance. Moreover, the
matchfield in the results contains the actual terms in the document, which is not the case with stemming.
Of course there are still legitimate use cases for stemming, but I felt it was useful to share my personal experience. I ended up replacing stemming with fuzzy match in more than a few applications in recent years, with better user experience.
I'm looking at using minisearch in place of lunr, but one requirement I have is language support, specifically for English and Dutch.
I see there are a few npm packages with stemmers, stop words, etc. Has anyone had success integrating them with minisearch?