search

MihaiValentin / lunr-languages

A collection of languages stemmers and stopwords for Lunr Javascript library
Other
432 stars 161 forks source link

Idea: Use Intl.Segmenter to reduce bundle size in browsers #90

Open knubie opened 2 years ago

knubie commented 2 years ago

Right now lunr-languages uses the default white-space-based stemmer for most languages, and handful of custom stemers for other languages (th, ja, zh). Additionally, the zh stemmer is only available in Node.js because it uses C++ extensions.

We could reduce bundle size, and make zh usable in the browser by switching to Intl.Segmenter for all languages.

The downside to this approach is that Firefox is the only major browser that has not implemented the Stemmer API, and we would also lose support for Node.js, which would be a deal breaker.

I have a working proof of concept here where I use the stemmer specifically for zh to gain the ability to use it in browser.