Add support for more languages

[mattico]

Each language needs a few components:

• stemmer - algorithm which reduces words to their root
• trimmer - removes all non-word characters
• stop word filter - removes unimportant words (if they the on then ...)

Right now we only have those for English. The stemmer and trimmer used to make the index should exactly match those used to search or there will be missing search results. elasticlunr.js uses https://github.com/weixsong/lunr-languages to support non-english languages. Since they used a JS version of snowball to generate their stemmers we can hopefully use bindings to the C version to make this easier. If not, at least the stemmers are made using simple functions unlike the English one which is a mess of regexes.

We'd also want an API for choosing a language. We could have pre-built pipelines for different languages. Also see how multi-language is done.

[Keats]

In terms of requirements I think having those 2 things is necessary:

• choosing a language by ISO 639-1 code
• return an enum of supported languages: this one is not super important but allows to check for content in unsupported language without trying to build the index and return an appropriate error.

The lunr.multilanguage approach sounds fine, I'm just wondering in the Gutenberg case whether when you are on the french version of the site you would want to see english results for example so being able to select a language on https://docs.rs/elasticlunr-rs/1.0.0/elasticlunr/struct.Index.html#method.new would be necessary imo so it would become:

let mut index = Index::new(&["title", "body"], Languages::French);
index.add_doc("1", &["C'est un titre", "C'est le contenu!"]);