olivernn / lunr.js

A bit like Solr, but much smaller and not as bright
http://lunrjs.com
MIT License
8.87k stars 547 forks source link

ValueError: The language 'arabic' is not supported. (I am not using arabic!) #473

Closed ronnydw closed 3 years ago

ronnydw commented 3 years ago

I get the following error in a mkdocs project. The documents are in Dutch, there is no arabic in the text anywhere. So this error seems strange. I also created an issue in mkdocs (https://github.com/mkdocs/mkdocs/issues/2149), but the the problem seems to be in lunr Are there any characters that are not supported, ie €, | , @ ...?

INFO - Building documentation... Traceback (most recent call last): File "/Users/ronny/anaconda/bin/mkdocs", line 8, in sys.exit(cli()) File "/Users/ronny/anaconda/lib/python3.6/site-packages/click/core.py", line 829, in call return self.main(args, kwargs) File "/Users/ronny/anaconda/lib/python3.6/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/Users/ronny/anaconda/lib/python3.6/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/ronny/anaconda/lib/python3.6/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "/Users/ronny/anaconda/lib/python3.6/site-packages/click/core.py", line 610, in invoke return callback(args, kwargs) File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/main.py", line 136, in serve_command kwargs File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/commands/serve.py", line 141, in serve config = builder() File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/commands/serve.py", line 129, in builder **kwargs File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/config/base.py", line 197, in load_config errors, warnings = cfg.validate() File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/config/base.py", line 107, in validate run_failed, run_warnings = self._validate() File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/config/base.py", line 62, in _validate self[key] = config_option.validate(value) File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/config/config_options.py", line 131, in validate return self.run_validation(value) File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/config/config_options.py", line 609, in run_validation plgins[item] = self.load_plugin(item, cfg) File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/config/config_options.py", line 617, in load_plugin Plugin = self.installed_plugins[name].load() File "/Users/ronny/anaconda/lib/python3.6/site-packages/pkg_resources/init.py", line 2341, in load return self.resolve() File "/Users/ronny/anaconda/lib/python3.6/site-packages/pkg_resources/init.py", line 2347, in resolve module = import(self.module_name, fromlist=['name'], level=0) File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/contrib/search/init.py", line 6, in from mkdocs.contrib.search.search_index import SearchIndex File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/contrib/search/search_index.py", line 7, in from lunr import lunr File "/Users/ronny/anaconda/lib/python3.6/site-packages/lunr/init.py", line 5, in from lunr.main import lunr File "/Users/ronny/anaconda/lib/python3.6/site-packages/lunr/main.py", line 6, in from lunr import languages as lang File "/Users/ronny/anaconda/lib/python3.6/site-packages/lunr/languages/init.py", line 104, in register_languages() File "/Users/ronny/anaconda/lib/python3.6/site-packages/lunr/languages/init.py", line 95, in register_languages language_stemmer = partial(nltk_stemmer, get_language_stemmer(language)) File "/Users/ronny/anaconda/lib/python3.6/site-packages/lunr/languages/stemmer.py", line 10, in get_language_stemmer return SnowballStemmer(SUPPORTED_LANGUAGES[language]) File "/Users/ronny/anaconda/lib/python3.6/site-packages/nltk/stem/snowball.py", line 90, in init raise ValueError("The language '{0}' is not supported.".format(language)) ValueError: The language 'arabic' is not supported.

olivernn commented 3 years ago

So, I can understand that that would be a surprising error given you are not trying to use arabic!

At first glance this appears to be an issue with lunr-languages, which is a plugin for lunr that provides support for non-english languages. However I'm pretty sure that project does not provide support for arabic (it is an often requested feature).

Looking at the stack trace I see a bunch of python files, could it be this is actually coming from a Python port of lunr?

ronnydw commented 3 years ago

Correct, mkdocs uses the python port of lunr. I created also an issue there: https://github.com/yeraydiazdiaz/lunr.py/issues/87

ronnydw commented 3 years ago

the problem went away by installing mkdocs in a virtual environment. So I guess it had to do with module dependencies