yeraydiazdiaz / lunr.py

A Python implementation of Lunr.js 🌖
http://lunr.readthedocs.io
MIT License
188 stars 16 forks source link

ValueError: The language 'arabic' is not supported. (I am not using arabic!) #87

Closed ronnydw closed 3 years ago

ronnydw commented 4 years ago

I get the following error in a mkdocs project. The documents are in Dutch, there is no arabic in the text anywhere. So this error seems strange. I also created an issue in mkdocs (mkdocs/mkdocs#2149), but the the problem seems to be in lunr Are there any characters that are not supported, ie €, | , @ ...?

INFO - Building documentation... Traceback (most recent call last): File "/Users/ronny/anaconda/bin/mkdocs", line 8, in sys.exit(cli()) File "/Users/ronny/anaconda/lib/python3.6/site-packages/click/core.py", line 829, in call return self.main(args, kwargs) File "/Users/ronny/anaconda/lib/python3.6/site-packages/click/core.py", line 782, in main rv = self.invoke(ctx) File "/Users/ronny/anaconda/lib/python3.6/site-packages/click/core.py", line 1259, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/Users/ronny/anaconda/lib/python3.6/site-packages/click/core.py", line 1066, in invoke return ctx.invoke(self.callback, ctx.params) File "/Users/ronny/anaconda/lib/python3.6/site-packages/click/core.py", line 610, in invoke return callback(args, kwargs) File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/main.py", line 136, in serve_command kwargs File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/commands/serve.py", line 141, in serve config = builder() File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/commands/serve.py", line 129, in builder **kwargs File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/config/base.py", line 197, in load_config errors, warnings = cfg.validate() File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/config/base.py", line 107, in validate run_failed, run_warnings = self._validate() File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/config/base.py", line 62, in _validate self[key] = config_option.validate(value) File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/config/config_options.py", line 131, in validate return self.run_validation(value) File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/config/config_options.py", line 609, in run_validation plgins[item] = self.load_plugin(item, cfg) File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/config/config_options.py", line 617, in load_plugin Plugin = self.installed_plugins[name].load() File "/Users/ronny/anaconda/lib/python3.6/site-packages/pkg_resources/init.py", line 2341, in load return self.resolve() File "/Users/ronny/anaconda/lib/python3.6/site-packages/pkg_resources/init.py", line 2347, in resolve module = import(self.module_name, fromlist=['name'], level=0) File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/contrib/search/init.py", line 6, in from mkdocs.contrib.search.search_index import SearchIndex File "/Users/ronny/anaconda/lib/python3.6/site-packages/mkdocs/contrib/search/search_index.py", line 7, in from lunr import lunr File "/Users/ronny/anaconda/lib/python3.6/site-packages/lunr/init.py", line 5, in from lunr.main import lunr File "/Users/ronny/anaconda/lib/python3.6/site-packages/lunr/main.py", line 6, in from lunr import languages as lang File "/Users/ronny/anaconda/lib/python3.6/site-packages/lunr/languages/init.py", line 104, in register_languages() File "/Users/ronny/anaconda/lib/python3.6/site-packages/lunr/languages/init.py", line 95, in register_languages language_stemmer = partial(nltk_stemmer, get_language_stemmer(language)) File "/Users/ronny/anaconda/lib/python3.6/site-packages/lunr/languages/stemmer.py", line 10, in get_language_stemmer return SnowballStemmer(SUPPORTED_LANGUAGES[language]) File "/Users/ronny/anaconda/lib/python3.6/site-packages/nltk/stem/snowball.py", line 90, in init raise ValueError("The language '{0}' is not supported.".format(language)) ValueError: The language 'arabic' is not supported.

yeraydiazdiaz commented 4 years ago

Hi @ronnydw, could you share your mkdocs.yml?

The language passed onto Lunr is set via the lang option.

Regardless Lunr.py incorrectly states that arabic is supported, so that will need fixing.

ronnydw commented 4 years ago

Hi @ronnydw, could you share your mkdocs.yml?

The language passed onto Lunr is set via the lang option.

Regardless Lunr.py incorrectly states that arabic is supported, so that will need fixing.

Hi @yeraydiazdiaz, I use "language: nl", don't know where the arabic comes from:

mkdocs.yaml:

site_name: Index Trackers theme: name: material language: nl features:

markdown_extensions:

nav:

use_directory_urls: false

Anyway my problem is solved. I guess it had to do with module dependencies. I installed mkdocs in a virtual environment and the problem went away

yeraydiazdiaz commented 3 years ago

I can't reproduce this exception, SnowballStemmer does support arabic, not sure what went wrong there.