search

squidfunk / mkdocs-material

Documentation that simply works
https://squidfunk.github.io/mkdocs-material/
MIT License
20.97k stars 3.55k forks source link

regex library depedency is causing installation problems #7676

Closed ssbarnea closed 1 week ago

ssbarnea commented 1 week ago

Context

MacOS arm64 python 3.12 or newer using uv instead of pip to install dependencies.

I raised this bug on mkdocs-material in order to re-evaluate the need for keeping this a direct dependency. If we are not able to find an alternative, we should at least consider making it a soft-dependency as it would be much easier to not use the search plugin that not being able to use mkdocs-material.

Bug description

AFAIK, regex library is only used inside the search plugin but is listed as a permanent dependency. Apparently bugs like https://github.com/mrabarnett/mrab-regex/issues/413 and https://github.com/mrabarnett/mrab-regex/issues/547 show that this binary dependency is problematic as it causes installation failures for a notable number of users.

This dependency is not using PEP-517 packaging.

Related links

Reproduction

-

Steps to reproduce

> uv pip install mkdocs-material
> python3 -m "import regex"
ModuleNotFoundError: No module named 'regex'

Browser

No response

Before submitting

squidfunk commented 1 week ago

Thanks for reporting. We need regex for Chinese search. I'm not sure how this could be made a "soft dependency" (I think you mean optional?), as search is a central plugin. pip is still the gold standard, so it sounds more like a problem with uv. If we can find another way to check for Chinese characters, we might swap it out. Other than that, we could only move it to an optional dependency in the next major release, since this is a breaking change for all Chinese users.

facelessuser commented 1 week ago

I would agree that if it works with pip, the standard, and not uv that maybe the problem is with uv. Regardless of uv being preferred by some. But maybe I don't have a full understanding of the real problem.

alexvoss commented 1 week ago

should this work?

python3 -m "import regex"

would that not be looking for a module "import regex"?

I installed regex as described in a venv and running "import regex" at the Python prompt works fine. Material also seems to work fine, thought I only tried a toy project. edit: replaced the -m in the command above with -c and it works as well.

facelessuser commented 1 week ago

I use regex all the time. I install it on various machines via pip and I never have had problem with import regex. I also don't heavily use uv, I think I've played with it, but I still rely on pip mainly.

ssbarnea commented 1 week ago

We don't need to make a decision right away, especially as it will take few days to figure out exactly what is causing this issue. Apparently is not uv fault. Now I suspect the way python is compiled/installed or the non standard packaging of regex wheel. -- I find it bit fascinating because it seems to be a recurring issue (first report ~5 years old), even black was affected by it https://github.com/psf/black/issues/1207 but they sorted it by removing it.

I don't publish anything in Chinese and I bet that people that do are less than 10-20% of mkdocs-material users. This would make it a good case for adding this as an extra. Whoever needs that functionality could install mkdocs[chinese] (or whatever name we decide to expose this feature as). And yes, I can understand that this would be breaking change.

facelessuser commented 1 week ago

If the exact problem could be quantified, a PR could be issued over at regex. I don't think anyone knows exactly what the issue is....

squidfunk commented 1 week ago

I don't publish anything in Chinese and I bet that people that do are less than 10-20% of mkdocs-material users. This would make it a good case for adding this as an extra. Whoever needs that functionality could install mkdocs[chinese] (or whatever name we decide to expose this feature as). And yes, I can understand that this would be breaking change.

This is exactly the argument that is a non-argument. 10-20% is a lot, and yes, Material is very popular among Chinese users, since it is one of the few SSGs that supports Chinese search to a reasonable degree, albeit it's still not perfect. Chinese is the second most spoken language in the world, so there's quite a lot of users out there. We do not want to make it anymore complicated, and as I mentioned, it would mean a breaking change which we're currently not planning for.

For this reason, we're currently not considering this change. I hope for your understanding.

kamilkrzyskow commented 6 days ago

Hello @ssbarnea, have you debugged where does uv install the packages and what executable does python3 actually run? Or what are the visible site-packages during Python execution?

I tested Windows 10 + Python 3.13 + venv + uv and it works for me. I tested Alpine Linux + Python 3.11 + venv + uv and it works for me.

In the other link you gave this example:

> uv pip install regex==2024.9.11
> python3 -c "import regex._regex as _regex"
ModuleNotFoundError: No module named 'regex._regex'

> uv pip uninstall regex
> pip install regex==2024.9.11
> python3 -c "import regex._regex as _regex"
success

This gave me the idea that perhaps uv and python3 use different Python instances, classic Python shenanigans šŸ˜… 

(test-uv) :~$ uv pip tree --verbose
DEBUG uv 0.4.30
DEBUG Searching for default Python interpreter in system path
DEBUG Found `cpython-3.11.9-linux-x86_64-musl` at `/home/frog/test-uv/bin/python3` (active virtual environment)
Using Python 3.11.9 environment at test-uv
mkdocs-material v9.5.44
# ...
(test-uv) :~$ python3 -c "import sys; print(*sys.path, sep='\n'); print(sys.executable)"

/usr/lib/python311.zip
/usr/lib/python3.11
/usr/lib/python3.11/lib-dynload
/home/frog/test-uv/lib/python3.11/site-packages
/home/frog/test-uv/bin/python3

Make sure the paths match between commands and there is no weird path to another version. Also check these commands:

$ uv pip show regex
Location: /home/frog/test-uv/lib/python3.11/site-packages
$ python3 -m pip show regex
Location: /home/frog/test-uv/lib/python3.11/site-packages
$ ls -la /home/frog/test-uv/lib/python3.11/site-packages/regex
total 2368
drwxr-sr-x    3 frog     frog          4096 Nov  7 04:42 .
drwxr-sr-x   67 frog     frog          4096 Nov  7 04:42 ..
-rw-r--r--    2 frog     frog            65 Nov  7 04:42 __init__.py
drwxr-sr-x    2 frog     frog          4096 Nov  7 04:42 __pycache__
-rwxr-xr-x    2 frog     frog       2003488 Nov  7 04:42 _regex.cpython-311-x86_64-linux-musl.so #Compiled regex._regex file unique for each OS variant
-rw-r--r--    2 frog     frog        141028 Nov  7 04:42 _regex_core.py
-rw-r--r--    2 frog     frog         32683 Nov  7 04:42 regex.py
-rw-r--r--    2 frog     frog        222040 Nov  7 04:42 test_regex.py

I didn't read much of it but here I see version 3.9 at the start of the code block and the traceback errors paths are from python3.8 šŸ¤” Very strange indeed šŸ”

kamilkrzyskow commented 5 days ago

This was tracked down to be an issue with uv, with pinning the python version, resulting in the installation of incorrect wheel packages.

I had a hunch we were dealing with a mixed bag of issues and the error in the CI run confirmed it:

  File "/home/runner/work/tox-ansible/tox-ansible/.tox/docs/lib/python3.12/site-packages/material/plugins/search/plugin.py", line 24, in <module>
    import regex as re
  File "/home/runner/work/tox-ansible/tox-ansible/.tox/docs/lib/python3.12/site-packages/regex/__init__.py", line 1, in <module>
    from .regex import *
  File "/home/runner/work/tox-ansible/tox-ansible/.tox/docs/lib/python3.12/site-packages/regex/regex.py", line 417, in <module>
    import regex._regex_core as _regex_core
  File "/home/runner/work/tox-ansible/tox-ansible/.tox/docs/lib/python3.12/site-packages/regex/_regex_core.py", line 21, in <module>
    import regex._regex as _regex
ModuleNotFoundError: No module named 'regex._regex'

The regex module was found site-packages/regex but the import chain errors out at import regex._regex as _regex. I assumed before, that the compiled module was missing altogether, but it turned out it's a version mismatch for the compiled Python version.

Crazy edge case shenanigans šŸ¤Æ

facelessuser commented 5 days ago

Looks like Occam's razor turned out to be correct in this case. Problem installing package with uv? It's probably a problem with uv šŸ™‚.