[bug]: `venv` creation fails on `linux-musl` builds

yozachar commented 1 month ago

I installed a standalone build with mise, but I'm unable to create virtual environments.

Error

$ python -m venv .venv
Error: Command '['/home/dan/check/.venv/bin/python', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

Info

$ python
Python 3.12.3 (main, Apr 15 2024, 18:29:24) [Clang 14.0.3 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
$ which python
/home/dan/.local/share/mise/installs/python/latest/bin/python
$ which pip
/home/dan/.local/share/mise/installs/python/latest/bin/pip
$ pip list
Package    Version
---------- -------
pip        24.0
setuptools 69.1.0
$ pip debug
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.
pip version: pip 24.0 from /home/dan/.local/share/mise/installs/python/latest/lib/python3.12/site-packages/pip (python 3.12)
sys.version: 3.12.3 (main, Apr 15 2024, 18:29:24) [Clang 14.0.3 ]
sys.executable: /home/dan/.local/share/mise/installs/python/latest/bin/python3.12
sys.getdefaultencoding: utf-8
sys.getfilesystemencoding: utf-8
locale.getpreferredencoding: UTF-8
sys.platform: linux
sys.implementation:
  name: cpython
'cert' config value: Not specified
REQUESTS_CA_BUNDLE: None
CURL_CA_BUNDLE: None
pip._vendor.certifi.where(): /home/dan/.local/share/mise/installs/python/latest/lib/python3.12/site-packages/pip/_vendor/certifi/cacert.pem
pip._vendor.DEBUNDLED: False
vendored library versions:
  CacheControl==0.13.1
  colorama==0.4.6
  distlib==0.3.8
  distro==1.8.0
  msgpack==1.0.5
  packaging==21.3
  platformdirs==3.8.1
  pyparsing==3.1.0
  pyproject-hooks==1.0.0
  requests==2.31.0
  certifi==2023.07.22
  chardet==5.1.0
  idna==3.4
  urllib3==1.26.17
  rich==13.4.2 (Unable to locate actual module version, using vendor.txt specified version)
  pygments==2.15.1
  typing_extensions==4.7.1 (Unable to locate actual module version, using vendor.txt specified version)
  resolvelib==1.0.1
  setuptools==68.0.0 (Unable to locate actual module version, using vendor.txt specified version)
  six==1.16.0
  tenacity==8.2.2 (Unable to locate actual module version, using vendor.txt specified version)
  tomli==2.0.1
  truststore==0.8.0
  webencodings==0.5.1 (Unable to locate actual module version, using vendor.txt specified version)
Compatible tags: 42
  cp312-cp312-linux_x86_64
  cp312-abi3-linux_x86_64
  cp312-none-linux_x86_64
  cp311-abi3-linux_x86_64
  cp310-abi3-linux_x86_64
  cp39-abi3-linux_x86_64
  cp38-abi3-linux_x86_64
  cp37-abi3-linux_x86_64
  cp36-abi3-linux_x86_64
  cp35-abi3-linux_x86_64
  ...
  [First 10 tags shown. Pass --verbose to show all.]

What's wrong. How do I fix it?

mustafa0x commented 1 month ago

works fine here (mise latest, python 3.12.3, ubuntu 24)

egnor commented 1 month ago

This happens to me as well (also: mise latest, python 3.12.3, ubuntu 24.04)

% mise use -g python@3.12
mise installing precompiled python from indygreg/python-build-standalone
mise if you experience issues with this python, switch to python-build
mise by running: mise settings set python_compile 1
mise python@3.12.3 ✓ installed                                                  mise ~/.config/mise/config.toml tools: python@3.12.3
mise node@20.13.1 python@3.12.3                                                 
% which python
/home/egnor/.local/share/mise/installs/python/3.12/bin/python
% which pip
/home/egnor/.local/share/mise/installs/python/3.12/bin/pip
% pip list
Package    Version
---------- -------
pip        24.0
setuptools 69.1.0
% python -m venv my_venv
Error: Command '['/home/egnor/my_venv/bin/python', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

digging deeper:

% /home/egnor/my_venv/bin/python -m ensurepip --upgrade --default-pip 
ERROR: Exception:
Traceback (most recent call last):
  File "/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
             ^^^^^^^^^^^^^^^
  File "/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl/pip/_internal/cli/req_command.py", line 245, in wrapper
    return func(self, options, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl/pip/_internal/commands/install.py", line 324, in run
    session = self.get_default_session(options)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl/pip/_internal/cli/req_command.py", line 95, in get_default_session
    self._session = self.enter_context(self._build_session(options))
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl/pip/_internal/cli/req_command.py", line 122, in _build_session
    session = PipSession(
              ^^^^^^^^^^^
  File "/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl/pip/_internal/network/session.py", line 342, in __init__
    self.headers["User-Agent"] = user_agent()
                                 ^^^^^^^^^^^^
  File "/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl/pip/_internal/network/session.py", line 150, in user_agent
    zip(["lib", "version"], libc_ver()),
                            ^^^^^^^^^^
  File "/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl/pip/_internal/utils/glibc.py", line 84, in libc_ver
    glibc_version = glibc_version_string()
                    ^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl/pip/_internal/utils/glibc.py", line 8, in glibc_version_string
    return glibc_version_string_confstr() or glibc_version_string_ctypes()
                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl/pip/_internal/utils/glibc.py", line 43, in glibc_version_string_ctypes
    process_namespace = ctypes.CDLL(None)
                        ^^^^^^^^^^^^^^^^^
  File "/home/egnor/.local/share/mise/installs/python/3.12/lib/python3.12/ctypes/__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: Dynamic loading not supported
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/egnor/.local/share/mise/installs/python/3.12/lib/python3.12/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
             ^^^^^^^^^^^^^^^^^
  File "/home/egnor/.local/share/mise/installs/python/3.12/lib/python3.12/ensurepip/__init__.py", line 284, in _main
    return _bootstrap(
           ^^^^^^^^^^^
  File "/home/egnor/.local/share/mise/installs/python/3.12/lib/python3.12/ensurepip/__init__.py", line 200, in _bootstrap
    return _run_pip([*args, *_PACKAGE_NAMES], additional_paths)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/egnor/.local/share/mise/installs/python/3.12/lib/python3.12/ensurepip/__init__.py", line 101, in _run_pip
    return subprocess.run(cmd, check=True).returncode
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/egnor/.local/share/mise/installs/python/3.12/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/egnor/my_venv/bin/python', '-W', 'ignore::DeprecationWarning', '-c', '\nimport runpy\nimport sys\nsys.path = [\'/tmp/tmpuvohtfdz/pip-24.0-py3-none-any.whl\'] + sys.path\nsys.argv[1:] = [\'install\', \'--no-cache-dir\', \'--no-index\', \'--find-links\', \'/tmp/tmpuvohtfdz\', \'--upgrade\', \'pip\']\nrunpy.run_module("pip", run_name="__main__", alter_sys=True)\n']' returned non-zero exit status 2.

This also happens if I install a (precompiled) 3.11. So I don't know what's going on. But it's a real problem!

mustafa0x commented 1 month ago

I wonder whether #262 is related. I suspect it is, since the issue seems to be: the venv for Poetry can't be installed.

egnor commented 1 month ago

[edited]

For me:

% python3
>>> import pip._internal.utils.glibc
>>> pip._internal.utils.glibc.glibc_version_string()
>>>

ok, not sure if None is expected there, but maybe it is since this isn't a glibc build?, anyway it doesn't crash BUT

Python 3.12.3 (main, Apr 15 2024, 18:29:24) [Clang 14.0.3 ] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import importlib.resources
>>> whl = importlib.resources.files("ensurepip") / "_bundled" / "pip-24.0-py3-none-any.whl"
>>> import sys
>>> sys.path = [str(whl)] + sys.path
>>> import pip._internal.utils.glibc
>>> pip._internal.utils.glibc.glibc_version_string()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/egnor/.local/share/mise/installs/python/3.12/lib/python3.12/ensurepip/_bundled/pip-24.0-py3-none-any.whl/pip/_internal/utils/glibc.py", line 8, in glibc_version_string
  File "/home/egnor/.local/share/mise/installs/python/3.12/lib/python3.12/ensurepip/_bundled/pip-24.0-py3-none-any.whl/pip/_internal/utils/glibc.py", line 43, in glibc_version_string_ctypes
  File "/home/egnor/.local/share/mise/installs/python/3.12/lib/python3.12/ctypes/__init__.py", line 379, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: Dynamic loading not supported

This is using the bundled pip wheel in the same way the pip installation phase in venv creation seems to... and comes to the same end?

egnor commented 1 month ago

heyyyy I think this old bug is related!

In python-build-standalone there's a patch to the linux cpython build which exists to work around an issue that was never properly fixed where a non-dynamically-linked Python couldn't bootstrap pip:

diff --git a/pip/_internal/utils/glibc.py b/pip/_internal/utils/glibc.py
index 819979d80..4ae91e364 100644
--- a/pip/_internal/utils/glibc.py
+++ b/pip/_internal/utils/glibc.py
@@ -47,7 +47,10 @@ def glibc_version_string_ctypes():
     # manpage says, "If filename is NULL, then the returned handle is for the
     # main program". This way we can let the linker do the work to figure out
     # which libc our process is actually using.
-    process_namespace = ctypes.CDLL(None)
+    try:
+        process_namespace = ctypes.CDLL(None)
+    except OSError:
+        return None
     try:
         gnu_get_libc_version = process_namespace.gnu_get_libc_version
     except AttributeError:

That lives on BUT apparently doesn't apply to the pip-24.0-py3-none-any.whl that's bundled with ensurepip and gets used for venv creation:

% unzip -p ~/.local/share/mise/installs/python/3.12/lib/python3.12/ensurepip/_bundled/pip-24.0-py3-none-any.whl pip/_internal/utils/glibc.py | grep -2 CDLL 
        return None

    # ctypes.CDLL(None) internally calls dlopen(NULL), and as the dlopen
    # manpage says, "If filename is NULL, then the returned handle is for the
    # main program". This way we can let the linker do the work to figure out
    # which libc our process is actually using.
    process_namespace = ctypes.CDLL(None)
    try:
        gnu_get_libc_version = process_namespace.gnu_get_libc_version

Note the lack of any try/except around the ctypes.CDLL(None) call -- the patch is not applied there. So that's how it breaks:

use standalone python build
go to create a venv
ensurepip uses its bundled, unpatched pip whl to install itself
the bundled, unpatched pip whl tries various ways to get the glibc version
this isn't a glibc build (it uses MUSL) so things like CS_GNU_LIBC_VERSION aren't set
so the bundled, unpatched pip whl tries to use ctypes.CDLL(None) without a guard
crash boom etc.

My question is how does this ever work? Clearly it works for @mustafa0x and for that matter used to work for me until recently. Is there some odd path I and @yozachar both followed that leads to us tripping over this pothole?

egnor commented 1 month ago

On a different computer I have access to, I don't see this problem, and the difference appears to be that the precompiled Python that mise is pulling down is the -linux-gnu dynamically linked version (instead of the -linux-musl version) and thus ctypes.CDLL(None) works as usual (but also isn't even needed because os.confstr("CS_GNU_LIBC_VERSION") returns a value). Looking into how that happens.

egnor commented 1 month ago

OK, I think this bug should be retitled: "venv creation fails on linux-musl builds". The cause is the issue with an unpatched embedded pip wheel as described above. The fix is... not obvious to me.

There is a DIFFERENT question which is: why is mise installing linux-musl precompiled Python builds on some unsuspecting Ubuntu systems (which definitely use glibc)? But, that's a mise issue, not a python-build-standalone issue.

charliermarsh commented 1 month ago

Thanks @egnor, that makes sense to me. Is there anything to do in python-build-standalone then?

charliermarsh commented 1 month ago

I guess the change would be... somehow apply that glibc.py patch even when the pip comes from elsewhere? (It makes sense that a dlopen call would fail in the musl build.)

egnor commented 1 month ago

I guess the change would be... somehow apply that glibc.py patch even when the pip comes from elsewhere? (It makes sense that a dlopen call would fail in the musl build.)

I think upstreaming that change would be the best thing! It's clearly a good robustness thing and afaik the only thing that stops pip from working on statically linked Python

charliermarsh commented 1 month ago

Yeah. It looks like there was some work on this in the context of python-build-standalone: https://github.com/pypa/pip/issues/6543#issuecomment-496321352. I guess we'd need to change it in CPython, pip, and setuptools? I haven't dug deeper than reading that thread.

charliermarsh commented 1 month ago

I can try to submit a PR for this in pip.

egnor commented 1 month ago

Yeah I'm not sure the full situation between pip, setuptools, ctypes (in cpython), etc, and it may have evolved since that thread. Certainly that specific patch seems... eminently good?

egnor commented 1 month ago

And, there's no rush since most of us don't need to use the -musl version, and those of us who do don't usually need pip.

charliermarsh commented 1 month ago

Yeah, in python-build-standalone at least I only see that patch on pip, but I'll do some research. Thanks for all your help here.

charliermarsh commented 1 month ago

Ok, it looks like @indygreg actually did patch it in setuptools: https://github.com/pypa/packaging/pull/294

charliermarsh commented 1 month ago

Added in https://github.com/pypa/pip/pull/12716.

indygreg / python-build-standalone

[bug]: `venv` creation fails on `linux-musl` builds #261

Error

Info