Closed zmanji closed 1 month ago
Two pips is getting big, I'm partial to Pex 3.x vendoring a single newer Pip.
That said, you're about to run into more problems anyhow using Python Build Standalone - those won't be able to build native wheels from sdists - sysconfig has many key bad paths embedded from the PBS build process.
I forgot to say, this is not an urgent/important thing. Just reporting it to share the news. I agree that PBS is not great for building wheels and fixing this is sure to reveal more issues.
Maybe it would be good just to tag this with a 'pex3' tag, and the issue can be revisited whenever pex3 is nearing release.
@zmanji - I'm re-reading this issue on a quest to re-look at all outstanding issues you've filed. They tend to be good ones.
due to https://github.com/pypa/pip/issues/6543 it's not possible for pex to resolve dependencies with a statically linked python.
AFAICT that linked issue is a red-herring since the fix was in in the Pip 19 series (and Pex only uses 20.3.4 vendored and newer). FWICT that fix tries to os.confstr("CS_GNU_LIBC_VERSION")
to get the glibc version before falling back to ctypes. Since a musl libc system won't have that conf string, everything is still broken at the Pip level (if Pip exists in a running state on musl libc systems it seems to me it must be a patched version of it - and I know the one IndyGreg ships is in-fact patched).
I just tried with https://github.com/indygreg/python-build-standalone/releases/download/20240713/cpython-3.10.14+20240713-x86_64-unknown-linux-musl-install_only.tar.gz and get the same backtrace:
./python/bin/python3 ./pex 'fortune' -o fortune.pex
pid 268587 -> /home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/bin/python -sE /home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /home/jsirois/.pex/pip/20.3.4-patched/pip_cache download --dest /home/jsirois/.pex/downloads/resolver_download.c6knb953/tmp.issues-2017.python.bin.python3.10 fortune --retries 5 --timeout 15 exited with 2 and STDERR:
ERROR: Exception:
Traceback (most recent call last):
File "/home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
status = self.run(options, args)
File "/home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
return func(self, options, args)
File "/home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/lib/python3.10/site-packages/pip/_internal/commands/download.py", line 92, in run
session = self.get_default_session(options)
File "/home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 78, in get_default_session
self._session = self.enter_context(self._build_session(options))
File "/home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 88, in _build_session
session = PipSession(
File "/home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/lib/python3.10/site-packages/pip/_internal/network/session.py", line 248, in __init__
self.headers["User-Agent"] = user_agent()
File "/home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/lib/python3.10/site-packages/pip/_internal/network/session.py", line 135, in user_agent
zip(["lib", "version"], libc_ver()),
File "/home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/lib/python3.10/site-packages/pip/_internal/utils/glibc.py", line 94, in libc_ver
glibc_version = glibc_version_string()
File "/home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/lib/python3.10/site-packages/pip/_internal/utils/glibc.py", line 18, in glibc_version_string
return glibc_version_string_confstr() or glibc_version_string_ctypes()
File "/home/jsirois/.pex/venvs/d9b28cb541ebce85a9949b0840e48c9b903a1e25/97e96d9832fbf2e84fd927347407abbc9f59d471/lib/python3.10/site-packages/pip/_internal/utils/glibc.py", line 52, in glibc_version_string_ctypes
process_namespace = ctypes.CDLL(None)
File "/tmp/issues-2017/python/lib/python3.10/ctypes/__init__.py", line 374, in __init__
self._handle = _dlopen(self._name, mode)
OSError: Dynamic loading not supported
The real fix only landed a few weeks ago afaict and is unreleased (will be released in 24.2 - probaly next weekend): https://github.com/pypa/pip/pull/12716 I'll see if I can pull in the glibc patches to the vendored Pip Pex uses, which should allow bootstrapping the latest Pip with Charlie's fix on musl libc systems.
Ok, That picked cleanly, although I had to remove a Python3.5+-style type annotation so this still works with Python 2.7: https://github.com/pex-tool/pip/pull/13
Ok, that does the trick for the OP. You can have your fortune read by Pex using Pip 20.3.4+patched on musl libc.
Alright - the fix for this is released in 2.12.0:
:; wget https://github.com/indygreg/python-build-standalone/releases/download/20221220/cpython-3.10.9+20221220-x86_64_v3-unknown-linux-musl-install_only.tar.gz
:; tar xvzf cpython-3.10.9+20221220-x86_64_v3-unknown-linux-musl-install_only.tar.gz
:; wget https://github.com/pantsbuild/pex/releases/download/v2.12.0/pex
:; ./fortune.pex /usr/share/games/fortunes/fortunes
You will stop at nothing to reach your objective, but only because your
brakes are defective.
due to https://github.com/pypa/pip/issues/6543 it's not possible for pex to resolve dependencies with a statically linked python.
To reproduce in a clean chroot.
Changing the pip version does not work because the vendored pip is used to fetch the newer pip. I see two possible solutions: