search

pex-tool / pex

A tool for generating .pex (Python EXecutable) files, lock files and venvs.
https://docs.pex-tool.org
Apache License 2.0
2.49k stars 254 forks source link

Support `--pip-version 24.1` and Python 3.13. #2435

Closed jsirois closed 2 weeks ago

jsirois commented 2 weeks ago

Pex has been testing against Python 3.13 alpha and then beta releases for a while now using a patched Pip with Python 3.13 support. Those patches have made it into the Pip 24.1 release; so now Pex can officially support Python 3.13.

The Pip 24.1 release notes: https://pip.pypa.io/en/stable/news/#v24-1

Closes #2406

jsirois commented 2 weeks ago

I do thing it's interesting/confusing that pex has a vendored packaging and uses the packaging vendored in pip as well. I know their usecases/needs are different but I wonder if this might cause issues down the road.

Agreed. One thing I started on in 2020 but never finished since its somewhat frivolous and self-defeating, is a massive re-structure of Pex modules to clearly delineate code needed at PEX runtime from code needed by the Pex tool at build time. The vendored Pip is only in the latter set and does not get included in a PEX files .bootstrap/ code, but the PEX bootstrap does need to evaluate both markers and specifiers, which it relies on packaging for. So this is the origin of the need for a a separate vendoring. It could be done differently though. Perhaps Pex could pluck out Pip's vendored packaging and re-vendor just that portion of Pip in the PEX .bootstrap/... except that PEX files can support Python 2.7 ~through 3.13 and no packaging works with all those versions; so currently you'll find this perhaps surprising thing:

;: pex -o empty.pex

:; zipinfo -1 empty.pex | grep packaging
.bootstrap/pex/vendor/_vendored/packaging_20_9/
.bootstrap/pex/vendor/_vendored/packaging_20_9/__init__.py
.bootstrap/pex/vendor/_vendored/packaging_20_9/packaging/
.bootstrap/pex/vendor/_vendored/packaging_20_9/packaging/__about__.py
.bootstrap/pex/vendor/_vendored/packaging_20_9/packaging/__init__.py
.bootstrap/pex/vendor/_vendored/packaging_20_9/packaging/_compat.py
.bootstrap/pex/vendor/_vendored/packaging_20_9/packaging/_structures.py
.bootstrap/pex/vendor/_vendored/packaging_20_9/packaging/_typing.py
.bootstrap/pex/vendor/_vendored/packaging_20_9/packaging/markers.py
.bootstrap/pex/vendor/_vendored/packaging_20_9/packaging/py.typed
.bootstrap/pex/vendor/_vendored/packaging_20_9/packaging/requirements.py
.bootstrap/pex/vendor/_vendored/packaging_20_9/packaging/specifiers.py
.bootstrap/pex/vendor/_vendored/packaging_20_9/packaging/tags.py
.bootstrap/pex/vendor/_vendored/packaging_20_9/packaging/utils.py
.bootstrap/pex/vendor/_vendored/packaging_20_9/packaging/version.py
.bootstrap/pex/vendor/_vendored/packaging_20_9/pyparsing.py
.bootstrap/pex/vendor/_vendored/packaging_21_3/
.bootstrap/pex/vendor/_vendored/packaging_21_3/__init__.py
.bootstrap/pex/vendor/_vendored/packaging_21_3/packaging/
.bootstrap/pex/vendor/_vendored/packaging_21_3/packaging/__about__.py
.bootstrap/pex/vendor/_vendored/packaging_21_3/packaging/__init__.py
.bootstrap/pex/vendor/_vendored/packaging_21_3/packaging/_manylinux.py
.bootstrap/pex/vendor/_vendored/packaging_21_3/packaging/_musllinux.py
.bootstrap/pex/vendor/_vendored/packaging_21_3/packaging/_structures.py
.bootstrap/pex/vendor/_vendored/packaging_21_3/packaging/markers.py
.bootstrap/pex/vendor/_vendored/packaging_21_3/packaging/py.typed
.bootstrap/pex/vendor/_vendored/packaging_21_3/packaging/requirements.py
.bootstrap/pex/vendor/_vendored/packaging_21_3/packaging/specifiers.py
.bootstrap/pex/vendor/_vendored/packaging_21_3/packaging/tags.py
.bootstrap/pex/vendor/_vendored/packaging_21_3/packaging/utils.py
.bootstrap/pex/vendor/_vendored/packaging_21_3/packaging/version.py
.bootstrap/pex/vendor/_vendored/packaging_21_3/pyparsing.py
.bootstrap/pex/vendor/_vendored/packaging_23_1/
.bootstrap/pex/vendor/_vendored/packaging_23_1/__init__.py
.bootstrap/pex/vendor/_vendored/packaging_23_1/packaging/
.bootstrap/pex/vendor/_vendored/packaging_23_1/packaging/__init__.py
.bootstrap/pex/vendor/_vendored/packaging_23_1/packaging/_elffile.py
.bootstrap/pex/vendor/_vendored/packaging_23_1/packaging/_manylinux.py
.bootstrap/pex/vendor/_vendored/packaging_23_1/packaging/_musllinux.py
.bootstrap/pex/vendor/_vendored/packaging_23_1/packaging/_parser.py
.bootstrap/pex/vendor/_vendored/packaging_23_1/packaging/_structures.py
.bootstrap/pex/vendor/_vendored/packaging_23_1/packaging/_tokenizer.py
.bootstrap/pex/vendor/_vendored/packaging_23_1/packaging/markers.py
.bootstrap/pex/vendor/_vendored/packaging_23_1/packaging/metadata.py
.bootstrap/pex/vendor/_vendored/packaging_23_1/packaging/py.typed
.bootstrap/pex/vendor/_vendored/packaging_23_1/packaging/requirements.py
.bootstrap/pex/vendor/_vendored/packaging_23_1/packaging/specifiers.py
.bootstrap/pex/vendor/_vendored/packaging_23_1/packaging/tags.py
.bootstrap/pex/vendor/_vendored/packaging_23_1/packaging/utils.py
.bootstrap/pex/vendor/_vendored/packaging_23_1/packaging/version.py
.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/packaging/
.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/packaging/__about__.py
.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/packaging/__init__.py
.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/packaging/_compat.py
.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/packaging/_structures.py
.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/packaging/markers.py
.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/packaging/requirements.py
.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/packaging/specifiers.py
.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/packaging/utils.py
.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/packaging/version.py

So its worse than you thought, but required to ensure the PEX can boot under various pythons and not choke on syntax errors in the packaging code it needs to run.