bdist_pex called from tox fails with python 3.12

[anisse]

When pex is called via bdist_pex from tox, it will fail, but only on python 3.12 (python 3.9, 3.10 and 3.11 work):

Failed to spawn a job for /home/runner/work/pex-minimal-repros/pex-minimal-repros/.tox/bundle/bin/python: received exit code 1 during execution of `['/tmp/tmptki_lyu7/pip/bin/python3.12', '-s', '-E', '-m', 'pip', 'install', '-U', 'pip']` while trying to execute `['/tmp/tmptki_lyu7/pip/bin/python3.12', '-s', '-E', '-m', 'pip', 'install', '-U', 'pip']`

I made a repo to reproduce the issue, and the full output of a failed build can be found here: https://github.com/anisse/pex-minimal-repros/actions/runs/9209146256/job/25333005165#step:4:168

[jsirois]

@anisse I won't be back at a keyboard to look at either this or #2412 until May 29th, but I will note that instead of using bdist_pex, you could just say pex --disable-cache . -r requirements.in --resolver-version pip-2020-resolver --pip-version 24.0 -c entry -o entry.pex. Here the assumed CWD is the project root such that . in that command line refers to the project (setup.py) there. Of course you'd need to do this once per console script, which is certainly less convenient than --bdist-all for a setup.py with many console scripts where you want 1 PEX binary per-each. For that though, you might consider using conscript.

[anisse]

Thank you for the quick answer, I might try the workaround. There is no emergency and I'll keep the repro repo up for a while.

[jsirois]

Ok, the issue surrounds Pex bootstrapping Pip 24.0 (it only vendors 20.3.4 and for any other --pip-version it must do a 1 time fetch of the requested Pip version (it actually creates a Pip Pex for that version in the ~/.pex/pip cache)). If I move the bootstrapping out from under the bdist_pex call, things work:

# ~/dev/anisse/pex-minimal-repros (340a212) [!?] via  v3.10.12 took 16s
:; git diff
diff --git a/tox.ini b/tox.ini
index 51d3188..a430b59 100644
--- a/tox.ini
+++ b/tox.ini
@@ -14,4 +14,5 @@ commands =
 [testenv:bundle]
 changedir = ./
 commands =
-    {envpython} setup.py bdist_pex --bdist-dir=dist --pex-args='--disable-cache -r requirements.in --resolver-version pip-2020-resolver --pip-version 24.0 -vvvvv' --bdist-all
+    {envpython} -m pex --pip-version 24.0 -- -c ''
+    {envpython} setup.py bdist_pex --bdist-dir=dist --pex-args='-r requirements.in --resolver-version pip-2020-resolver --pip-version 24.0 -vvvvv' --bdist-all

# ~/dev/anisse/pex-minimal-repros (340a212) [!?] via  v3.10.12
:; tox -ebundle
.pkg: _optional_hooks> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_wheel> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: prepare_metadata_for_build_wheel> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
bundle: install_package> python -I -m pip install --force-reinstall --no-deps .tox/.tmp/package/2/repro-0.0.1.tar.gz
bundle: commands[0]> .tox/bundle/bin/python -m pex --pip-version 24.0 -- -c ''
bundle: commands[1]> .tox/bundle/bin/python setup.py bdist_pex --bdist-dir=dist '--pex-args=-r requirements.in --resolver-version pip-2020-resolver --pip-version 24.0 -vvvvv' --bdist-all
running bdist_pex
Writing entry to dist/entry
.pkg: _exit> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
  bundle: OK (6.11=setup[2.23]+cmd[0.48,3.39] seconds)
  congratulations :) (6.13 seconds)

Elsewise, if I use the workaround suggested above, no issues:

# ~/dev/anisse/pex-minimal-repros (340a212) [!?] via  v3.10.12 took 8s
:; tox -ebundle
.pkg: _optional_hooks> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_wheel> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: prepare_metadata_for_build_wheel> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
bundle: install_package> python -I -m pip install --force-reinstall --no-deps .tox/.tmp/package/4/repro-0.0.1.tar.gz
bundle: commands[0]> .tox/bundle/bin/python -m pex --disable-cache . -r requirements.in --resolver-version pip-2020-resolver --pip-version 24.0 -c entry -o dist/entry.pex
.pkg: _exit> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
  bundle: OK (16.89=setup[2.27]+cmd[14.62] seconds)
  congratulations :) (16.91 seconds)

# ~/dev/anisse/pex-minimal-repros (340a212) [!?] via  v3.10.12 took 17s
:; dist/entry.pex
Hello  1.26.4

# ~/dev/anisse/pex-minimal-repros (340a212) [!?] via  v3.10.12
:; git diff
diff --git a/tox.ini b/tox.ini
index 51d3188..e956882 100644
--- a/tox.ini
+++ b/tox.ini
@@ -14,4 +14,4 @@ commands =
 [testenv:bundle]
 changedir = ./
 commands =
-    {envpython} setup.py bdist_pex --bdist-dir=dist --pex-args='--disable-cache -r requirements.in --resolver-version pip-2020-resolver --pip-version 24.0 -vvvvv' --bdist-all
+    {envpython} -m pex --disable-cache . -r requirements.in --resolver-version pip-2020-resolver --pip-version 24.0 -c entry -o dist/entry.pex

So the issue is very narrowly bdist_pex when run under tox / virtualenv with an empty Pex cache that contains no pre-bootstrapped Pip 24.0. By adding logging to the Pex code in .tox/bundle I can see that when Pex attempts to bootstrap Pip 24.0, the -mensurepip ... subprocess succeeds but installs no Pip since the ensurepip module thinks the .tox/bundle Pip somehow satisfies the pex cache Pip venv need for a Pip:

...
pex: Executing: /tmp/tmphmwrg8pp/pip/bin/python3.12 -s -E -m ensurepip -U --default-pip -vvv
>>> STDOUT:
Using pip 24.0 from /tmp/tmp9jwsfv_v/pip-24.0-py3-none-any.whl/pip (python 3.12)
Non-user install because user site-packages disabled
Ignoring indexes: https://pypi.org/simple
Created temporary directory: /tmp/pip-build-tracker-nbnvhasb
Initialized build tracking at /tmp/pip-build-tracker-nbnvhasb
Created build tracker: /tmp/pip-build-tracker-nbnvhasb
Entered build tracker: /tmp/pip-build-tracker-nbnvhasb
Created temporary directory: /tmp/pip-install-xwu_1ldj
Created temporary directory: /tmp/pip-ephem-wheel-cache-n0e36n07
Looking in links: /tmp/tmp9jwsfv_v
Requirement already satisfied: pip in ./.tox/bundle/lib/python3.12/site-packages (24.0)
0 location(s) to search for versions of pip:
Found link file:///tmp/tmp9jwsfv_v/pip-24.0-py3-none-any.whl, version: 24.0
Local files found: /tmp/tmp9jwsfv_v/pip-24.0-py3-none-any.whl
Given no hashes to check 1 links for project 'pip': discarding no candidates
Created temporary directory: /tmp/pip-unpack-60cc48mp
Removed build tracker: '/tmp/pip-build-tracker-nbnvhasb'

>>> STDERR:

>>> ensured pip!
...

[jsirois]

Actually, I was wrong, the issue is specifically to do with Python 3.12.

Here, Python 3.11 works fine:

# ~/dev/anisse/pex-minimal-repros (340a212) [?] via  v3.10.12 took 19s
:; rm -rf dist && PYTHON=python3.11 tox -ebundle
.pkg: _optional_hooks> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_wheel> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: prepare_metadata_for_build_wheel> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
bundle: install_package> python -I -m pip install --force-reinstall --no-deps .tox/.tmp/package/13/repro-0.0.1.tar.gz
bundle: commands[0]> .tox/bundle/bin/python setup.py bdist_pex --bdist-dir=dist '--pex-args=--disable-cache -r requirements.in --resolver-version pip-2020-resolver --pip-version 24.0 -vvvvv' --bdist-all
running bdist_pex
Writing entry to dist/entry
.pkg: _exit> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
  bundle: OK (14.01=setup[0.60]+cmd[13.40] seconds)
  congratulations :) (14.03 seconds)

# ~/dev/anisse/pex-minimal-repros (340a212) [?] via  v3.10.12 took 14s
:; dist/entry
Hello  1.26.4

As do Python 3.10 and Python 3.9:

# ~/dev/anisse/pex-minimal-repros (340a212) [?] via  v3.10.12
:; rm -rf dist && PYTHON=python3.10 tox -ebundle
bundle: recreate env because python changed version_info=[3, 11, 9, 'final', 0]->[3, 10, 12, 'final', 0] | executable='/usr/bin/python3.11'->'/usr/bin/python3.10'
bundle: remove tox env folder .tox/bundle
bundle: install_deps> python -I -m pip install -r toxpex.in
.pkg: _optional_hooks> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_wheel> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: prepare_metadata_for_build_wheel> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
bundle: install_package> python -I -m pip install --force-reinstall --no-deps .tox/.tmp/package/14/repro-0.0.1.tar.gz
bundle: commands[0]> .tox/bundle/bin/python setup.py bdist_pex --bdist-dir=dist '--pex-args=--disable-cache -r requirements.in --resolver-version pip-2020-resolver --pip-version 24.0 -vvvvv' --bdist-all
running bdist_pex
Writing entry to dist/entry
.pkg: _exit> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
  bundle: OK (16.91=setup[4.05]+cmd[12.86] seconds)
  congratulations :) (16.93 seconds)

# ~/dev/anisse/pex-minimal-repros (340a212) [?] via  v3.10.12 took 17s
:; rm -rf dist && PYTHON=python3.9 tox -ebundle
bundle: recreate env because python changed version_info=[3, 10, 12, 'final', 0]->[3, 9, 19, 'final', 0] | executable='/usr/bin/python3.10'->'/home/jsirois/.pyenv/versions/3.9.19/bin/python3.9'
bundle: remove tox env folder .tox/bundle
bundle: install_deps> python -I -m pip install -r toxpex.in
.pkg: _optional_hooks> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_wheel> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: prepare_metadata_for_build_wheel> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
bundle: install_package> python -I -m pip install --force-reinstall --no-deps .tox/.tmp/package/15/repro-0.0.1.tar.gz
bundle: commands[0]> .tox/bundle/bin/python setup.py bdist_pex --bdist-dir=dist '--pex-args=--disable-cache -r requirements.in --resolver-version pip-2020-resolver --pip-version 24.0 -vvvvv' --bdist-all
running bdist_pex
Writing entry to dist/entry
.pkg: _exit> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
  bundle: OK (18.64=setup[4.50]+cmd[14.14] seconds)
  congratulations :) (18.66 seconds)

But Python 3.12 does not as we already know:

# ~/dev/anisse/pex-minimal-repros (340a212) [?] via  v3.10.12 took 18s
:; rm -rf dist && PYTHON=python3.12 tox -ebundle
bundle: recreate env because python changed version_info=[3, 9, 19, 'final', 0]->[3, 12, 3, 'final', 0] | executable='/home/jsirois/.pyenv/versions/3.9.19/bin/python3.9'->'/usr/bin/python3.12'
bundle: remove tox env folder .tox/bundle
bundle: install_deps> python -I -m pip install -r toxpex.in
.pkg: _optional_hooks> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_sdist> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_wheel> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: prepare_metadata_for_build_wheel> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
.pkg: build_sdist> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
bundle: install_package> python -I -m pip install --force-reinstall --no-deps .tox/.tmp/package/16/repro-0.0.1.tar.gz
bundle: commands[0]> .tox/bundle/bin/python setup.py bdist_pex --bdist-dir=dist '--pex-args=--disable-cache -r requirements.in --resolver-version pip-2020-resolver --pip-version 24.0 -vvvvv' --bdist-all
running bdist_pex
Writing entry to dist/entry
Failed to create pex via /home/jsirois/dev/anisse/pex-minimal-repros/.tox/bundle/bin/python -s -m pex /home/jsirois/dev/anisse/pex-minimal-repros --disable-cache -r requirements.in --resolver-version pip-2020-resolver --pip-version 24.0 -vvvvv --output-file dist/entry --script entry:
pex: Searching for pyenv root...
pex: Searching for pyenv root...: 3.5ms
pex: Hashing pex
pex: Hashing pex: 14.9ms
pex: Isolating pex
pex: Isolating pex :: Extracting pex to /tmp/tmpiea5djjp/isolated/40f0762964e8eb1357cd4b395b9bca26768ea227
pex: Isolating pex: 14.5ms
pex:   Extracting pex to /tmp/tmpiea5djjp/isolated/40f0762964e8eb1357cd4b395b9bca26768ea227: 14.2ms
pex: Executing: PYTHONPATH=/tmp/tmpiea5djjp/isolated/40f0762964e8eb1357cd4b395b9bca26768ea227 /home/jsirois/dev/anisse/pex-minimal-repros/.tox/bundle/bin/python -s -c import os
import sys

from pex.atomic_directory import atomic_directory
from pex.common import safe_open
from pex.interpreter import PythonIdentity

encoded_identity = PythonIdentity.get(binary='/home/jsirois/dev/anisse/pex-minimal-repros/.tox/bundle/bin/python').encode()
with atomic_directory('/tmp/tmpiea5djjp/interpreters/10f69cfed52f8453382807bd2db5ee69d3c4900c/fc840903f3dece9153f3bf591b5c871bb0918bbb/70ea3c46c40db9bb22b379a91deca637f96ff163') as cache_dir:
    if not cache_dir.is_finalized():
        with safe_open(
            os.path.join(cache_dir.work_dir, 'INTERP-INFO'), 'w'
        ) as fp:
            fp.write(encoded_identity)

pex: Building pex
pex: Building pex :: Adding distributions from pexes:
pex: Building pex :: Resolving distributions for requirements: /home/jsirois/dev/anisse/pex-minimal-repros requirements.in
pex: Building pex :: Resolving distributions for requirements: /home/jsirois/dev/anisse/pex-minimal-repros requirements.in :: Resolving requirements.
pex: Building pex :: Resolving distributions for requirements: /home/jsirois/dev/anisse/pex-minimal-repros requirements.in :: Resolving requirements. :: Resolving for:
  /home/jsirois/dev/anisse/pex-minimal-repros/.tox/bundle/bin/python
pex: Executing: PYTHONPATH=/tmp/tmpiea5djjp/isolated/40f0762964e8eb1357cd4b395b9bca26768ea227 /usr/bin/python3.12 -s -c import os
import sys

from pex.atomic_directory import atomic_directory
from pex.common import safe_open
from pex.interpreter import PythonIdentity

encoded_identity = PythonIdentity.get(binary='/usr/bin/python3.12').encode()
with atomic_directory('/tmp/tmpiea5djjp/interpreters/10f69cfed52f8453382807bd2db5ee69d3c4900c/fc840903f3dece9153f3bf591b5c871bb0918bbb/adf41768fe953d4706a190825ecd3db960adcbdf') as cache_dir:
    if not cache_dir.is_finalized():
        with safe_open(
            os.path.join(cache_dir.work_dir, 'INTERP-INFO'), 'w'
        ) as fp:
            fp.write(encoded_identity)

pex: Ignoring enclosing venv /home/jsirois/dev/anisse/pex-minimal-repros/.tox/bundle and using its base interpreter /usr/bin/python3.12 to create venv at /tmp/tmpsic4noij/pip instead.
pex: Scrubbed PYTHONPATH=/home/jsirois/dev/anisse/pex-minimal-repros/.tox/bundle/lib/python3.12/site-packages/pex/vendor/_vendored/attrs:/home/jsirois/dev/anisse/pex-minimal-repros:/usr/lib/python312.zip:/usr/lib/python3.12:/usr/lib/python3.12/lib-dynload:/home/jsirois/dev/anisse/pex-minimal-repros/.tox/bundle/lib/python3.12/site-packages from the virtualenv creation environment.
pex: Executing: /usr/bin/python3.12 -s -E -m venv --without-pip /tmp/tmpsic4noij/pip
pex: Executing: PYTHONPATH=/tmp/tmpiea5djjp/isolated/40f0762964e8eb1357cd4b395b9bca26768ea227 /tmp/tmpsic4noij/pip/bin/python3.12 -s -c import os
import sys

from pex.atomic_directory import atomic_directory
from pex.common import safe_open
from pex.interpreter import PythonIdentity

encoded_identity = PythonIdentity.get(binary='/tmp/tmpsic4noij/pip/bin/python3.12').encode()
with atomic_directory('/tmp/tmpiea5djjp/interpreters/10f69cfed52f8453382807bd2db5ee69d3c4900c/fc840903f3dece9153f3bf591b5c871bb0918bbb/e412c8449d184414d1d03b6eca04673129be9357') as cache_dir:
    if not cache_dir.is_finalized():
        with safe_open(
            os.path.join(cache_dir.work_dir, 'INTERP-INFO'), 'w'
        ) as fp:
            fp.write(encoded_identity)

pex: Executing: /tmp/tmpsic4noij/pip/bin/python3.12 -s -E -m ensurepip -U --default-pip
pex: Executing: /tmp/tmpsic4noij/pip/bin/python3.12 -s -E -m pip install -U pip
Failed to spawn a job for /home/jsirois/dev/anisse/pex-minimal-repros/.tox/bundle/bin/python: received exit code 1 during execution of `['/tmp/tmpsic4noij/pip/bin/python3.12', '-s', '-E', '-m', 'pip', 'install', '-U', 'pip']` while trying to execute `['/tmp/tmpsic4noij/pip/bin/python3.12', '-s', '-E', '-m', 'pip', 'install', '-U', 'pip']`

bundle: exit 1 (2.07 seconds) /home/jsirois/dev/anisse/pex-minimal-repros> .tox/bundle/bin/python setup.py bdist_pex --bdist-dir=dist '--pex-args=--disable-cache -r requirements.in --resolver-version pip-2020-resolver --pip-version 24.0 -vvvvv' --bdist-all pid=48643
.pkg: _exit> python /home/jsirois/bin/tox.venv/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta __legacy__
  bundle: FAIL code 1 (7.96=setup[5.89]+cmd[2.07] seconds)
  evaluation failed :( (7.98 seconds)

So something about ensurepip under Python 3.12 is different than under prior versions and it causes Pex's Pip 24.0 PEX venv creation process to fail.

[jsirois]

@anisse as I indicated in #2413, I won't have time to dig into a fix for this Python 3.12 + --disable-cache + bdist_pex-specific issue until May 29th, but I did have some extra time today to debug a bit deeper anyhow.

[jsirois]

Ok, by experiment, this is fixed by not relying on ensurepip and instead modifying Virtualenv.create to take a new defaulted argument of include_pip=False here and only specifying --no-pip / --without-pip when it is False: https://github.com/pex-tool/pex/blob/52ce99f183db28c23321026340f17281087fee6d/pex/venv/virtualenv.py#L163-L256

If I remove the venv.install_pip here and instead pass include_pip=True in the Virtualenv.create call, everything works fine: https://github.com/pex-tool/pex/blob/52ce99f183db28c23321026340f17281087fee6d/pex/pip/installation.py#L102-L103

[jsirois]

@anisse this should be solved with Pex 2.8.0:

• https://github.com/pex-tool/pex/releases/tag/v2.8.0
• https://pypi.org/project/pex/2.8.0/