`run` environment gets old setuptools from wrong Python

[jaraco]

In https://github.com/coherent-oss/system/issues/17#issuecomment-2334917629, we've adopted pipx to run a cli tool (coherent.test). The command is simple:

pipx run --python 3.12 coherent.test

One of the dependencies of coherent.test is setuptools (by way of setuptools-scm).

Nevertheless, setuptools fails to import:

  File "/opt/pipx/.cache/d1eeadd94b582be/lib/python3.12/site-packages/setuptools_scm/_integration/setuptools.py", line 10, in <module>
    import setuptools
  File "/opt/pipx/shared/lib/python3.10/site-packages/setuptools/__init__.py", line 10, in <module>
    import distutils.core
ModuleNotFoundError: No module named 'distutils'

As you can see from the traceback, even though Python 3.12 was indicated, the dependency on setuptools was satisfied by the pipx install under Python 3.10. Under Python 3.10, one might expect distutils to be supplied by the stdlib, so might install Setuptools in such a way that it relies on the stdlib, but on Python 3.12, distutils is gone, so relies entirely on the vendored copy of distutils.

It seems a bug to me that pipx would allow any dependency to be satisfied by an environment for another Python version, especially if --system-site-packages wasn't part of the command.

This issue occurs on both Ubuntu and Windows in GitHub CI, so a fairly standard environment, but not one over which I have much control.

Any advice on why the wrong Setuptools is being involved here?

[jaraco]

In https://github.com/jaraco/pipx-1539/commit/57411d4d399fcff20139686b538f51bf780c5ddd, I attempted to create a minimal reproducer, but in that minimal environment, the issue doesn't reproduce, so there's some other factor at play.

[jaraco]

In this run (from https://github.com/jaraco/pipx-1539/commit/c120c9ee84aa8eea97c27cf63415cd546b410f5d), I've replicated the issue minimally, only using pipx run (remembering to include --python) and setuptools.

[jaraco]

For posterity, here's the workflow:

name: tests

on:
  push:

jobs:
  test:
    strategy:
      fail-fast: false
      matrix:
        python:
        - "3.12"
        platform:
        - ubuntu-latest
        - macos-latest
        - windows-latest

    runs-on: ${{ matrix.platform }}

    steps:
      - uses: actions/checkout@v4
      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python }}
      - name: Run tests for ${{ matrix.python }} (${{ runner.os }})
        run: pipx run --python ${{ matrix.python}} --spec setuptools python -c "import setuptools"

And here's the output:

Run pipx run --python 3.12 --spec setuptools python -c "import setuptools"
⚠️  python is already on your PATH and installed at
    /opt/hostedtoolcache/Python/3.12.5/x64/bin/python. Downloading and running
    anyway.
creating virtual environment...
installing setuptools...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/pipx/shared/lib/python3.10/site-packages/setuptools/__init__.py", line 10, in <module>
    import distutils.core
ModuleNotFoundError: No module named 'distutils'
Error: Process completed with exit code 1.

[jaraco]

I tried replicating the issue using Docker, but even using ubuntu:jammy and installing python3-setuptools and pipx using apt, the issue doesn't replicate. Curiously there, pipx doesn't accept --python 3.12, suggesting there may be an older version of pipx at play from apt.

[jaraco]

Aha! Using the latest pipx, I'm able to replicate the issue in a Docker image:

FROM ubuntu:jammy

RUN apt update
RUN apt upgrade -y
RUN apt install -y software-properties-common
RUN apt-add-repository -y ppa:deadsnakes
RUN apt update

# Disable interactive on the installs below
ENV DEBIAN_FRONTEND=noninteractive

# Set TZ to avoid spurious errors from Sphinx (nektos/act#1853)
ENV TZ=UTC

# Install Pythons
RUN apt install -y python3.10 python3.10-dev python3.10-venv
RUN apt install -y python3.12 python3.12-dev python3.12-venv

RUN apt install -y python3-setuptools python3-pip

RUN python3 -m pip install pipx

CMD pipx run --python 3.12 --spec setuptools python -c "import setuptools"

[jaraco]

I can see that when launching python, python3.10/site-packages is on sys.path:

 🐚 docker run -it @$(docker build -q .) pipx run --python 3.12 --spec setuptools python
Python 3.12.5 (main, Aug 17 2024, 16:46:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python312.zip', '/usr/lib/python3.12', '/usr/lib/python3.12/lib-dynload', '/root/.cache/pipx/6b9620d37d49ab9/lib/python3.12/site-packages', '/root/.local/share/pipx/shared/lib/python3.10/site-packages']

[jaraco]

I thought maybe site-packages was relevant, as that affects .pth file processing.

>>> site.getsitepackages()
['/root/.cache/pipx/6b9620d37d49ab9/lib/python3.12/site-packages', '/root/.cache/pipx/6b9620d37d49ab9/local/lib/python3.12/dist-packages', '/root/.cache/pipx/6b9620d37d49ab9/lib/python3/dist-packages', '/root/.cache/pipx/6b9620d37d49ab9/lib/python3.12/dist-packages']

I see that the python3.10 path doesn't appear there.

But then I realized that the main reason that that the distutils shim doesn't take effect is it's an older version of setuptools whose SETUPTOOLS_USE_DISTUTILS defaults to stdlib. So even on Python 3.12, the distutils hack has no effect, but because there's no distutils present in the stdlib, that causes the failure.

I did notice that setting SETUPTOOLS_USE_DISTUTILS=local works around the issue (setuptools is able to import), though it's still problemmatic that pipx run setuptools or pipx run setuptools-scm ends up with an old copy of Setuptools as installed to Python 3.10 by the system packager.

I guess setuptools-scm is partly to blame in that it doesn't declare a minimum version.

My question to the pipx team - is it reasonable/expected for pipx shared libs for the wrong Python version to end up on the Python path for the package installation and app invocation?

[jaraco]

I've confirmed that $PIPX_SHARED_LIBS/lib/python3.10 doesn't exist until the first pipx run, presumably when it's "creating shared libraries" and "upgrading shared libraries". Too bad the upgraded version is a version that's almost 3 years old. I've also confirmed that apt remove -y python3-setuptools has no effect. pipx gets setuptools from elsewhere.

[jaraco]

In #1078, it appears as if someone has attempted to remove setuptools and wheel from the shared libraries, but that appears not to be the case.

[jaraco]

It appears that pipx gets setuptools as the default behavior for python3.10 -m venv. Probably since setuptools is not meant to be there, it should be uninstalled.

[jaraco]

I could potentially work around this issue by updating the Ubuntu image to 24.04, which bumps the default python3 to 3.12, which produces a venv without setuptools. However, that won't address the issue for the Windows host.

[jaraco]

I'm so dang close to having a usable workaround. In https://github.com/jaraco/pipx-1539/commit/c2a124f4832db2ca852d21e5df18426bb0a06aaf, I have a GitHub action that works around the issue on Linux by running a script to uninstall setuptools. The same script, however, fails to run on Windows due to bin/Scripts mismatch.

[jaraco]

In https://github.com/jaraco/pipx-1539/actions/runs/10752003994, I have a run that succeeds... and all it requires to work around the issue is to add the following to the github workflow:

      - name: Install wget (Windows only)
        if: runner.os == 'Windows'
        run: choco install wget -y
      - name: Workaround pypa/pipx#1539
        run: wget https://raw.githubusercontent.com/jaraco/pipx-1539/main/workaround-1539.py -O - | python3

Still, it would be nice if pipx had a more permanent, integrated solution.

[chrysle]

Sorry for the oversight. What would you recommend? Uninstalling setuptools manually internally? This would have to be done always for every venv creation, which could make it costly; of course only for Python < 3.12, afterwards it's no longer a dependency.

[jaraco]

What would you recommend? Uninstalling setuptools manually internally?

That's what I was thinking. Other options to consider:

• Run venv --without-pip to create the env and then install pip explicitly.
• Upgrade setuptools, so at least it's not an old copy.

My preference is uninstall or avoid installing.

This would have to be done always for every venv creation, which could make it costly; of course only for Python < 3.12, afterwards it's no longer a dependency.

That seems the least disruptive option. And the cost is small comparable to the cost of already installing an unwanted package.