Stop bundling setuptools for Python 3.12+

[edmorley]

In Python 3.12, the stdlib's ensurepip and venv modules were updated to no longer install setuptools alongside pip. This was viable since as of pip v22.1, for pre-PEP-517/518 packages pip will now default to the isolated build environment mode (along with a fallback legacy setuptools build backend, with setuptools and wheel automatically installed), if the setuptools package isn't installed globally.

As such, when using the default Python configure options (which include --with-ensurepip, which causes ensurepip to run during the build), the resultant Python build only includes pip and not setuptools for Python 3.12+.

For example, this then means the Python distribution used by the official GitHub Actions setup-python action (which don't override the upstream CPython configure defaults) automatically only ships setuptools for Python 3.11 and older:

# Python 3.11
$ curl -sSfL https://github.com/actions/python-versions/releases/download/3.11.9-9947079978/python-3.11.9-linux-24.04-arm64.tar.gz | tar -zt | grep -E 'site-packages/[^/]+/$'
./lib/python3.11/site-packages/pip-24.0.dist-info/
./lib/python3.11/site-packages/_distutils_hack/
./lib/python3.11/site-packages/pkg_resources/
./lib/python3.11/site-packages/pip/
./lib/python3.11/site-packages/setuptools/
./lib/python3.11/site-packages/setuptools-65.5.0.dist-info/

# Python 3.12
$ curl -sSfL https://github.com/actions/python-versions/releases/download/3.12.5-10375840348/python-3.12.5-linux-24.04-arm64.tar.gz | tar -zt | grep -E 'site-packages/[^/]+/$'
./lib/python3.12/site-packages/pip-24.2.dist-info/
./lib/python3.12/site-packages/pip/

However, the builds performed by this project configure Python using --without-ensurepip, and then manually install pip and setuptools afterwards: https://github.com/indygreg/python-build-standalone/blob/90d40b46d929eabafee1e3320ab54d32efdf258e/cpython-unix/build-cpython.sh#L546-L547

This means that when Python 3.12 builds were added in 2a5f3a9d4a4c1b15c2bf8db26a042d1d1efb29d6 they didn't automatically pick up this upstream CPython change.

For parity with upstream Python's behaviour, the builds here should stop explicitly installing setuptools for Python 3.12+.

See also:

• https://github.com/python/cpython/issues/95299
• https://github.com/python/cpython/commit/ece20dba120a1a4745721c49f8d7389d4b1ee2a7
• https://github.com/pypa/virtualenv/issues/2487
• https://github.com/astral-sh/uv/pull/1613
• https://github.com/pypa/get-pip/pull/218
• https://github.com/docker-library/python/pull/954

[edmorley]

I've opened #301 to fix this.

[indygreg]

Hi Ed. Thanks for pointing out the discrepancy between our distributions and the official CPython ones.

Aside from parity with CPython, can you articulate any harms from continuing to ship setuptools?

When creating this project several years ago, I made the decision to distribute a modern pip and setuptools with the distributions - newer versions than what CPython's distributions were shipping via ensurepip - because I thought it was in the best interest of users. Not having pip/setuptools installed by default is just annoying to end-users because often one of the first things done with a Python distribution is install something, and this usually required pip/setuptools.

Fast forward a few years. The Python packaging space has definitely evolved. pyproject.toml is now a thing and popular. We have alternate build backends. And tools like uv are challenging the definition of what Python workflows can look like.

I still think it makes sense to have setuptools in the base distribution because a lot of packages out there still use setuptools as their build backend. Since pyproject.toml doesn't provide a way to pin content digests, build backend installation is somewhat insecure because it goes to the internet without end-to-end content integrity checking.

I think that having setuptools in the distribution may actually benefit end-users by eliminating a potential source of insecurity by having to go to the internet and download setuptools without content integrity protection.

Or do modern build frontends and their build isolation always need a setuptools wheel and thus always attempt to download a setuptools wheel, ignoring the setuptools install already in the distribution? If so, then I can possibly be convinced that setuptools in the distribution is adding no value (in 3.12+) and should be jettisoned.

[edmorley]

Aside from parity with CPython, can you articulate any harms from continuing to ship setuptools?

Mainly that inconsistency within the ecosystem is potentially confusing for end users, and also unfairly penalises tools/environments that are trying to do the right thing and already match the stdlib's behaviour. As a secondary benefit, dropping setuptools reduces the download size and size of the install on disk (or in image).

Or do modern build frontends and their build isolation always need a setuptools wheel and thus always attempt to download a setuptools wheel, ignoring the setuptools install already in the distribution?

Good question :-)

As of pip v23.1+, pip will now default to PEP-517 style build isolation if the wheel package is not installed: https://github.com/pypa/pip/pull/11871

(This is on top of the pip v22.1+ behaviour of using build isolation if setuptools is not installed: https://github.com/pypa/pip/pull/10717)

As such, given that the Python builds produced by this repo don't include wheel, pip will already be using build isolation (for projects still using setup.py) and so re-downloading setuptools for build-time usage already.

Plus for projects that have switched to an explicit pyproject.toml backend, they will always use build isolation (regardless of whether wheel is installed) and redownload setuptools even if it's installed globally.

[indygreg]

I'm not swayed by inconsistency arguments: Python bootstrapping and packaging has been a hot mess for 2 decades and I think it is important to challenge what CPython and the PyPA ecosystem of tools are doing because clearly there are shortcomings. End-users are harmed by these shortcomings and I would like to avoid perpetuating sub-optimal state if possible.

As such, given that the Python builds produced by this repo don't include wheel, pip will already be using build isolation (for projects still using setup.py) and so re-downloading setuptools for build-time usage already.

My knee jerk upon reading this was oh no, we should distribute wheel to help keep users secure.

Then I read:

Plus for projects that have switched to an explicit pyproject.toml backend, they will always use build isolation (regardless of whether wheel is installed) and redownload setuptools even if it's installed globally.

So I guess the pyproject.toml way is insecure by design and there's not much this project can do about it. This makes me sad the Python ecosystem is footgunning itself with a forced software supply chain weakness. But it is what it is.

Since I've been told pyproject.toml is the future (actually the PyPA tools have been effectively forcing me to adopt it), it would be like fighting back the tide. I'd be in an ivory tower if I thought continuing to ship setuptools would actually do anything meaningful to keep users secure since ~everyone has adopted pyproject.toml.

So unless there's a compelling argument I'm missing, let's drop setuptools from 3.12+ to maintain parity with the larger ecosystem.

[edmorley]

Yeah, for packages that have migrated from setup.py to pyproject.toml installing wheel would make no difference.

In addition, for any users that use these Python builds and always create a virtual environment using either python -m venv or the standalone virtualenv project (both of which no longer install setuptools or wheel by default for Python 3.12+), then they won't be using the system site-packages install of setuptools (or wheel, if it were added) shipped with these builds - even when they install a setup.py using package either.

[zanieb]

Or do modern build frontends and their build isolation always need a setuptools wheel and thus always attempt to download a setuptools wheel, ignoring the setuptools install already in the distribution? If so, then I can possibly be convinced that setuptools in the distribution is adding no value (in 3.12+) and should be jettisoned.

Yeah, in uv we won't pull setuptools from the distribution without explicit opt-in to disable build isolation.

Similarly, uv doesn't install any seed packages into virtual environments by default.

[edmorley]

So I guess the pyproject.toml way is insecure by design and there's not much this project can do about it.

Agreed, though IMO there is definitely something package managers can do about it. And whilst it seems there isn't the appetite or resources to tackle this in pip (xref https://github.com/pypa/pip/issues/12942 / https://github.com/pypa/pip/issues/12956), the comments in https://github.com/astral-sh/uv/issues/5190 make me hopeful that it may be something uv can do in the future :-)