Cannot target Ubuntu 24.04 Noble when foreign

Describe the bug

Building the Docker image fails when targeting ubuntu:noble.

Docker|  > [4/6] RUN PIP_BREAK_SYSTEM_PACKAGES=1     python3 -m pip install --no-cache-dir --upgrade pip setuptools wheel:
Docker| 0.987 Collecting wheel
Docker| 1.014   Downloading wheel-0.43.0-py3-none-any.whl.metadata (2.2 kB)
Docker| 1.064 Downloading setuptools-69.5.1-py3-none-any.whl (894 kB)
Docker| 
Docker| 1.248 Downloading wheel-0.43.0-py3-none-any.whl (65 kB)
Docker| 1.251    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.8/65.8 kB 20.5 MB/s eta 0:00:00
Docker| 1.286 Installing collected packages: wheel, setuptools
Docker| 1.286   Attempting uninstall: wheel
Docker| 1.289     Found existing installation: wheel 0.42.0
Docker| 1.290 ERROR: Cannot uninstall wheel 0.42.0, RECORD file not found. Hint: The package was installed by debian.
Docker| ------
Docker| Dockerfile:25
Docker| --------------------
Docker|   24 |     # ignores this configuration in the distro and allows for installing the latest pip.
Docker|   25 | >>> RUN PIP_BREAK_SYSTEM_PACKAGES=1 \
Docker|   26 | >>>     python3 -m pip install --no-cache-dir --upgrade pip setuptools wheel
Docker|   27 |     
Docker| --------------------
Docker| ERROR: failed to solve: process "/bin/sh -c PIP_BREAK_SYSTEM_PACKAGES=1     python3 -m pip install --no-cache-dir --upgrade pip setuptools wheel" did not complete successfully: exit code: 1
Docker| ------------------------------------------------------------------
Leaving Docker context.

Building Docker image... errored

Error building Docker container image for helloworld.

Steps to reproduce

briefcase build linux system --target ubuntu:noble

Expected behavior

Docker image build is successful for Ubuntu 24.04.

It appears as though PIP_BREAK_SYSTEM_PACKAGES=1 isn't cutting it on Ubuntu 24.04 to ignore the safeguards of overwriting system-managed Python packages.

The root issue isn't immediately clear to me....as I'm not sure if Canonical needs to fix the way it's installing wheel so pip doesn't choke on it....or if pip should be accommodating whatever's going on in Ubuntu 24.04.

Nonetheless, a stopgap is to add --ignore-installed to this pip install command.

Screenshots

No response

Environment

Operating System: pop os 22.04
Python version: 3.10.12
Software versions:
- Briefcase: 0.3.18.dev206+g38fbb22b.d20240419

Logs

briefcase.2024_04_19-18_09_38.package.log

Additional context

No response

The root issue isn't immediately clear to me....as I'm not sure if Canonical needs to fix the way it's installing wheel so pip doesn't choke on it....or if pip should be accommodating whatever's going on in Ubuntu 24.04.

My immediate concern is that Canonical has (ahem) "fixed" their copy of pip. This wouldn't be the first time that they have... taken liberties with Python's published behavior... to fit their own conception of what Python should look like. And then it becomes the Python ecosystem's problem to deal with the fact that "Python on Ubuntu" doesn't behave like Python does anywhere else.

Given Ubuntu's history, it would not surprise me for a moment to discover that some enterprising release manager has disabled PIP_BREAK_SYSTEM_PACKAGES handling.

Nonetheless, a stopgap is to add --ignore-installed to this pip install command.

If this works, my immediate reaction is that is likely not just a stopgap, but the actual solution.

pip and wheel are packaged as ubuntu packages, and will not be updated in the way that the Python ecosystem expects. If you install python3-pip on a 22.04 box today, you currently get pip 22.0.2; python3-wheel installs 0.37.1. That behavior is, for our purposes, broken; we very intentionally want pip and wheel to be updated. The change is being made inside a Docker container, so there's virtually no impact on anything other than Briefcase's own build process.

I did find https://github.com/pypa/pip/issues/11631 which suggests that Fedora is/was patching pip to basically skip the uninstalling of system packages and just continuing on to install in to /usr/local/.

Looking more closely at Ubuntu's packaging of wheel:

In Lunar, they shipped wheel with an egg-info dir and pip gets caught up in a few errors for detecting the /usr env and not having files to uninstall....but it moves on to installing to /usr/local anyway
In Mantic, they moved to a dist-info dir....but because they left the RECORD file there, pip would actually delete the apt-installed version of wheel and then install the requested version in to /usr/local 👀
In Noble, they dropped the RECORD file and now pip just fails with this error

So, the source of this issue may ultimately just be working out the kinks with PIP_BREAK_SYSTEM_PACKAGES.

I reported the issue to Canonical: https://bugs.launchpad.net/ubuntu/+source/wheel/+bug/2063151

beeware / briefcase