Can't call host tools that are written in Python and use system packages

[lazka]

Upstream issue https://gitlab.gnome.org/GNOME/pygobject/-/issues/622

First of, we are going to remove this usage upstream, so not that big of a problem right now.

Problem (from what I understand):

• use a meson project that uses for example import('gnome').generate_gir()
• install the package with pip
• generate_gir() internally calls /usr/bin/g-ir-scanner which has a shebang of /usr/bin/python3
• pip sets a PYTHONPATH and sitecustomize.py which prevents any imports of system site-packages of all Python things called during the build
• /usr/bin/g-ir-scanner can't import its dependency modules and fails

Possible solution:

The only idea I could come up with is that pip itself would need to detect when the interpreter used isn't the one of the isolated build env (in sitecustomize.py) and reverts its restrictions and unsets the pythonpath: https://github.com/pypa/pip/blob/7881c5303c9afa152bb9938a2751e3df4f1077bc/src/pip/_internal/build_env.py#L97-L156

[dnicolodi]

How is this a meson-python issue? If your analysis is correct, there isn't anything that meson-python can do about it. The default pip behavior is to build the package in an isolated Python environment, thus what you observe is pretty much by design. You can disable this with the --no-build-isolation pip option. Another option is to make g-ir-scanner into a Python package and add it to the pygobject's build-system.requires.

[lazka]

It maybe isn't. I just wasn't sure where to file this otherwise.

Would you agree that pip could be smarter about only isolating things it owns?

Feel free to close this.

(both solutions aren't possible, g-ir-scanner needs to match with the version of the system libraries, and disabling isolation breaks because of missing build dependencies)

[dnicolodi]

Would you agree that pip could be smarter about only isolating things it owns?

I'm not sure there is a way for pip to be smarter. I assume things are done the way they are for a reason.

(both solutions aren't possible, g-ir-scanner needs to match with the system libraries, and disabling isolation breaks because of missing build dependencies)

Disabling build isolation requires to take care of the build dependencies, indeed, but I don't see alternatives.

[lazka]

ok, thanks

Let's hope no toolchain starts using Python in the future :P

[dnicolodi]

Let's hope no toolchain starts using Python in the future :P

I think your use case and your concerns are very valid. But you would need to bring them up with the pip developers to see whether there is a possible solution. Dependencies on system tools and library are impossible or at least very difficult to manage in Python packages. @rgommers is doing some work in this area.

[eli-schwartz]

In theory, build isolation could use an actual venv with a python executable in the subprocess PATH. I am guessing (but don't know for sure) that they opted to implement build isolation using PYTHONPATH because it's a bit more lightweight?

[dnicolodi]

IIRC build uses a real venv for the build isolation environment. @lazka do some of the problems go away if you build the wheel with build?

[lazka]

I can confirm that pypa/build works:

FROM ubuntu:24.04

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install --no-install-recommends -y \
    python3-dev \
    python3-pip \
    libcairo-dev \
    gobject-introspection \
    libgirepository-1.0-dev \
    gir1.2-glib-2.0 \
    gir1.2-freedesktop-dev \
    git

RUN git clone https://gitlab.gnome.org/GNOME/pygobject && cd pygobject && git checkout 3.48.0
RUN pip install --break-system-packages build==1.1.1

# broken:
# RUN cd pygobject && pip wheel .

# works:
RUN cd pygobject && pyproject-build .