Backend subprocess exited when trying to invoke get_requires_for_build_wheel

[Opalo]

I have a custom pre-commit hook that runs pip-compile wrapped in a docker container. Out of nowhere it started to fail with the following message:

Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Failed to parse /app/pyproject.toml

The command I use to run the pip-compile is:

pip-compile --allow-unsafe --verbose --extra-index-url=https://pypi.org/simple --index-url=http://pypi.data.internal --output-file=pyproject.lock --trusted-host=pypi.data.internal pyproject.toml

while the Dockerfile is quite simple:

#syntax = docker/dockerfile:experimental
ARG AIRFLOW_VERSION=2.7.1
ARG PYTHON_VERSION=3.9

FROM apache/airflow:${AIRFLOW_VERSION}-python${PYTHON_VERSION}

USER root

RUN apt-get update -qy
RUN apt-get install -qy default-libmysqlclient-dev pkg-config build-essential

USER airflow

RUN pip install --upgrade pip
RUN pip install pip-tools==7.3.0

ENV PIP_USER="false"

VOLUME /app

WORKDIR /app

ENTRYPOINT []

and pyproject.toml seems to have nothing suspicious:

[build-system]
requires = [
  "setuptools>=63.0.0",
  "wheel"
]
build-backend = "setuptools.build_meta"

[project]
name = "pipeline_commerce"
version = "2.7.1+3.0.10"
description = "Supporting code for DAGs from 'commerce' site."
readme = "README.md"
authors = [
  {name = "LOL", email = "lol@lol.com"}
]
classifiers = []
dependencies = [
  "airflow-provider-fivetran-async==1.1.2",
  "apache-airflow-providers-mysql==5.3.1",
  "lol-airflow-dbt==2.7.1+3.0.10" # an internal lib resolved through pypi.data.internal
]
requires-python = ">=3.9"

[project.optional-dependencies]
dev = [
  "pytest==7.3.0"
]

The whole pip-compile output is:

airflow@11c9338b1bbe:/app$ pip-compile --allow-unsafe --verbose --extra-index-url=https://pypi.org/simple --index-url=http://pypi.data.internal --output-file=pyproject.lock --trusted-host=pypi.data.internal pyproject.toml
/home/airflow/.local/lib/python3.9/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
Using pip-tools configuration defaults found in 'pyproject.toml'.
adding trusted host: 'pypi.data.internal' (from line 9 of pyproject.lock)
Creating virtualenv isolated environment...
find interpreter for spec PythonSpec(path=/usr/local/bin/python)
proposed PythonInfo(spec=CPython3.9.18.final.0-64, exe=/usr/local/bin/python, platform=linux, version='3.9.18 (main, Sep  7 2023, 13:38:29) \n[GCC 10.2.1 20210110]', encoding_fs_io=utf-8-utf-8)
create virtual environment via CPython3Posix(dest=/tmp/build-env-tau5pks5, clear=False, no_vcs_ignore=False, global=False)
add seed packages via FromAppData(download=False, pip=bundle, via=copy, app_data_dir=/home/airflow/.local/share/virtualenv)
changing mode of /tmp/build-env-tau5pks5/bin/pip3.9 to 755
changing mode of /tmp/build-env-tau5pks5/bin/pip to 755
changing mode of /tmp/build-env-tau5pks5/bin/pip3 to 755
changing mode of /tmp/build-env-tau5pks5/bin/pip-3.9 to 755
Installing packages in isolated environment... (setuptools>=63.0.0)
Getting build dependencies for wheel...
Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Failed to parse /app/pyproject.toml

What's interesting it works correctly on macOS and fails on Linux GH action worker. Nothing (I'm aware of) has changed in the setup. I don't even know how to debug that although I was already debugging it for a couple of hours.

Environment Versions

1. OS Type: Linux 11c9338b1bbe 5.10.198-187.748.amzn2.x86_64 #1 SMP Tue Oct 24 19:49:54 UTC 2023 x86_64 GNU/Linux and Debian GNU/Linux 11 \n \l
2. Python version: Python 3.9.18
3. pip version: pip 23.3.1 from /home/airflow/.local/lib/python3.9/site-packages/pip (python 3.9)
4. pip-tools version: pip-compile, version 7.3.0 (I also got the following warning when running pip-compile --version:

   /home/airflow/.local/lib/python3.9/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
   warnings.warn("Setuptools is replacing distutils.") 

Steps to replicate

1. docker build -t pyproject-lock-pipelines .
2. docker run --entrypoint /bin/bash -v $PWD:/app -it pyproject-lock-pipelines
3. pip-compile --allow-unsafe --verbose --output-file=pyproject.lock pyproject.toml

Expected result

Locks are generated correctly.

Actual result

It fails with the message as above.

[Opalo]

It was related to missing write permissions. The docker container could write to the host and it resulted in such error. But the error from build invocation is very cryptic. What helped was running python -m build since it explicitly wrote that there're no write permissions.