pip-compile fails to generate lock file with airflow docker image

[Opalo]

I want to generate a lock file using airlfow dockerfile. The reason for that is that the lock file be used to install the dependencies later on on the exactly the same file. Unfortunately the pip-compile fails with:

ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.
[notice] A new release of pip is available: 23.2 -> 23.3.1
[notice] To update, run: python -m pip install --upgrade pip
Traceback (most recent call last):
  File "/home/airflow/.local/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/home/airflow/.local/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/airflow/.local/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/airflow/.local/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/airflow/.local/lib/python3.9/site-packages/piptools/scripts/compile.py", line 551, in cli
    metadata = project_wheel_metadata(
  File "/home/airflow/.local/lib/python3.9/site-packages/build/util.py", line 47, in project_wheel_metadata
    env.install(builder.build_system_requires)
  File "/home/airflow/.local/lib/python3.9/site-packages/build/env.py", line 145, in install
    _subprocess(cmd)
  File "/home/airflow/.local/lib/python3.9/site-packages/build/env.py", line 64, in _subprocess
    subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
  File "/usr/local/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/tmp/build-env-kboqypie/bin/python', '-Im', 'pip', 'install', '--use-pep517', '--no-warn-script-location', '-r', '/tmp/build-reqs-yjljfkcu.txt']' returned non-zero exit status 1.

Environment Versions

1. OS Type

   airflow@d0abc91dfc5f:/app$ uname -a
   Linux d0abc91dfc5f 6.1.51-0-virt #1-Alpine SMP Mon, 04 Sep 2023 08:04:05 +0000 aarch64 GNU/Linux
   airflow@d0abc91dfc5f:/app$ cat /etc/issue
   Debian GNU/Linux 11 \n \l

2. Python version: Python 3.9.18
3. pip version: pip 23.0.1 from /usr/local/lib/python3.9/site-packages/pip (python 3.9)
4. pip-tools version: 7.3.0

Steps to replicate

1. Use the dockerfile below:

   
   #syntax = docker/dockerfile:experimental
   ARG AIRFLOW_VERSION=2.7.1
   ARG PYTHON_VERSION=3.9

FROM apache/airflow:${AIRFLOW_VERSION}-python${PYTHON_VERSION}

RUN pip install pip-tools==7.3.0

VOLUME /app

WORKDIR /app

ENTRYPOINT []

2. And the `pyproject.toml` below:

[build-system] requires = [ "setuptools>=63.0.0", "wheel" ] build-backend = "setuptools.build_meta"

[project] name = "whatever" version = "1.0.0" description = "LOL" dependencies = [ "apache-airflow-providers-presto==5.1.4" ] requires-python = ">=3.9"

3. Build the docker image:

docker build -t lol .

4. Run it with:

docker run --rm -v $PWD:/app --entrypoint /bin/bash -it lol

and run:

pip-compile -o pyproject.lock pyproject.toml

inside container.

#### Expected result

`pyproject.lock` file get generated.

#### Actual result

airflow@0e10011b11cc:/app$ pip-compile -o pyproject.lock pyproject.toml ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

[notice] A new release of pip is available: 23.2 -> 23.3.1 [notice] To update, run: python -m pip install --upgrade pip Traceback (most recent call last): File "/home/airflow/.local/bin/pip-compile", line 8, in sys.exit(cli()) File "/home/airflow/.local/lib/python3.9/site-packages/click/core.py", line 1157, in call return self.main(args, kwargs) File "/home/airflow/.local/lib/python3.9/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/home/airflow/.local/lib/python3.9/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/airflow/.local/lib/python3.9/site-packages/click/core.py", line 783, in invoke return __callback(args, *kwargs) File "/home/airflow/.local/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), args, **kwargs) File "/home/airflow/.local/lib/python3.9/site-packages/piptools/scripts/compile.py", line 551, in cli metadata = project_wheel_metadata( File "/home/airflow/.local/lib/python3.9/site-packages/build/util.py", line 47, in project_wheel_metadata env.install(builder.build_system_requires) File "/home/airflow/.local/lib/python3.9/site-packages/build/env.py", line 143, in install _subprocess(cmd) File "/home/airflow/.local/lib/python3.9/site-packages/build/env.py", line 64, in _subprocess subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT) File "/usr/local/lib/python3.9/subprocess.py", line 528, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/tmp/build-env-0phmpzua/bin/python', '-Im', 'pip', 'install', '--use-pep517', '--no-warn-script-location', '-r', '/tmp/build-reqs-6bv05q8q.txt']' returned non-zero exit status 1.

Does anybody know what is the reason for that?

I've also tried `--no-build-isolation` option to no avail: 

ModuleNotFoundError: No module named 'poetry'

which is even worse.

[webknjaz]

ERROR: Can not perform a '--user' install. User site-packages are not visible in this virtualenv.

If I were to guess, you probably have the PIP_USER=1 set somewhere.

[Opalo]

That was it, very helpful! Thanks you very much!!