concurrent pip-tools processes may interfere with each other

[tboddyspargo]

My team uses a monorepo that contains a number of different python projects and applications. To simplify dependency management, we use a recursive Makefile strategy to run pip-compile in different directories and potentially with different python versions with a single entrypoint command.

As this process is simple, but can take some time, we've started to invoke make with the -j argument (allowing different targets to be built in parallel. I've recently come to realize that there seems to be some contention or interference between these different parallel invocations of python -m piptools compile.

I realize this is probably fairly uncommon, but I was hopeful that adding more isolation of the dependency resolution process or temporary directories might be a simple solution. Also, please let me know if this is more of a pip issue than a pip-tools issue. I've done some research suggesting that while parallel pip execution isn't a guarantee, it often works. Other than that, the traceback itself was the only thing leading me to raise the issue here.

Environment Versions

1. OS Type: macOS 14
2. Python version: multiple 3.8.16, 3.9.11, 3.11.6
3. pip version: 23.3.1
4. pip-tools version: 7.3.0
5. make version: GNU Make 3.81

Steps to replicate

Apologies for the complicated repro. I've attached a "mini monorepo" example that demonstrates the issue.

compile-error.zip

1. Use pyenv to install the three versions of python listed in the .python-version files.
2. Open a shell session to the compile-error directory (from the unzipped example project)
3. Run make -j
4. If you don't see the error at first, try a couple more times. The repro has been pretty consistent for me.

Expected result

I hoped that these parallel executions wouldn't interfere with each other, not hit exceptions, and produce a compiled requirements list.

Actual result

Collecting hatchling>1.17.0
  Using cached hatchling-1.18.0-py3-none-any.whl (75 kB)
Collecting packaging>=21.3
  Using cached packaging-23.2-py3-none-any.whl (53 kB)
Collecting editables>=0.3
  Using cached editables-0.5-py3-none-any.whl (5.1 kB)
Collecting pluggy>=1.0.0
  Using cached pluggy-1.3.0-py3-none-any.whl (18 kB)
WARNING: Skipping page https://pypi.org/simple/pathspec/ because the GET request got Content-Type: .The only supported Content-Type is text/html
Collecting hatchling>1.17.0
  Using cached hatchling-1.17.1-py3-none-any.whl (75 kB)
ERROR: Cannot install -r /var/folders/8b/15h_47_d0_b3fq29xjvzjzgm0000gn/T/build-reqs-lhhf9h4d.txt (line 1) because these package versions have conflicting dependencies.

The conflict is caused by:
    hatchling 1.18.0 depends on pathspec>=0.10.1
    hatchling 1.17.1 depends on pathspec>=0.10.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
WARNING: You are using pip version 22.0.4; however, version 23.3.1 is available.
You should consider upgrading via the '/private/var/folders/8b/15h_47_d0_b3fq29xjvzjzgm0000gn/T/build-env-r0cqsfyf/bin/python -m pip install --upgrade pip' command.
Traceback (most recent call last):
  File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/piptools/__main__.py", line 19, in <module>
    cli()
  File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/piptools/scripts/compile.py", line 551, in cli
    metadata = project_wheel_metadata(
  File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/build/util.py", line 54, in project_wheel_metadata
    env.install(builder.build_system_requires)
  File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/build/env.py", line 214, in install
    _subprocess(cmd)
  File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/build/env.py", line 79, in _subprocess
    raise e
  File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/site-packages/build/env.py", line 76, in _subprocess
    subprocess.run(cmd, check=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
  File "/Users/tyler/.pyenv/versions/3.9.11/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/private/var/folders/8b/15h_47_d0_b3fq29xjvzjzgm0000gn/T/build-env-r0cqsfyf/bin/python', '-Im', 'pip', 'install', '--use-pep517', '--no-warn-script-location', '-r', '/var/folders/8b/15h_47_d0_b3fq29xjvzjzgm0000gn/T/build-reqs-lhhf9h4d.txt']' returned non-zero exit status 1.

[AndydeCleyre]

I don't know why your resolution conflicts occur, but will say that when running pip-compile in parallel you should use the --cache-dir option (or PIP_TOOLS_CACHE_DIR parameter) to use an isolated cache for each process (see #1083).

[tboddyspargo]

Thanks, @AndydeCleyre! I tried adding that option to our Makefile setup, but I'm unfortunately still getting the ResolutionImpossible exception. I expect that resolution is not, in fact, impossible, but that something else is causing a failure that's currently indistinguishable from a resolution failure. Perhaps this suggests that the issue is more related to how

Based on the traceback and the likelihood of the ResolutionImpossible exception regardless of --cache-dir, I'm suspicious that two separate, parallel pip-tools processes could end up contending for the same temporary build environment (e.g. /private/var/folders/8b/15h_47_d0_b3fq29xjvzjzgm0000gn/T/build-env-r0cqsfyf).

Do you think that's possible or have any other experience/insights about that potential risk area? Is there another mechanism to control the location/isolation of the dependency resolution build environment that results from the call to project_wheel_metadata?