Closed MycroftKang closed 1 year ago
iamjpotts
commented
1 year ago I just ran into this on an Azure DevOps pipeline run from the ubuntu-latest vmPool. It did not happen in the Docker containers that were being built based on python:3.9-slim images, and did not happen on the same pipeline an hour earlier.
Worked around it by downgrading to poetry<1.4
kaqmak
commented
1 year ago We are getting the same issue on our Azure DevOps pipeline with poetry=1.6.1 on image python:3.11-slim-bullseye. Repeatedly rerunning the failed pipeline eventually "solves" the problem.
Lock files are updated, and running poetry self add setuptools does not alleviate the problem.
euri10
commented
1 year ago I tried every other option presented here, all failed.
What striked me is that I have several stages in my gitlab CI, all with the very same poetry install, all were working and just one failed at some point recently.
The only difference is the image used : so I switched from image: python that was the only stage failing in my CI to image: python:3.11 and it is suddently working again.
robbotorigami
commented
1 year ago I spent a couple hours looking into this, and I think I've identified the root cause (or at least a root cause) for this issue. I've set up a test repo, that fairly reliably replicates this in the github actions runner.
I believe the root cause is when multiple setup.py based dependencies are being installed from a source distribution for the first time on a system (in my example, future and restructuredtext-lint). Poetry will build both of these dependencies into a wheel in parallel. For each dependency it identifies that they depend on setuptools, checks if setuptools is in the artifact cache, finds that it isn't, and then attempts to download a distribution for setuptools. Since there are two threads running in parallel trying to download the same file, there is a race condition where one thread succeeds and one fails (I believe this is predominately an issue for windows, with how it handles concurrent access to the file system). For the one that fails, the exception is caught, and pased up the call trace as a status return code until it is silently ignored here: installer.run(). Then, poetry proceeds to attempt to build the distribution, which fails because setuptools was never installed in the virtual environment for that dependency. I modify the executor in my github actions run so that it prints out the stack trace when exceptions are encountered, which can be seen here: Test #14. The relevant section:
Traceback (most recent call last):
File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 767, in _download_link
original_archive = self._download_archive(operation, link)
File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 857, in _download_archive
with atomic_open(archive) as f:
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\contextlib.py", line 142, in __exit__
next(self.gen)
File "C:\Users\runneradmin\.local\venv\lib\site-packages\requests\utils.py", line 307, in atomic_open
os.replace(tmp_name, filename)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\runneradmin\\AppData\\Local\\pypoetry\\Cache\\artifacts\\8d\\aa\\9e\\5ebe452d7977c93fb8e0db884b6b167954ac9814e1f25125b4552d76c8\\tmpui9rw792' -> 'C:\\Users\\runneradmin\\AppData\\Local\\pypoetry\\Cache\\artifacts\\8d\\aa\\9e\\5ebe452d7977c93fb8e0db884b6b167954ac9814e1f25125b4552d76c8\\setuptools-68.2.2-py3-none-any.whl'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 286, in _execute_operation
result = self._do_execute_operation(operation)
File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 387, in _do_execute_operation
result: int = getattr(self, f"_execute_{method}")(operation)
File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 512, in _execute_install
status_code = self._install(operation)
File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 550, in _install
archive = self._download(operation)
File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 754, in _download
return self._download_link(operation, link)
File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 773, in _download_link
cached_file.unlink(missing_ok=True)
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\pathlib.py", line 1[20](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:21)6, in unlink
self._accessor.unlink(self)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\runneradmin\\AppData\\Local\\pypoetry\\Cache\\artifacts\\8d\\aa\\9e\\5ebe452d7977c93fb8e0db884b6b167954ac9814e1f25125b4552d76c8\\setuptools-68.2.2-py3-none-any.whl'
Traceback (most recent call last):
File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 767, in _download_link
original_archive = self._download_archive(operation, link)
File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 857, in _download_archive
with atomic_open(archive) as f:
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\contextlib.py", line 142, in __exit__
next(self.gen)
File "C:\Users\runneradmin\.local\venv\lib\site-packages\requests\utils.py", line 307, in atomic_open
os.replace(tmp_name, filename)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\runneradmin\\AppData\\Local\\pypoetry\\Cache\\artifacts\\be\\01\\ba\\eabeb6a7ad74ee2a0542e0479545034db7365479427d5[26](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:27)7875[27](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:28)6c906\\tmpn385zi8y' -> 'C:\\Users\\runneradmin\\AppData\\Local\\pypoetry\\Cache\\artifacts\\be\\01\\ba\\eabeb6a7ad74ee2a0542e0479545034db7365479427d5267875276c906\\wheel-0.41.2-py3-none-any.whl'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line [28](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:29)6, in _execute_operation
result = self._do_execute_operation(operation)
File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 387, in _do_execute_operation
result: int = getattr(self, f"_execute_{method}")(operation)
File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 512, in _execute_install
status_code = self._install(operation)
File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 550, in _install
archive = self._download(operation)
File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 754, in _download
return self._download_link(operation, link)
File "C:\Users\runneradmin\.local\venv\lib\site-packages\poetry\installation\executor.py", line 773, in _download_link
cached_file.unlink(missing_ok=True)
File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\pathlib.py", line 1206, in unlink
self._accessor.unlink(self)
PermissionError: [WinError [32](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:33)] The process cannot access the file because it is being used by another process: 'C:\\Users\\runneradmin\\AppData\\Local\\pypoetry\\Cache\\artifacts\\be\\01\\ba\\eabeb6a7ad74ee2a0542e04795450[34](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:35)db7[36](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:37)5479427d5267875276c906\\wheel-0.[41](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:42).2-py3-none-any.whl'
ChefBuildError
Backend 'setuptools.build_meta:__legacy__' is not available.
at ~\.local\venv\lib\site-packages\poetry\installation\chef.py:147 in _prepare
1[43](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:44)|
1[44](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:45)| error = ChefBuildError("\n\n".join(message_parts))
1[45](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:46)|
1[46](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:47)| if error is not None:
> 1[47](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:48)| raise error from None
1[48](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:49)|
1[49](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:50)| return path
1[50](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:51)|
1[51](https://github.com/robbotorigami/poetry-bug-7611/actions/runs/6412718462/job/17410500811#step:6:52)| def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:
Note: This error originates from the build backend, and is likely not a problem with poetry but with future (0.18.3) not supporting PEP 517 builds. You can verify this by running 'pip wheel --use-pep517 "future (==0.18.3)"'.Which shows the errors resulting from concurrent access to the artifact cache
Note that when replicating this issue setuptools needs to not be present already in the artifact cache (which is presumably why this is mostly affecting CI, and why pre installing setuptools fixes it in some cases). poetry cache clear --all . doesn't seem to remove it, so I've been manually removing the folder when I've been testing this.
I'm not exactly sure how this problem should be solved. The right solution probably involves some sort of locking mechanism for the artifact cache being implemented to prevent concurrent downloads of the same distribution package. The return value of installer.run should probably be checked in some manner to aid in future debugging. It could also be used in a retry loop, though that seems like a much hackier solution than locking.
Another workaround that should address the issue is:
poetry config installer.max-workers 1since this will prevent two builds from happening in parallel (this will likely be much slower than running poetry install repeatedly though, so might not actually be of value).
radoering
commented
1 year ago Great finding. Sounds very plausible. 👍
The right solution probably involves some sort of locking mechanism for the artifact cache being implemented to prevent concurrent downloads of the same distribution package.
Agreed.
The return value of installer.run should probably be checked in some manner to aid in future debugging.
We have already added better logging in master (see #8479), but still had no idea why the installation of build dependencies fails.
iamjpotts
commented
1 year ago I started seeing the failures in python:3.9-slim based images also. On the Azure CI VM, setup tools was already installed, so that wasn't a viable workaround.
@robbotorigami this works so far:
poetry config installer.max-workers 1Thanks to @robbotorigami (https://github.com/python-poetry/poetry/issues/7611#issuecomment-1747836233) I was able to reproduce the issue and provide a fix in #8517. Feel free to try the PR.
MattOates
commented
1 year ago @radoering I tied the PR above locally and it doesn't resolve the issue for me locally. But neither do any of the proposed work arounds mentioned in this thread to date. I also have the issue irregardless of poetry and python version combo. So I suspect there is some other environment things at play too. Im on a very fresh macOS setup.
robbotorigami
commented
1 year ago I've tested #8517 in the CI workflows where I was seeing issues, and I no longer see any failures. The fix for the concurrency issue seems sensible and should resolve any issues arising from concurrent downloads. Based on the comments in this thread, it seems likely that this was the issue that most (but not all) of the users encountering this error were running into. I suspect that for those having success with the suggested workarounds, this PR will solve the issue.
inducer
commented
1 year ago Sorry to (possibly?) be the bearer of bad news here: This build appears to fail (on Windows) with a very similar symptom to what is described here, despite using poetry 1.7.0, which contains #8517. :thinking:
� Installing django-codemirror-widget (0.5.0)
ChefBuildError
Backend 'setuptools.build_meta:__legacy__' is not available.
Traceback (most recent call last):
File "C:\Users\runneradmin\.local\venv\Lib\site-packages\pyproject_hooks\_in_process\_in_process.py", line 77, in _build_backend
obj = import_module(mod_path)
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\hostedtoolcache\windows\Python\3.12.0\x64\Lib\importlib\__init__.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
File "<frozen importlib._bootstrap>", line 1304, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1381, in _gcd_import
File "<frozen importlib._bootstrap>", line 1354, in _find_and_load
File "<frozen importlib._bootstrap>", line 1325, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 929, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 994, in exec_module
File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
File "C:\Users\RUNNER~1\AppData\Local\Temp\tmp5cl80_pe\.venv\Lib\site-packages\setuptools\__init__.py", line 8, in <module>
import distutils.core
ModuleNotFoundError: No module named 'distutils'
at ~\.local\venv\Lib\site-packages\poetry\installation\chef.py:166 in _prepare
162|
163| error = ChefBuildError("\n\n".join(message_parts))
164|
165| if error is not None:
> 166| raise error from None
167|
168| return path
169|
170| def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:
Note: This error originates from the build backend, and is likely not a problem with poetry but with django-codemirror-widget (0.5.0) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "django-codemirror-widget (==0.5.0)"'.Notably, the suggested command succeeds:
pip wheel --no-cache-dir --use-pep517 "django-codemirror-widget (==0.5.0)" andreas@arc 15:25
/home/andreas/src/env-3.12/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_envs.py:111: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
from pip._vendor.pkg_resources import find_distributions
Collecting django-codemirror-widget==0.5.0
Downloading django-codemirror-widget-0.5.0.tar.gz (6.6 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: django-codemirror-widget
Building wheel for django-codemirror-widget (pyproject.toml) ... done
Created wheel for django-codemirror-widget: filename=django_codemirror_widget-0.5.0-py3-none-any.whl size=7918 sha256=994b804c1d4228457b59ce78a09d2be116f56aeba47a5c968504343582afb26c
Stored in directory: /tmp/pip-ephem-wheel-cache-1cu5q9ol/wheels/35/bd/a5/48fb66e57f3161108f210d9dbcb36732bf7afcd6e63a4f38da
/home/andreas/src/env-3.12/lib/python3.12/site-packages/pip/_internal/metadata/importlib/_dists.py:73: DeprecationWarning: Unimplemented abstract methods {'locate_file'}
return cls(files, info_location)
Successfully built django-codemirror-widget
scottshambaugh
commented
1 year ago I am also continuing to see the same build failure here on Windows, Ubuntu, and MacOSX despite using poetry 1.7.0 which contained the supposed fix, see this workflow: https://github.com/scottshambaugh/monaco/actions/runs/6763933260/job/18381632398
timwie
commented
1 year ago It's a different error, the key difference being ModuleNotFoundError: No module named 'distutils'. That's because distutils is removed in Python 3.12. @scottshambaugh this is fixed in numpy 1.26.
scottshambaugh
commented
1 year ago Thank you @timwie! That did indeed resolve my error.
dimbleby
commented
11 months ago that is not even close to being the same error: and by reproducing it with pip you have anyway proved that it is not a poetry problem.
PedroH183
commented
9 months ago Same issue here, the Interim Solutions works for me.
github-actions[bot]
commented
8 months ago This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
windows-2022GitHub runner)-vvvoption) and have included the output below.Issue
Installing a package that depends on setuptools fails intermittently. This bug does not occur in poetry 1.3 and is found in poetry 1.4.
View full logs
Interim Solutions
There are several workarounds below that mitigate this issue.
Downgrade poetry version to 1.3
Try
poetry installseveral times