Closed apollo13 closed 3 years ago
Verified that this issue also exists on beta 5
I think I have a problem that might be related to this issue.
I have a service which includes a dependency that pins setuptools
to 40.8.0 and doing poetry install
fails the first time with:
[EnvCommandError]
Command ['/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/bin/python', '-m', 'pip', 'install', '-e', '/home/circleci/project'] errored with the following return code 2, and output:
Looking in indexes: https://pypi.org/simple, ***************************************************
Obtaining file:///home/circleci/project
ERROR: Exception:
Traceback (most recent call last):
File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 407, in check_if_exists
self.satisfied_by = pkg_resources.get_distribution(str(no_marker))
File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 481, in get_distribution
dist = get_provider(dist)
File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 357, in get_provider
return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 900, in require
needed = self.resolve(parse_requirements(requirements))
File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 791, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pip._vendor.pkg_resources.ContextualVersionConflict: (setuptools 41.2.0 (/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages), Requirement.parse('setuptools==40.8.0'), {'usher-events'})
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 188, in main
status = self.run(options, args)
File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 345, in run
resolver.resolve(requirement_set)
File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 196, in resolve
self._resolve_one(requirement_set, req)
File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 359, in _resolve_one
abstract_dist = self._get_abstract_dist_for(req_to_install)
File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 291, in _get_abstract_dist_for
req, self.require_hashes, self.use_user_site, self.finder,
File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 255, in prepare_editable_requirement
req.check_if_exists(use_user_site)
File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 418, in check_if_exists
self.req.name
File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 481, in get_distribution
dist = get_provider(dist)
File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 357, in get_provider
return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 900, in require
needed = self.resolve(parse_requirements(requirements))
File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 791, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pip._vendor.pkg_resources.ContextualVersionConflict: (setuptools 41.2.0 (/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages), Requirement.parse('setuptools==40.8.0'), {'usher-events'})
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]
Exited with code 1
With the following pyproject.toml
[tool.poetry]
name = "usher-events"
version = "0.0.0"
description = "Blahblah"
authors = ["Blabla <hackers@blabla.co>"]
packages = [{ include = "usher_events" }]
[tool.poetry.dependencies]
python = "^3.7"
aws-xray-sdk = "^2.4.2"
botocore = "^1.12.237"
colorama = "^0.4.1"
datadog_lambda = "^0.6.0"
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
If I do poetry install a second time it works without issue. The problem is that on CI it fails every time as CI is always starting fresh. I have poetry 0.12.17 and python 3.7.4.
Can confirm that setuptools as a dependency does not appear to be handled properly. There's a package that depends on setuptools in my lockfile, but there is not associated entry in the lockfile for setuptools itself.
Lockfile snippet:
[[package]]
category = "main"
description = "Library to implement a well-behaved Unix daemon process."
name = "python-daemon"
optional = false
python-versions = "*"
version = "2.2.4"
[package.dependencies]
docutils = "*"
lockfile = ">=0.10"
setuptools = "*"
This issue appears to affect pip
as a package too.
This:
diff --git a/poetry/puzzle/provider.py b/poetry/puzzle/provider.py
index a077e28..1b13212 100644
--- a/poetry/puzzle/provider.py
+++ b/poetry/puzzle/provider.py
@@ -60,7 +60,7 @@ class Indicator(ProgressIndicator):
class Provider:
- UNSAFE_PACKAGES = {"setuptools", "distribute", "pip"}
+ UNSAFE_PACKAGES = {}
def __init__(self, package, pool, io): # type: (Package, Pool, Any) -> None
self._package = package
fixes my issue. Now the question is: Why are those unsafe and what are the consequences?
It seems there's not too much reasoning behind this (https://github.com/pypa/pip/issues/6459)
At the very least, we should introduce an --allow-unsafe
or similar flag, if not remove the unsafe packages hard-coding altogether.
Also having this issue with 2 packages that require separate versions of setuptools. (pathvalidate and google-auth).
Install works the first time, but when install
is ran again with a cached virtualenv (same lockfile, no changes to pyproject.toml):
[EnvCommandError]
Command ['/root/repo/.venv/bin/pip', 'install', '-e', '/root/repo'] errored with the following return code 2, and output:
Obtaining file:///root/repo
Exception:
Traceback (most recent call last):
File "/root/repo/.venv/lib/python3.6/site-packages/pip/req/req_install.py", line 1025, in check_if_exists
self.satisfied_by = pkg_resources.get_distribution(str(no_marker))
File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 558, in get_distribution
dist = get_provider(dist)
File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 432, in get_provider
return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 968, in require
needed = self.resolve(parse_requirements(requirements))
File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 859, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pip._vendor.pkg_resources.ContextualVersionConflict: (setuptools 39.0.1 (/root/repo/.venv/lib/python3.6/site-packages), Requirement.parse('setuptools>=40.3.0'), {'google-auth'})
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/repo/.venv/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/root/repo/.venv/lib/python3.6/site-packages/pip/commands/install.py", line 324, in run
requirement_set.prepare_files(finder)
File "/root/repo/.venv/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/root/repo/.venv/lib/python3.6/site-packages/pip/req/req_set.py", line 521, in _prepare_file
req_to_install.check_if_exists()
File "/root/repo/.venv/lib/python3.6/site-packages/pip/req/req_install.py", line 1036, in check_if_exists
self.req.name
File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 558, in get_distribution
dist = get_provider(dist)
File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 432, in get_provider
return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 968, in require
needed = self.resolve(parse_requirements(requirements))
File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 859, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pip._vendor.pkg_resources.ContextualVersionConflict: (setuptools 39.0.1 (/root/repo/.venv/lib/python3.6/site-packages), Requirement.parse('setuptools>=40.3.0'), {'google-auth'})
I'm assuming this is what's happening:
setuptools 40.3.0
is installed to satisfy the requirement for google-auth
.setuptools 39.0.1
then is installed and satisfies the requirement for pathvalidate
, overwriting the one installed for google-auth.Once reran, the dependency already exists for google-auth, but it's not the right version, and thus causes the above error.
Just a guess, though.
I am considering opening a PR for this. I plan to change the resolving behaviour so that the UNSAFE_PACKAGES
are considered in the same way as any others but block them from being installed/exported unless an --allow-unsafe
flag is set.
This will be my first poetry contribution so any advice, e.g. wrt anyone worth running this by before I start work, would be much appreciated.
This:
diff --git a/poetry/puzzle/provider.py b/poetry/puzzle/provider.py index a077e28..1b13212 100644 --- a/poetry/puzzle/provider.py +++ b/poetry/puzzle/provider.py @@ -60,7 +60,7 @@ class Indicator(ProgressIndicator): class Provider: - UNSAFE_PACKAGES = {"setuptools", "distribute", "pip"} + UNSAFE_PACKAGES = {} def __init__(self, package, pool, io): # type: (Package, Pool, Any) -> None self._package = package
fixes my issue. Now the question is: Why are those unsafe and what are the consequences?
@sdispater ?
Hello. Packaging setuptools may be needed for some reason poetry may not be aware of, so it is important to provide an option to not ignore it
Pinning it inside the virtualenv is also important. pipenv did the mistake of automatically installing the newer version of setuptools on new environment bootstrap, leaving it imposible to pin to a previous, working version in the lockfile (45.0.0 broke something, for a while it was impossible to go back to a version <45). So please allow pinning setuptools and other "special" packages (pip, poetry itself,...).
The reason why they are marked as "unsafe" is because Poetry relies on them internally (well mostly pip
).
However, Poetry will eventually no longer use them and when that happens this restriction will no longer be needed. Until then, I prefer that we leave it as it is to avoid unexpected side effects.
@sdispater, thanks for getting back to us. Did you see my proposal?
change the resolving behaviour so that the UNSAFE_PACKAGES are considered in the same way as any others but block them from being installed/exported unless an --allow-unsafe flag is set.
This would seem to address problem of locking a set of packages that are inconsistent due to the unsafe packages being completely ignored at resolve time without affecting the default install/export behaviour. Would you support a PR implementing this?
Sorry to push the subject but it's causing us significant pain right now: we're having to add dummy constraints updated manually to workaround this. It's a high maintenance and patchy state of affairs.
Any word on this? Just encountered this same issue; it'd be nice if there were a solution.
Same here. I found VersionConflict
with the pip
and setuptools
versions when trying to install tensorflow
and tensorboard
from poetry. The only workaround I found was to substitute the poetry install
by:
poetry run pip install -U pip
poetry run pip install -U setuptools
poetry install
This is not optimal as the pip and setuptools versions are not locked. Any solution to this? Seems to be a critical bug...
Issue #1651 seems to be tackling this point, although no activity since February
Same situation here as @ivallesp , tensorboard
requires setuptools >= 41.0.0
but for some reason, 40.8.0
gets installed. This gives me a ContextualVersionConflict
runtime error from pkg_resources
.
Even adding a fake dependency to my project for a more recent setuptools doesn't help, nothing gets installed when I run poetry.
Possibly poetry just should install and manage its setuptools and pip requirements in different/edited/hacked sys.path
(...venv/lib/poetry-packages/ instead of site-packages), and then the venv itself can have no restrictions. IE: poetry should never have 'unsafe' packages, because its environment is separately managed by itself. That way we don't have to wait until poetry no longer uses setuptools and/or pip. A side effect of this is that poetry can then bootstrap to use any support packages it needs (arrow or other fun things), because they won't be a problem.
@earonesty brings up an interesting point. I'd argue that poetry
itself is usually installed globally or in another venv (ie via pipx
) so it should not conflict with anything in the venv at all. Does that address some of your concerns @sdispater? I am asking because this issue is the final show stopper for us to embrace poetry fully. I understand the motivation behind your argument, but if poetry
isn't in the managed venv anyways, there are far less chances of issues. On the other hand, everyone installing gunicorn
or tensorflow
currently has problems with poetry
.
@insysion Did you open a PR?
This is a serious bug. It causes the installation of older, buggier versions of setuptools like in https://github.com/pypa/setuptools/issues/1963. There's no reason to break such projects.
I'd be happy to try to open a PR if people like the idea of putting poetry deps in another folder (. poetry_deps ?)
note: i suspect poetry's dependence on msgpack also causes problems for the same reason. there's no way to alter msgpack's version when poetry is running. ie: you need to be sure the msgpack you run is identical to poetry's or you will fail on machines, like windows, that lock shared libs while in use.
Poetry's recommended installation approaches (get-poetry.py, pip install --user
, and pipx install
) all keep Poetry's stuff in a separate place from where this code is working; unless you also set poetry config virtualenvs.create false
and don't have separate envs created, then Poetry's dependency resolution would be resolving for a separate virutal-env anyway.
So it seems like it should be perfectly safe to set UNSAFE_PACKAGES = {}
when running inside a project-specific virtual env that is separate from Poetry's virtual env (if it has one).
@earonesty: I suspect that msgpack problem is different, as it's part of poetry self update
, and that is operating on the Poetry venv. There should be no conflict between Poetry's msgpack and a project's msgpack version, unless the user is deliberately putting the project and Poetry into the same venv.
Poetry's recommended installation approaches (get-poetry.py,
pip install --user
, andpipx install
) all keep Poetry's stuff in a separate place from where this code is working; unless you also setpoetry config virtualenvs.create false
and don't have separate envs created, then Poetry's dependency resolution would be resolving for a separate virutal-env anyway.
@TBBle Though Poetry itself runs from its own virtualenv, I was under the impression (based on the existence of things like UNSAFE_PACKAGES
and previous discussion related to this topic) that when performing operations on the project virtualenv itself, it effectively invokes pip
(and therefore setuptools
and wheel
) from the project virtualenv. I'm not 100%, tbh.
Ah, right, I didn't realise that.
I would hope that its dependency on pip
etc. in the project env would not be as-strict as for poetry's own venv though. So UNSAFE_PACKAGES
could therefore be reapproached as a set of dependencies (versioned matches, presumably something like pip >= ...
for pip flags used by Poetry) for the project env, which should be mixed into the project's own dependencies and resolved like any other overlapping requirements.
To resolve this problem, it would need to be able to distinguish when the pip
, setuptools
, and wheel
dependencies are only from using Poetry (and should hence be excluded from poetry export
), and when they come from the project or one of its dependencies.
For what it’s worth I ran into this while trying to use Snyk on my Poetry projects. Snyk is unusable because of this – snyk test
raises an error when failing to retrieve the setuptools
metadata from poetry.lock
:
Error processing poetry project. Unable to find dependencies in poetry.lock for package: setuptools
I’ve reported this to Snyk over at https://github.com/snyk/snyk/issues/367#issuecomment-757972874 even though it’s clearly a Poetry issue.
Is this behavior still necessary? As a user I would either like poetry
to include dependencies like wheel
when I include it in my pyproject.toml
, to print a warning that this dependency will be ignored, or to allow me to opt-out of this behavior with a flag.
So I did dig through this issue again and came to the following conclusions:
distribute
is a very very old package (2013) that should not cause any problemssetuptools
and wheel
are required to install a source package inside a virtualenv. They are not required when wheels are installed (they are just extracted) or the source package contains a pyproject.toml (this explicitly trigger PEP-517 behavior, see https://pip.pypa.io/en/stable/reference/pip/#pep-517-and-518-support and causes build-isolation to be used, ie pip installs setuptools manually)pip
is needed inside the venv to actually install things; I'd strongly recommend to keep it this way because then pip easily knows what to install where. Please note that pip vendors basically everything (this includes setuptools, see https://github.com/pypa/pip/blob/master/src/pip/_vendor/vendor.txt for the full list) that is required for pip to function.So assuming we want to keep pip
in the list of unsafe packages, I think we can freely remove distribute
and that leaves us with setuptools
& wheel
. As noticed above, installing wheels requires neither of those two packages and the same applies to any source package with a pyproject.toml
since that will build it's own venv (more or less).
So what to do about packages that use the "old" legacy build-format. Those do indeed require setuptools
(and in the worst case a certain version), see the following example about installing Django:
➜ /tmp python -mvenv testing
➜ /tmp ./testing/bin/pip uninstall wheel setuptools
WARNING: Skipping wheel as it is not installed.
Found existing installation: setuptools 49.1.3
Uninstalling setuptools-49.1.3:
Would remove:
/tmp/testing/bin/easy_install
/tmp/testing/bin/easy_install-3.9
/tmp/testing/lib/python3.9/site-packages/easy_install.py
/tmp/testing/lib/python3.9/site-packages/pkg_resources/*
/tmp/testing/lib/python3.9/site-packages/setuptools-49.1.3.dist-info/*
/tmp/testing/lib/python3.9/site-packages/setuptools/*
Proceed (y/n)? y
Successfully uninstalled setuptools-49.1.3
➜ /tmp ./testing/bin/pip install --no-binary Django Django
Collecting Django
Using cached Django-3.1.6.tar.gz (9.6 MB)
ERROR: Command errored out with exit status 1:
command: /tmp/testing/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-gckw7rgk/django/setup.py'"'"'; __file__='"'"'/tmp/pip-install-gckw7rgk/django/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-a61yhryq
cwd: /tmp/pip-install-gckw7rgk/django/
Complete output (3 lines):
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'setuptools'
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
As you can see the virtualenv needs setuptools so pip can install Django, which does make sense since it needs to execute the setup()
function in setup.py
. But as I said before the presence of pyproject.toml
alone (even empty) would trigger a different behavior in pip (namely build isolation). Luckily this behavior can get forced one the cmdline:
➜ /tmp ./testing/bin/pip install --use-pep517 --no-binary Django Django
Collecting Django
Using cached Django-3.1.6.tar.gz (9.6 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing wheel metadata ... done
Collecting sqlparse>=0.2.2
Using cached sqlparse-0.4.1-py3-none-any.whl (42 kB)
Collecting pytz
Using cached pytz-2021.1-py2.py3-none-any.whl (510 kB)
Collecting asgiref<4,>=3.2.10
Using cached asgiref-3.3.1-py3-none-any.whl (19 kB)
Building wheels for collected packages: Django
Building wheel for Django (PEP 517) ... done
Created wheel for Django: filename=Django-3.1.6-py3-none-any.whl size=7834188 sha256=b2a4a23721a1ac19cdfb763d7cab620c5a8d16bf35f754259e33ccd461e45ba3
Stored in directory: /home/florian/.cache/pip/wheels/0d/89/6b/c66c6f61ef318e1e11cbc16aead880ccacca0360e2810a440c
Successfully built Django
Installing collected packages: sqlparse, pytz, asgiref, Django
Successfully installed Django-3.1.6 asgiref-3.3.1 pytz-2021.1 sqlparse-0.4.1
Eh voila, pip downloaded Django (it also downloaded setuptools
& wheel
which you can see with -v
), created a .whl
file and installed that inside.
The commandline option --use-pep517
exists since pip 19.0 and is implicitly enabled for any project that contains a pyproject.toml
(more and more). Using this option pip
no longer has any dependency on setuptools/wheel
in the venv. Unless poetry uses them somehow else in interesting ways I think we could remove them from the unsafe list.
All in all this leads us to the following requirement: If poetry can require at least pip 19.0 and perform installations unconditionally with pip install --use-pep517
then there does not seem to be any import of setuptools
or wheel
in the venv (I verified that by just removing those packages from the venv).
I will try to adjust my local poetry with those changes and report back.
Would poetry
maintainers accept a PR adding a flag like --allow-unsafe
that would allow these "unsafe" packages to be included in poetry.lock
?
My particular use case is to generate a requirements file that I can use without poetry
and I want pip
, setuptools
, and wheel
pinned in the file.
@bmw There exisits a merge request waiting final review that removes those unsafe packages completely. So there will be no need for such a flag.
@bmw @apollo13 jfyi https://github.com/python-poetry/poetry-core/pull/168#issuecomment-817216870 if you add currently unsafe packages to your dependencies, older versions of poetry might not like it all that much.
Also see https://github.com/python-poetry/poetry/discussions/3916 (would appreciate some feedback there).
Hi, are there any updates on this? Just ran into the same issue when I specify https://github.com/pyeve/cerberus/blob/master/setup.py as a dependency
edit: I see it's being fixed in an upcoming release https://github.com/python-poetry/poetry/issues/5351#issuecomment-1077304977
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
-vvv
option).Issue
Using the
pyproject.toml
file from the Gist and executingpoetry lock
I get the following lockfile:setuptools is correctly discovered as dependency of gunicorn (see https://github.com/benoitc/gunicorn/blob/94ab2091173c6037b504f94e56f4e88816d540bf/setup.py#L71-L77 -- it does indeed require it), but the lockfile does not contain any hashes for setuptools. If I now export to requirements.txt the file looks like this:
Which is not installable with
pip install --require-hashes --force-reinstall -r requirements.txt
(note the--require-hases
there):If I instead add Django via
poetry add Django
, thepoetry.lock
file will contain Django's dependencies (pytz
andsqlparse
):and also export them to the requirements file:
I fear there is some special casing going on for
setuptools
, it would be great ifsetuptools
also ended up in the lock and requirement files if it is a direct dependency of on of the added packages.