python-poetry / poetry

Python packaging and dependency management made easy
https://python-poetry.org
MIT License
31.71k stars 2.27k forks source link

Poetry export does not include setuptools even when it is a direct dependency. #1584

Closed apollo13 closed 3 years ago

apollo13 commented 4 years ago

Issue

Using the pyproject.toml file from the Gist and executing poetry lock I get the following lockfile:

[[package]]
category = "main"
description = "WSGI HTTP Server for UNIX"
name = "gunicorn"
optional = false
python-versions = ">=3.4"
version = "20.0.0"

[package.dependencies]
setuptools = ">=3.0"

[package.extras]
eventlet = ["eventlet (>=0.9.7)"]
gevent = ["gevent (>=0.13)"]
setproctitle = ["setproctitle"]
tornado = ["tornado (>=0.2)"]

[metadata]
content-hash = "20c8a1cc7d149b3ad58ca5063dfd37654f03ba65ee8b484d2faefe7ce28b7daf"
python-versions = "^3.7"

[metadata.files]
gunicorn = [
    {file = "gunicorn-20.0.0-py2.py3-none-any.whl", hash = "sha256:0806b5e8a2eb8ba9ac1be65d7b743ec896fc25f5d6cb16c5e051540157b315bb"},
    {file = "gunicorn-20.0.0.tar.gz", hash = "sha256:ef69dea4814df95e64e3f40b47b7ffedc6911c5009233be9d01cfd0d14aa3f50"},
]

setuptools is correctly discovered as dependency of gunicorn (see https://github.com/benoitc/gunicorn/blob/94ab2091173c6037b504f94e56f4e88816d540bf/setup.py#L71-L77 -- it does indeed require it), but the lockfile does not contain any hashes for setuptools. If I now export to requirements.txt the file looks like this:

gunicorn==20.0.0 \
    --hash=sha256:0806b5e8a2eb8ba9ac1be65d7b743ec896fc25f5d6cb16c5e051540157b315bb \
    --hash=sha256:ef69dea4814df95e64e3f40b47b7ffedc6911c5009233be9d01cfd0d14aa3f50

Which is not installable with pip install --require-hashes --force-reinstall -r requirements.txt (note the --require-hases there):

Collecting gunicorn==20.0.0 (from -r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/60/0d/3dbda0324f5bf007f3274e5ea09f0f3bcbf0ca01a75b80ff4f1ff9f8ecfd/gunicorn-20.0.0-py2.py3-none-any.whl
Collecting setuptools>=3.0 (from gunicorn==20.0.0->-r requirements.txt (line 1))
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    setuptools>=3.0 from https://files.pythonhosted.org/packages/d9/de/554b6310ac87c5b921bc45634b07b11394fe63bc4cb5176f5240addf18ab/setuptools-41.6.0-py2.py3-none-any.whl#sha256=3e8e8505e563631e7cb110d9ad82d135ee866b8146d5efe06e42be07a72db20a (from gunicorn==20.0.0->-r requirements.txt (line 1))
WARNING: You are using pip version 19.1.1, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

If I instead add Django via poetry add Django, the poetry.lock file will contain Django's dependencies (pytz and sqlparse):

pytz = [
    {file = "pytz-2019.3-py2.py3-none-any.whl", hash = "sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d"},
    {file = "pytz-2019.3.tar.gz", hash = "sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be"},
]
sqlparse = [
    {file = "sqlparse-0.3.0-py2.py3-none-any.whl", hash = "sha256:40afe6b8d4b1117e7dff5504d7a8ce07d9a1b15aeeade8a2d10f130a834f8177"},
    {file = "sqlparse-0.3.0.tar.gz", hash = "sha256:7c3dca29c022744e95b547e867cee89f4fce4373f3549ccd8797d8eb52cdb873"},
]

and also export them to the requirements file:

django==2.2.7 \
    --hash=sha256:89c2007ca4fa5b351a51a279eccff298520783b713bf28efb89dfb81c80ea49b \
    --hash=sha256:16040e1288c6c9f68c6da2fe75ebde83c0a158f6f5d54f4c5177b0c1478c5b86
gunicorn==20.0.0 \
    --hash=sha256:0806b5e8a2eb8ba9ac1be65d7b743ec896fc25f5d6cb16c5e051540157b315bb \
    --hash=sha256:ef69dea4814df95e64e3f40b47b7ffedc6911c5009233be9d01cfd0d14aa3f50
pytz==2019.3 \
    --hash=sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d \
    --hash=sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be
sqlparse==0.3.0 \
    --hash=sha256:40afe6b8d4b1117e7dff5504d7a8ce07d9a1b15aeeade8a2d10f130a834f8177 \
    --hash=sha256:7c3dca29c022744e95b547e867cee89f4fce4373f3549ccd8797d8eb52cdb873

I fear there is some special casing going on for setuptools, it would be great if setuptools also ended up in the lock and requirement files if it is a direct dependency of on of the added packages.

apollo13 commented 4 years ago

Verified that this issue also exists on beta 5

mbelang commented 4 years ago

I think I have a problem that might be related to this issue.

I have a service which includes a dependency that pins setuptools to 40.8.0 and doing poetry install fails the first time with:

[EnvCommandError]                                                                                                                                                                                                   
Command ['/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/bin/python', '-m', 'pip', 'install', '-e', '/home/circleci/project'] errored with the following return code 2, and output:                                     
Looking in indexes: https://pypi.org/simple, ***************************************************                                                                                                                                 
Obtaining file:///home/circleci/project                                                                                                                                                                                          
ERROR: Exception:                                                                                                                                                                                                                
Traceback (most recent call last):                                                                                                                                                                                               
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 407, in check_if_exists                                                                
    self.satisfied_by = pkg_resources.get_distribution(str(no_marker))                                                                                                                                                           
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 481, in get_distribution                                                          
    dist = get_provider(dist)                                                                                                                                                                                                    
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 357, in get_provider                                                              
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]                                                                                                                                                         
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 900, in require                                                                   
    needed = self.resolve(parse_requirements(requirements))                                                                                                                                                                      
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 791, in resolve                                                                   
    raise VersionConflict(dist, req).with_context(dependent_req)                                                                                                                                                                 
pip._vendor.pkg_resources.ContextualVersionConflict: (setuptools 41.2.0 (/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages), Requirement.parse('setuptools==40.8.0'), {'usher-events'})  
During handling of the above exception, another exception occurred:                                                                                                                                                              
Traceback (most recent call last):                                                                                                                                                                                               
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 188, in main                                                                          
    status = self.run(options, args)                                                                                                                                                                                             
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 345, in run                                                                           
    resolver.resolve(requirement_set)                                                                                                                                                                                            
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 196, in resolve                                                                         
    self._resolve_one(requirement_set, req)                                                                                                                                                                                      
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 359, in _resolve_one                                                                    
    abstract_dist = self._get_abstract_dist_for(req_to_install)                                                                                                                                                                  
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/legacy_resolve.py", line 291, in _get_abstract_dist_for                                                          
    req, self.require_hashes, self.use_user_site, self.finder,                                                                                                                                                                   
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/operations/prepare.py", line 255, in prepare_editable_requirement                                                
    req.check_if_exists(use_user_site)                                                                                                                                                                                           
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 418, in check_if_exists                                                                
    self.req.name                                                                                                                                                                                                                
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 481, in get_distribution                                                          
    dist = get_provider(dist)                                                                                                                                                                                                    
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 357, in get_provider                                                              
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]                                                                                                                                                         
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 900, in require                                                                   
    needed = self.resolve(parse_requirements(requirements))                                                                                                                                                                      
  File "/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages/pip/_vendor/pkg_resources/__init__.py", line 791, in resolve                                                                   
    raise VersionConflict(dist, req).with_context(dependent_req)                                                                                                                                                                 
pip._vendor.pkg_resources.ContextualVersionConflict: (setuptools 41.2.0 (/home/circleci/.cache/pypoetry/virtualenvs/usher-events-py3.7/lib/python3.7/site-packages), Requirement.parse('setuptools==40.8.0'), {'usher-events'})  
WARNING: You are using pip version 19.2.3, however version 19.3.1 is available.                                                                                                                                                  
You should consider upgrading via the 'pip install --upgrade pip' command.                                                                                                                                                       

install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

Exited with code 1

With the following pyproject.toml

[tool.poetry]
name = "usher-events"
version = "0.0.0"
description = "Blahblah"
authors = ["Blabla <hackers@blabla.co>"]
packages = [{ include = "usher_events" }]

[tool.poetry.dependencies]
python = "^3.7"
aws-xray-sdk = "^2.4.2"
botocore = "^1.12.237"
colorama = "^0.4.1"
datadog_lambda = "^0.6.0"

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

If I do poetry install a second time it works without issue. The problem is that on CI it fails every time as CI is always starting fresh. I have poetry 0.12.17 and python 3.7.4.

ryancausey commented 4 years ago

Can confirm that setuptools as a dependency does not appear to be handled properly. There's a package that depends on setuptools in my lockfile, but there is not associated entry in the lockfile for setuptools itself.

Lockfile snippet:

[[package]]
category = "main"
description = "Library to implement a well-behaved Unix daemon process."
name = "python-daemon"
optional = false
python-versions = "*"
version = "2.2.4"

[package.dependencies]
docutils = "*"
lockfile = ">=0.10"
setuptools = "*"
insysion commented 4 years ago

This issue appears to affect pip as a package too.

apollo13 commented 4 years ago

This:

diff --git a/poetry/puzzle/provider.py b/poetry/puzzle/provider.py
index a077e28..1b13212 100644
--- a/poetry/puzzle/provider.py
+++ b/poetry/puzzle/provider.py
@@ -60,7 +60,7 @@ class Indicator(ProgressIndicator):

 class Provider:

-    UNSAFE_PACKAGES = {"setuptools", "distribute", "pip"}
+    UNSAFE_PACKAGES = {}

     def __init__(self, package, pool, io):  # type: (Package, Pool, Any) -> None
         self._package = package

fixes my issue. Now the question is: Why are those unsafe and what are the consequences?

insysion commented 4 years ago

It seems there's not too much reasoning behind this (https://github.com/pypa/pip/issues/6459)

At the very least, we should introduce an --allow-unsafe or similar flag, if not remove the unsafe packages hard-coding altogether.

kminehart commented 4 years ago

Also having this issue with 2 packages that require separate versions of setuptools. (pathvalidate and google-auth).

Install works the first time, but when install is ran again with a cached virtualenv (same lockfile, no changes to pyproject.toml):

[EnvCommandError]
Command ['/root/repo/.venv/bin/pip', 'install', '-e', '/root/repo'] errored with the following return code 2, and output: 
Obtaining file:///root/repo
Exception:
Traceback (most recent call last):
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/req/req_install.py", line 1025, in check_if_exists
    self.satisfied_by = pkg_resources.get_distribution(str(no_marker))
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 558, in get_distribution
    dist = get_provider(dist)
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 432, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 968, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 859, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pip._vendor.pkg_resources.ContextualVersionConflict: (setuptools 39.0.1 (/root/repo/.venv/lib/python3.6/site-packages), Requirement.parse('setuptools>=40.3.0'), {'google-auth'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/commands/install.py", line 324, in run
    requirement_set.prepare_files(finder)
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/req/req_set.py", line 521, in _prepare_file
    req_to_install.check_if_exists()
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/req/req_install.py", line 1036, in check_if_exists
    self.req.name
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 558, in get_distribution
    dist = get_provider(dist)
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 432, in get_provider
    return working_set.find(moduleOrReq) or require(str(moduleOrReq))[0]
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 968, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/root/repo/.venv/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 859, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pip._vendor.pkg_resources.ContextualVersionConflict: (setuptools 39.0.1 (/root/repo/.venv/lib/python3.6/site-packages), Requirement.parse('setuptools>=40.3.0'), {'google-auth'})

I'm assuming this is what's happening:

Once reran, the dependency already exists for google-auth, but it's not the right version, and thus causes the above error.

Just a guess, though.

insysion commented 4 years ago

I am considering opening a PR for this. I plan to change the resolving behaviour so that the UNSAFE_PACKAGES are considered in the same way as any others but block them from being installed/exported unless an --allow-unsafe flag is set.

This will be my first poetry contribution so any advice, e.g. wrt anyone worth running this by before I start work, would be much appreciated.

eliadl commented 4 years ago

This:

diff --git a/poetry/puzzle/provider.py b/poetry/puzzle/provider.py
index a077e28..1b13212 100644
--- a/poetry/puzzle/provider.py
+++ b/poetry/puzzle/provider.py
@@ -60,7 +60,7 @@ class Indicator(ProgressIndicator):

 class Provider:

-    UNSAFE_PACKAGES = {"setuptools", "distribute", "pip"}
+    UNSAFE_PACKAGES = {}

     def __init__(self, package, pool, io):  # type: (Package, Pool, Any) -> None
         self._package = package

fixes my issue. Now the question is: Why are those unsafe and what are the consequences?

@sdispater ?

gsemet commented 4 years ago

Hello. Packaging setuptools may be needed for some reason poetry may not be aware of, so it is important to provide an option to not ignore it

Pinning it inside the virtualenv is also important. pipenv did the mistake of automatically installing the newer version of setuptools on new environment bootstrap, leaving it imposible to pin to a previous, working version in the lockfile (45.0.0 broke something, for a while it was impossible to go back to a version <45). So please allow pinning setuptools and other "special" packages (pip, poetry itself,...).

sdispater commented 4 years ago

The reason why they are marked as "unsafe" is because Poetry relies on them internally (well mostly pip).

However, Poetry will eventually no longer use them and when that happens this restriction will no longer be needed. Until then, I prefer that we leave it as it is to avoid unexpected side effects.

insysion commented 4 years ago

@sdispater, thanks for getting back to us. Did you see my proposal?

change the resolving behaviour so that the UNSAFE_PACKAGES are considered in the same way as any others but block them from being installed/exported unless an --allow-unsafe flag is set.

This would seem to address problem of locking a set of packages that are inconsistent due to the unsafe packages being completely ignored at resolve time without affecting the default install/export behaviour. Would you support a PR implementing this?

Sorry to push the subject but it's causing us significant pain right now: we're having to add dummy constraints updated manually to workaround this. It's a high maintenance and patchy state of affairs.

ptramsey commented 4 years ago

Any word on this? Just encountered this same issue; it'd be nice if there were a solution.

ivallesp commented 4 years ago

Same here. I found VersionConflict with the pip and setuptools versions when trying to install tensorflow and tensorboard from poetry. The only workaround I found was to substitute the poetry install by:

poetry run pip install -U pip poetry run pip install -U setuptools poetry install

This is not optimal as the pip and setuptools versions are not locked. Any solution to this? Seems to be a critical bug...

Issue #1651 seems to be tackling this point, although no activity since February

ghost commented 4 years ago

Same situation here as @ivallesp , tensorboard requires setuptools >= 41.0.0 but for some reason, 40.8.0 gets installed. This gives me a ContextualVersionConflict runtime error from pkg_resources.

Even adding a fake dependency to my project for a more recent setuptools doesn't help, nothing gets installed when I run poetry.

earonesty commented 4 years ago

Possibly poetry just should install and manage its setuptools and pip requirements in different/edited/hacked sys.path (...venv/lib/poetry-packages/ instead of site-packages), and then the venv itself can have no restrictions. IE: poetry should never have 'unsafe' packages, because its environment is separately managed by itself. That way we don't have to wait until poetry no longer uses setuptools and/or pip. A side effect of this is that poetry can then bootstrap to use any support packages it needs (arrow or other fun things), because they won't be a problem.

apollo13 commented 4 years ago

@earonesty brings up an interesting point. I'd argue that poetry itself is usually installed globally or in another venv (ie via pipx) so it should not conflict with anything in the venv at all. Does that address some of your concerns @sdispater? I am asking because this issue is the final show stopper for us to embrace poetry fully. I understand the motivation behind your argument, but if poetry isn't in the managed venv anyways, there are far less chances of issues. On the other hand, everyone installing gunicorn or tensorflow currently has problems with poetry.

alexchandel commented 4 years ago

@insysion Did you open a PR?

This is a serious bug. It causes the installation of older, buggier versions of setuptools like in https://github.com/pypa/setuptools/issues/1963. There's no reason to break such projects.

earonesty commented 4 years ago

I'd be happy to try to open a PR if people like the idea of putting poetry deps in another folder (. poetry_deps ?)

earonesty commented 3 years ago

note: i suspect poetry's dependence on msgpack also causes problems for the same reason. there's no way to alter msgpack's version when poetry is running. ie: you need to be sure the msgpack you run is identical to poetry's or you will fail on machines, like windows, that lock shared libs while in use.

TBBle commented 3 years ago

Poetry's recommended installation approaches (get-poetry.py, pip install --user, and pipx install) all keep Poetry's stuff in a separate place from where this code is working; unless you also set poetry config virtualenvs.create false and don't have separate envs created, then Poetry's dependency resolution would be resolving for a separate virutal-env anyway.

So it seems like it should be perfectly safe to set UNSAFE_PACKAGES = {} when running inside a project-specific virtual env that is separate from Poetry's virtual env (if it has one).

@earonesty: I suspect that msgpack problem is different, as it's part of poetry self update, and that is operating on the Poetry venv. There should be no conflict between Poetry's msgpack and a project's msgpack version, unless the user is deliberately putting the project and Poetry into the same venv.

mikenerone commented 3 years ago

Poetry's recommended installation approaches (get-poetry.py, pip install --user, and pipx install) all keep Poetry's stuff in a separate place from where this code is working; unless you also set poetry config virtualenvs.create false and don't have separate envs created, then Poetry's dependency resolution would be resolving for a separate virutal-env anyway.

@TBBle Though Poetry itself runs from its own virtualenv, I was under the impression (based on the existence of things like UNSAFE_PACKAGES and previous discussion related to this topic) that when performing operations on the project virtualenv itself, it effectively invokes pip (and therefore setuptools and wheel) from the project virtualenv. I'm not 100%, tbh.

TBBle commented 3 years ago

Ah, right, I didn't realise that.

I would hope that its dependency on pip etc. in the project env would not be as-strict as for poetry's own venv though. So UNSAFE_PACKAGES could therefore be reapproached as a set of dependencies (versioned matches, presumably something like pip >= ... for pip flags used by Poetry) for the project env, which should be mixed into the project's own dependencies and resolved like any other overlapping requirements.

To resolve this problem, it would need to be able to distinguish when the pip, setuptools, and wheel dependencies are only from using Poetry (and should hence be excluded from poetry export), and when they come from the project or one of its dependencies.

thibaudcolas commented 3 years ago

For what it’s worth I ran into this while trying to use Snyk on my Poetry projects. Snyk is unusable because of this – snyk test raises an error when failing to retrieve the setuptools metadata from poetry.lock:

Error processing poetry project. Unable to find dependencies in poetry.lock for package: setuptools

I’ve reported this to Snyk over at https://github.com/snyk/snyk/issues/367#issuecomment-757972874 even though it’s clearly a Poetry issue.

pop commented 3 years ago

Is this behavior still necessary? As a user I would either like poetry to include dependencies like wheel when I include it in my pyproject.toml, to print a warning that this dependency will be ignored, or to allow me to opt-out of this behavior with a flag.

apollo13 commented 3 years ago

So I did dig through this issue again and came to the following conclusions:

So assuming we want to keep pip in the list of unsafe packages, I think we can freely remove distribute and that leaves us with setuptools & wheel. As noticed above, installing wheels requires neither of those two packages and the same applies to any source package with a pyproject.toml since that will build it's own venv (more or less).

So what to do about packages that use the "old" legacy build-format. Those do indeed require setuptools (and in the worst case a certain version), see the following example about installing Django:

➜  /tmp python -mvenv testing
➜  /tmp ./testing/bin/pip uninstall wheel setuptools
WARNING: Skipping wheel as it is not installed.
Found existing installation: setuptools 49.1.3
Uninstalling setuptools-49.1.3:
  Would remove:
    /tmp/testing/bin/easy_install
    /tmp/testing/bin/easy_install-3.9
    /tmp/testing/lib/python3.9/site-packages/easy_install.py
    /tmp/testing/lib/python3.9/site-packages/pkg_resources/*
    /tmp/testing/lib/python3.9/site-packages/setuptools-49.1.3.dist-info/*
    /tmp/testing/lib/python3.9/site-packages/setuptools/*
Proceed (y/n)? y
  Successfully uninstalled setuptools-49.1.3
➜  /tmp ./testing/bin/pip install --no-binary Django Django
Collecting Django
  Using cached Django-3.1.6.tar.gz (9.6 MB)
    ERROR: Command errored out with exit status 1:
     command: /tmp/testing/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-gckw7rgk/django/setup.py'"'"'; __file__='"'"'/tmp/pip-install-gckw7rgk/django/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-a61yhryq
         cwd: /tmp/pip-install-gckw7rgk/django/
    Complete output (3 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'setuptools'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

As you can see the virtualenv needs setuptools so pip can install Django, which does make sense since it needs to execute the setup() function in setup.py. But as I said before the presence of pyproject.toml alone (even empty) would trigger a different behavior in pip (namely build isolation). Luckily this behavior can get forced one the cmdline:

➜  /tmp ./testing/bin/pip install --use-pep517 --no-binary Django Django
Collecting Django
  Using cached Django-3.1.6.tar.gz (9.6 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting sqlparse>=0.2.2
  Using cached sqlparse-0.4.1-py3-none-any.whl (42 kB)
Collecting pytz
  Using cached pytz-2021.1-py2.py3-none-any.whl (510 kB)
Collecting asgiref<4,>=3.2.10
  Using cached asgiref-3.3.1-py3-none-any.whl (19 kB)
Building wheels for collected packages: Django
  Building wheel for Django (PEP 517) ... done
  Created wheel for Django: filename=Django-3.1.6-py3-none-any.whl size=7834188 sha256=b2a4a23721a1ac19cdfb763d7cab620c5a8d16bf35f754259e33ccd461e45ba3
  Stored in directory: /home/florian/.cache/pip/wheels/0d/89/6b/c66c6f61ef318e1e11cbc16aead880ccacca0360e2810a440c
Successfully built Django
Installing collected packages: sqlparse, pytz, asgiref, Django
Successfully installed Django-3.1.6 asgiref-3.3.1 pytz-2021.1 sqlparse-0.4.1

Eh voila, pip downloaded Django (it also downloaded setuptools & wheel which you can see with -v), created a .whl file and installed that inside.

The commandline option --use-pep517 exists since pip 19.0 and is implicitly enabled for any project that contains a pyproject.toml (more and more). Using this option pip no longer has any dependency on setuptools/wheel in the venv. Unless poetry uses them somehow else in interesting ways I think we could remove them from the unsafe list.

All in all this leads us to the following requirement: If poetry can require at least pip 19.0 and perform installations unconditionally with pip install --use-pep517 then there does not seem to be any import of setuptools or wheel in the venv (I verified that by just removing those packages from the venv).

I will try to adjust my local poetry with those changes and report back.

bmw commented 3 years ago

Would poetry maintainers accept a PR adding a flag like --allow-unsafe that would allow these "unsafe" packages to be included in poetry.lock?

My particular use case is to generate a requirements file that I can use without poetry and I want pip, setuptools, and wheel pinned in the file.

apollo13 commented 3 years ago

@bmw There exisits a merge request waiting final review that removes those unsafe packages completely. So there will be no need for such a flag.

abn commented 3 years ago

@bmw @apollo13 jfyi https://github.com/python-poetry/poetry-core/pull/168#issuecomment-817216870 if you add currently unsafe packages to your dependencies, older versions of poetry might not like it all that much.

abn commented 3 years ago

Also see https://github.com/python-poetry/poetry/discussions/3916 (would appreciate some feedback there).

rajaie-sg commented 2 years ago

Hi, are there any updates on this? Just ran into the same issue when I specify https://github.com/pyeve/cerberus/blob/master/setup.py as a dependency

edit: I see it's being fixed in an upcoming release https://github.com/python-poetry/poetry/issues/5351#issuecomment-1077304977

github-actions[bot] commented 8 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.