search

python-poetry / poetry

Python packaging and dependency management made easy
https://python-poetry.org
MIT License
31.62k stars 2.27k forks source link

Poetry tries to remove setuptools and then complains it can't find it #4242

Closed ghost closed 2 years ago

ghost commented 3 years ago

Issue

When running poetry install, Poetry tries to remove setuptools, and then later complains that setuptools is not found when trying to install future.

I suspect this may be because win10toast depends on setuptools, but that package is skipped due to the platform not being Windows.

❯ poetry install -vvv
Creating virtualenv test in /home/nyuszika7h/test/.venv
Using virtualenv: /home/nyuszika7h/test/.venv
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 7 installs, 0 updates, 1 removal, 6 skipped

  • Removing importlib-metadata (4.6.0): Pending...
  • Removing importlib-metadata (4.6.0): Skipped for the following reason: Not currently installed
  • Removing pypiwin32 (223): Pending...
  • Removing pypiwin32 (223): Skipped for the following reason: Not currently installed
  • Removing pywin32 (301): Pending...
  • Removing pywin32 (301): Skipped for the following reason: Not currently installed
  • Removing win10toast (0.9): Pending...
  • Removing win10toast (0.9): Skipped for the following reason: Not currently installed
  • Removing typing-extensions (3.10.0.0): Pending...
  • Removing typing-extensions (3.10.0.0): Skipped for the following reason: Not currently installed
  • Removing zipp (3.4.1): Pending...
  • Removing zipp (3.4.1): Skipped for the following reason: Not currently installed
  • Removing setuptools (57.0.0): Pending...
  • Removing setuptools (57.0.0): Removing...
  • Removing setuptools (57.0.0)
  • Installing soupsieve (2.2.1): Pending...
  • Installing soupsieve (2.2.1): Installing...
  • Installing soupsieve (2.2.1)
  • Installing beautifulsoup4 (4.9.3): Pending...
  • Installing beautifulsoup4 (4.9.3): Installing...
  • Installing cssutils (2.3.0): Pending...
  • Installing cssutils (2.3.0): Installing...
  • Installing future (0.18.2): Pending...
  • Installing future (0.18.2): Installing...
  • Installing future (0.18.2): Failed

  EnvCommandError

  Command ['/home/nyuszika7h/test/.venv/bin/pip', 'install', '--no-deps', 'file:///home/nyuszika7h/.cache/pypoetry/artifacts/f8/58/55/86be1f567b212fdd98854d12815964a49db8fb1bcff725018e5f95c61d/future-0.18.2.tar.gz'] errored with the following return code 1, and output:
  Processing /home/nyuszika7h/.cache/pypoetry/artifacts/f8/58/55/86be1f567b212fdd98854d12815964a49db8fb1bcff725018e5f95c61d/future-0.18.2.tar.gz
      ERROR: Command errored out with exit status 1:
       command: /home/nyuszika7h/test/.venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-1yncrz1a/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-1yncrz1a/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-gd9atrnx
           cwd: /tmp/pip-req-build-1yncrz1a/
      Complete output (3 lines):
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
      ModuleNotFoundError: No module named 'setuptools'
      ----------------------------------------
  WARNING: Discarding file:///home/nyuszika7h/.cache/pypoetry/artifacts/f8/58/55/86be1f567b212fdd98854d12815964a49db8fb1bcff725018e5f95c61d/future-0.18.2.tar.gz. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  WARNING: You are using pip version 21.1.2; however, version 21.1.3 is available.
  You should consider upgrading via the '/home/nyuszika7h/test/.venv/bin/python -m pip install --upgrade pip' command.

  at ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/utils/env.py:1101 in _run
      1097│                 output = subprocess.check_output(
      1098│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1099│                 )
      1100│         except CalledProcessError as e:
    → 1101│             raise EnvCommandError(e, input=input_)
      1102│
      1103│         return decode(output)
      1104│
      1105│     def execute(self, bin, *args, **kwargs):

  • Installing lxml (4.6.3): Pending...
  • Installing lxml (4.6.3): Installing...
  • Installing six (1.16.0): Pending...
  • Installing six (1.16.0): Installing...
  • Installing six (1.16.0)
  • Installing cssutils (2.3.0)
  • Installing future (0.18.2): Pending...
  • Installing future (0.18.2): Installing...
  • Installing future (0.18.2): Failed

  EnvCommandError

  Command ['/home/nyuszika7h/test/.venv/bin/pip', 'install', '--no-deps', 'file:///home/nyuszika7h/.cache/pypoetry/artifacts/f8/58/55/86be1f567b212fdd98854d12815964a49db8fb1bcff725018e5f95c61d/future-0.18.2.tar.gz'] errored with the following return code 1, and output:
  Processing /home/nyuszika7h/.cache/pypoetry/artifacts/f8/58/55/86be1f567b212fdd98854d12815964a49db8fb1bcff725018e5f95c61d/future-0.18.2.tar.gz
      ERROR: Command errored out with exit status 1:
       command: /home/nyuszika7h/test/.venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-1yncrz1a/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-1yncrz1a/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-gd9atrnx
           cwd: /tmp/pip-req-build-1yncrz1a/
      Complete output (3 lines):
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
      ModuleNotFoundError: No module named 'setuptools'
      ----------------------------------------
  WARNING: Discarding file:///home/nyuszika7h/.cache/pypoetry/artifacts/f8/58/55/86be1f567b212fdd98854d12815964a49db8fb1bcff725018e5f95c61d/future-0.18.2.tar.gz. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  WARNING: You are using pip version 21.1.2; however, version 21.1.3 is available.
  You should consider upgrading via the '/home/nyuszika7h/test/.venv/bin/python -m pip install --upgrade pip' command.

  at ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/utils/env.py:1101 in _run
      1097│                 output = subprocess.check_output(
      1098│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1099│                 )
      1100│         except CalledProcessError as e:
    → 1101│             raise EnvCommandError(e, input=input_)
      1102│
      1103│         return decode(output)
      1104│
      1105│     def execute(self, bin, *args, **kwargs):

  • Installing lxml (4.6.3): Pending...
  • Installing lxml (4.6.3): Installing...
  • Installing lxml (4.6.3)
  • Installing six (1.16.0): Pending...
  • Installing six (1.16.0): Installing...
  • Installing six (1.16.0)
  • Installing beautifulsoup4 (4.9.3)
  • Installing cssutils (2.3.0): Pending...
  • Installing cssutils (2.3.0): Installing...
  • Installing cssutils (2.3.0)
  • Installing future (0.18.2): Pending...
  • Installing future (0.18.2): Installing...
  • Installing future (0.18.2): Failed

  EnvCommandError

  Command ['/home/nyuszika7h/test/.venv/bin/pip', 'install', '--no-deps', 'file:///home/nyuszika7h/.cache/pypoetry/artifacts/f8/58/55/86be1f567b212fdd98854d12815964a49db8fb1bcff725018e5f95c61d/future-0.18.2.tar.gz'] errored with the following return code 1, and output:
  Processing /home/nyuszika7h/.cache/pypoetry/artifacts/f8/58/55/86be1f567b212fdd98854d12815964a49db8fb1bcff725018e5f95c61d/future-0.18.2.tar.gz
      ERROR: Command errored out with exit status 1:
       command: /home/nyuszika7h/test/.venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-1yncrz1a/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-1yncrz1a/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-gd9atrnx
           cwd: /tmp/pip-req-build-1yncrz1a/
      Complete output (3 lines):
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
      ModuleNotFoundError: No module named 'setuptools'
      ----------------------------------------
  WARNING: Discarding file:///home/nyuszika7h/.cache/pypoetry/artifacts/f8/58/55/86be1f567b212fdd98854d12815964a49db8fb1bcff725018e5f95c61d/future-0.18.2.tar.gz. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  WARNING: You are using pip version 21.1.2; however, version 21.1.3 is available.
  You should consider upgrading via the '/home/nyuszika7h/test/.venv/bin/python -m pip install --upgrade pip' command.

  at ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/utils/env.py:1101 in _run
      1097│                 output = subprocess.check_output(
      1098│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1099│                 )
      1100│         except CalledProcessError as e:
    → 1101│             raise EnvCommandError(e, input=input_)
      1102│
      1103│         return decode(output)
      1104│
      1105│     def execute(self, bin, *args, **kwargs):

  • Installing lxml (4.6.3): Pending...
  • Installing lxml (4.6.3): Installing...
  • Installing lxml (4.6.3)
  • Installing six (1.16.0): Pending...
  • Installing six (1.16.0): Installing...
  • Installing six (1.16.0)

To clarify: Running poetry lock from Poetry 1.2.0a1 seems to cause the lock file to include setuptools, which triggers the issue with Poetry <= 1.1.7. Running poetry lock from Poetry 1.1.7 makes it exclude setuptools and there is no issue.

Install with Poetry 1.1.7 Install with Poetry 1.2.0a1
Lock file from Poetry 1.1.7 :white_check_mark: :white_check_mark:
Lock file from Poetry 1.2.0a1 :x: :white_check_mark:

Also, both lock files were generated on Linux. I have not tested on Windows yet.

TheElectronWill commented 3 years ago

Can reproduce on windows and linux: poetry 1.2.0a1 generates a lock file that, when installed, sometimes, removes setuptools and makes some packages fail to install.

nyanpasu64 commented 3 years ago

Does editing your pyproject.toml to add "setuptools" to the [build-system] requires stop the error from occurring? I don't know if it's a fix or a workaround.

[build-system]
-requires = ["poetry_core>=1.0"]
+requires = ["setuptools", "poetry_core>=1.0"]
build-backend = "poetry.core.masonry.api"
dimaqq commented 3 years ago

The delta between 1.2.0 prerelease and 1.1.7 stable: (that is going back to stable)

diff --git a/poetry.lock b/poetry.lock
index dc6b6820f..f0e8e3fdd 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -499,7 +499,6 @@
 pickleshare = "*"
 prompt-toolkit = ">=2.0.0,<3.0.0 || >3.0.0,<3.0.1 || >3.0.1,<3.1.0"
 pygments = "*"
-setuptools = ">=18.5"
 traitlets = ">=4.2"

 [package.extras]
@@ -1020,18 +1019,6 @@

-[[package]]
-name = "setuptools"
-version = "57.4.0"
-description = "Easily download, build, install, upgrade, and uninstall Python packages"
-category = "dev"
-optional = false
-python-versions = ">=3.6"
-
-[package.extras]
-docs = ["sphinx", "jaraco.packaging (>=8.2)", "rst.linker (>=1.9)", "jaraco.tidelift (>=1.4)", "pygments-github-lexers (==0.0.5)", "sphinx-inline-tabs", "sphinxcontrib-towncrier", "furo"]
-testing = ["pytest (>=4.6)", "pytest-checkdocs (>=2.4)", "pytest-flake8", "pytest-cov", "pytest-enabler (>=1.0.1)", "mock", "flake8-2020", "virtualenv (>=13.0.0)", "pytest-virtualenv (>=1.2.7)", "wheel", "paver", "pip (>=19.1)", "jaraco.envs", "pytest-xdist", "sphinx", "jaraco.path (>=3.2.0)", "pytest-black (>=0.3.7)", "pytest-mypy"]
-
@@ -1082,7 +1069,6 @@
 packaging = "*"
 Pygments = ">=2.0"
 requests = ">=2.0.0"
-setuptools = "*"
 six = ">=1.5"
 snowballstemmer = ">=1.1"
 sphinxcontrib-websupport = "*"
@@ -1994,10 +1980,6 @@
-setuptools = [
-    {file = "setuptools-57.4.0-py3-none-any.whl", hash = "sha256:a49230977aa6cfb9d933614d2f7b79036e9945c4cdd7583163f4e920b83418d6"},
-    {file = "setuptools-57.4.0.tar.gz", hash = "sha256:6bac238ffdf24e8806c61440e755192470352850f3419a52f26ffe0a1a64f465"},
-]
 six = [
     {file = "six-1.15.0-py2.py3-none-any.whl", hash = "sha256:8b74bedcbbbaca38ff6d7491d76f2b06b3592611af620f8426e82dddb04a5ced"},
     {file = "six-1.15.0.tar.gz", hash = "sha256:30639c035cdb23534cd4aa2dd52c3bf48f06e5f4a941509c8bafd8ce11080259"},

In other words, 1.2.0 pre includes setuptools into poetry.lock while 1.1.7 does not.

I guess that 1.1.7 wants to build [package wheel? of a] dependency in a clean environment, and that's why it tries to [temporarily] remove setuptools...

apollo13 commented 3 years ago

So the only issue here is that an older poetry cannot read a lock file generated by the newer version? If that is the case then I think this is expected behavior.

ghost commented 3 years ago

Breaking backwards compatibility in a minor release is not the best experience. Is there really no way to make any required changes in a way that it doesn't lead to mysterious breakage on earlier versions?

apollo13 commented 3 years ago

I don't think so. In this special case it was actually a bug fix (1.1 was broken in the sense that it did not properly record setuptools in the lockfile). There is really no good way that an old version could ever cope with that I think.

ghost commented 3 years ago

Does editing your pyproject.toml to add "setuptools" to the [build-system] requires stop the error from occurring? I don't know if it's a fix or a workaround.

[build-system]
-requires = ["poetry_core>=1.0"]
+requires = ["setuptools", "poetry_core>=1.0"]
build-backend = "poetry.core.masonry.api"

Sorry for the late reply about this. I think that fixed my issues before, but with 1.2.0 -> 1.1.x it does not fix the issue. But poetry add setuptools seems to fix it. So I guess the workaround is either that, or not using Poetry 1.2.0 on my end until it's stable, and telling users to upgrade once it is. But it would be nice if there was a way to tell users that this project requires a specific version of Poetry rather than a scary error message. requires = ['poetry-core>=1.2.0'] does not seem to change anything,

abn commented 2 years ago

The build-system.requires part is something poetry itself does not use at the moment. This is used by the non-poetry PEP 517 frontends (eg: pip, build etc.)

See also #2789.

As to why setuptools appear in the lockfile, this is related to #2826. This change meant that it is no longer excluded from dependency resolution. This means if a dependency in your project's dependency tree requires setuptools it will be added to the lock file.

https://github.com/python-poetry/poetry/issues/3153#issuecomment-972796519

As you identify, since a project in your tree requires setuptools and that is not enabled for the platform, setuptools get removed as it is not required.

Additionally, changes in #3835 should now mean that pip install now adds --use-pep517 when building a depdency from an sdist.

I'd recommend you try using poetry from master.

rafalkrupinski commented 2 years ago

I'm on poetry 1.1.12 I've removed my poetry.lock and recreated it and I'm still facing the issue

rr- commented 2 years ago

For me having explicit dependency on setuptools in normal dependencies does not make a difference, it still removes it.

It completely breaks our Cython builds.

abi-jey commented 2 years ago

I have the same problem as well that trying to slowly write my poetry plugins, but this issue with incompatibility between 1.1 and 1.2 would not allow me to use 1.2 and plugins.

danielloader commented 2 years ago

I've just bitten the bullet and tested 1.2.0b1 and wholly moved over to it - gambling a bit but half way house between the two isn't feasible any more. Especially with the --dev becoming --group dev and the subsequent trying to parse the group key in pyproject.toml

abn commented 2 years ago

I am going to close this as it seems this will be resolved in 1.2 and is already on mater.

emilhe commented 2 years ago

In case anyone are looking for a workaround to make a .lock file generated with Poetry 1.2.x compatible with Poetry 1.1.x, I created a small gist to do so,

curl https://gist.githubusercontent.com/emilhe/0c7b1a33b2d02f17331242bf4fffd07c/raw/8da0665a58f469c980e7661d7f8c36f3bd3af992/strip_setuptools.py | python - && poetry install

In my case, I needed it because I am using Poetry 1.2.x locally (I am having various issues with 1.1.x), while my cloud provider supports only 1.1.x.

antonakospanos commented 2 years ago

In my case the issue is reproduced using poetry 1.2.0. The new versions adds in poetry.lock the following lines:

[package.dependencies]
setuptools = "*"

[[package]]
name = "setuptools"
version = "63.4.3"
description = "Easily download, build, install, upgrade, and uninstall Python packages"
category = "main"
optional = false
python-versions = ">=3.7"

Which when a another colleague (or our CI) runs poetry install using poetry 1.1.14 fails as follows:

Installing typing-extensions (4.3.0)
..
EnvCommandError
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'error'
  error: subprocess-exited-with-error
  × python setup.py egg_info did not run successfully.
  exit code: 1
  ╰─> [1 lines of output]
ERROR: Can not execute `setup.py` since setuptools is not available in the build environment.
jomach commented 1 year ago

I still have this problem on 1.2.x

qkponton commented 1 year ago

Having same or similar issue with Poetry (version 1.3.2) Anyone found a fix for this?

Command ['/home/visitor/afm/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--isolated', '--no-input', '--prefix', '/home/visitor/afm', '--no-deps', '/home/visitor/.cache/pypoetry/artifacts/dd/72/1c/1fb84326375a883f33f9419395a2175a0608c799d63478062ef3b3c7d8/future-0.18.3.tar.gz'] errored with the following return code 1, and output: Processing /home/visitor/.cache/pypoetry/artifacts/dd/72/1c/1fb84326375a883f33f9419395a2175a0608c799d63478062ef3b3c7d8/future-0.18.3.tar.gz Installing build dependencies: started Installing build dependencies: still running... Installing build dependencies: still running... Installing build dependencies: still running... Installing build dependencies: still running... Installing build dependencies: still running... Installing build dependencies: still running... Installing build dependencies: finished with status 'error' error: subprocess-exited-with-error × pip subprocess to install build dependencies did not run successfully. │ exit code: 1 ╰─> [7 lines of output] WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fbfa837a2b0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/ WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fbfa837a5b0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/ WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fbfa837a7c0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/ WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fbfa837a8e0>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/ WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ConnectTimeoutError(<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7fbfa837aa60>, 'Connection to pypi.org timed out. (connect timeout=15)')': /simple/setuptools/ ERROR: Could not find a version that satisfies the requirement setuptools>=40.8.0 (from versions: none) ERROR: No matching distribution found for setuptools>=40.8.0 [end of output]

And I'm also wondering why pip is trying to reach "pypi.org" although I've defined a private index...

poetry env info

Virtualenv Python: 3.9.11 Implementation: CPython Path: /home/visitor/afm Executable: /home/visitor/afm/bin/python Valid: True

/home/visitor/afm/bin/pip config -v list For variant 'global', will try loading '/etc/xdg/pip/pip.conf' For variant 'global', will try loading '/etc/pip.conf' For variant 'user', will try loading '/home/visitor/.pip/pip.conf' For variant 'user', will try loading '/home/visitor/.config/pip/pip.conf' For variant 'site', will try loading '/home/visitor/afm/pip.conf' global.index='http://my-private/repository/PyPI/' global.index-url='http://my-private/repository/PyPI/simple/' global.trusted-host='my-private'

qkponton commented 1 year ago

I found the related topic: https://github.com/python-poetry/poetry/issues/3249 thanks

github-actions[bot] commented 8 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.