python-poetry / poetry

Python packaging and dependency management made easy
https://python-poetry.org
MIT License
31.08k stars 2.26k forks source link

Missing build dependency after `poetry update`. #2197

Closed supakeen closed 2 years ago

supakeen commented 4 years ago

Hola, I'm a big fan of poetry. After a recent poetry update command my builds seem to fail in Travis. Here's the output of one such builds: https://travis-ci.org/github/supakeen/pinnwand/jobs/664093263

The pyproject.toml can be found here: https://github.com/supakeen/pinnwand/blob/8cc7e4e29207785ad50d21c850fd7e9ad59171de/pyproject.toml and its poetry.lock here: https://github.com/supakeen/pinnwand/blob/8cc7e4e29207785ad50d21c850fd7e9ad59171de/poetry.lock

From asking around a bit it seems "PEP517", and "isolated builds" are mentioned in relation to this. Is this poetry missing a build dependency for sqlalchemy 1.3.15?

The changeset to these files that started these failing builds is this one:https://github.com/supakeen/pinnwand/commit/8cc7e4e29207785ad50d21c850fd7e9ad59171de

rvanlaar commented 4 years ago

We're having another problem with SQLAlchemy 1.3.14 and 1.3.14.

I can install SQLAlchemy with a new poetry project, i.e. only sqlalchemy is specified in pyproject.toml.

A pdf with the _struct importerror showed me that '/usr/local/lib/python3.5/lib-dynload' was missing during the installation. That path is available when running poetry run python

import sys 
print(sys.path)
Command ['/root/.cache/pypoetry/virtualenvs/edubroker-kom-3b6oOu6H-py3.5/bin/pip', 'install', '--no-deps', '-U', 'sqlalchemy==1.3.15'] errored with the following return code 1, and output: 
Collecting sqlalchemy==1.3.15
  Downloading SQLAlchemy-1.3.15.tar.gz (6.1 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /root/.cache/pypoetry/virtualenvs/edubroker-kom-3b6oOu6H-py3.5/bin/python3.5 /root/.cache/pypoetry/virtualenvs/edubroker-kom-3b6oOu6H-py3.5/lib/python3.5/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpzs_m3r9e
       cwd: /tmp/pip-install-ai8w37ab/sqlalchemy
  Complete output (4 lines):
  Traceback (most recent call last):
    File "/root/.cache/pypoetry/virtualenvs/edubroker-kom-3b6oOu6H-py3.5/lib/python3.5/site-packages/pip/_vendor/pep517/_in_process.py", line 15, in <module>
      from glob import glob
  ImportError: No module named 'glob'

Today the error changed:

[EnvCommandError]
Command ['/root/.cache/pypoetry/virtualenvs/edubroker-kom-3b6oOu6H-py3.5/bin/pip', 'install', '--no-deps', 'sqlalchemy==1.3.15'] errored with the following return code 1, and output: 
Collecting sqlalchemy==1.3.15
  Downloading SQLAlchemy-1.3.15.tar.gz (6.1 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /root/.cache/pypoetry/virtualenvs/edubroker-kom-3b6oOu6H-py3.5/bin/python3.5 /root/.cache/pypoetry/virtualenvs/edubroker-kom-3b6oOu6H-py3.5/lib/python3.5/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmp6gknnhpu
       cwd: /tmp/pip-install-5je3brzl/sqlalchemy
  Complete output (10 lines):
  Traceback (most recent call last):
    File "/root/.cache/pypoetry/virtualenvs/edubroker-kom-3b6oOu6H-py3.5/lib/python3.5/site-packages/pip/_vendor/pep517/_in_process.py", line 21, in <module>
      import shutil
    File "/usr/local/lib/python3.5/shutil.py", line 13, in <module>
      import tarfile
    File "/usr/local/lib/python3.5/tarfile.py", line 48, in <module>
      import struct
    File "/usr/local/lib/python3.5/struct.py", line 13, in <module>
      from _struct import *
  ImportError: No module named '_struct'
  ----------------------------------------
rvanlaar commented 4 years ago

It seems this is caused by sqlalchemy having both a pyproject.toml and a setup.py. I've created an issue there.

zzzeek commented 4 years ago

hi what pip version? Per pypa, nothing is wrong with SQLAlchemy's setup: https://github.com/pypa/pip/issues/7874#issuecomment-601449946

supakeen commented 4 years ago

The version of pip on the system I use to poetry update is the following:

pip 20.0.2 from /home/user/src/pinnwand/venv/lib/python3.8/site-packages/pip (python 3.8)

The pip version used for installation is upgraded automatically at the start but in this case was 20.0.2 as well as seen in the build output.

If this is in fact a pip error, should I move over to your issue there?

Another error I've seen when upgrading on my secondary development machine is the following, which is different from the one on the travis build:

Updating dependencies
Resolving dependencies... (4.8s)

Writing lock file

Package operations: 1 install, 9 updates, 0 removals

  - Updating packaging (20.1 -> 20.3)
  - Updating pyyaml (5.3 -> 5.3.1)
  - Updating click (7.0 -> 7.1.1)
  - Updating pygments (2.5.2 -> 2.6.1)
  - Updating pytest (5.3.5 -> 5.4.1)
  - Updating tomlkit (0.5.8 -> 0.5.11)
  - Updating virtualenv (20.0.7 -> 20.0.13)
  - Installing pygments-better-html (0.1.3)
  - Updating sqlalchemy (1.3.13 -> 1.3.15)

[EnvCommandError]
Command ['/home/user/src/pinnwand/venv/bin/pip', 'install', '--no-deps', '-U', 'sqlalchemy==1.3.15'] errored with the following return code 1, and output:
Collecting sqlalchemy==1.3.15
  Downloading SQLAlchemy-1.3.15.tar.gz (6.1 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /home/user/src/pinnwand/venv/bin/python3.8 /home/user/src/pinnwand/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpbj9ri7zx
       cwd: /tmp/pip-install-arpi55q9/sqlalchemy
  Complete output (41 lines):
  Traceback (most recent call last):
    File "/home/user/src/pinnwand/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2451, in resolve
      return functools.reduce(getattr, self.attrs, module)
  AttributeError: type object 'Distribution' has no attribute '_finalize_feature_opts'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/user/src/pinnwand/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 257, in <module>
      main()
    File "/home/user/src/pinnwand/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 240, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/user/src/pinnwand/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py", line 91, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-w07d2z55/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 147, in get_requires_for_build_wheel
      return self._get_build_requires(
    File "/tmp/pip-build-env-w07d2z55/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 128, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-w07d2z55/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 249, in run_setup
      super(_BuildMetaLegacyBackend,
    File "/tmp/pip-build-env-w07d2z55/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 143, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 220, in <module>
      run_setup(True)
    File "setup.py", line 145, in run_setup
      setup(
    File "/tmp/pip-build-env-w07d2z55/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 143, in setup
      _install_setup_requires(attrs)
    File "/tmp/pip-build-env-w07d2z55/overlay/lib/python3.8/site-packages/setuptools/__init__.py", line 131, in _install_setup_requires
      dist = distutils.core.Distribution(dict(
    File "/tmp/pip-build-env-w07d2z55/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 424, in __init__
      _Distribution.__init__(self, {
    File "/usr/lib/python3.8/distutils/dist.py", line 292, in __init__
      self.finalize_options()
    File "/tmp/pip-build-env-w07d2z55/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 706, in finalize_options
      ep.load()(self)
    File "/home/user/src/pinnwand/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2443, in load
      return self.resolve()
    File "/home/user/src/pinnwand/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 2453, in resolve
      raise ImportError(str(exc))
  ImportError: type object 'Distribution' has no attribute '_finalize_feature_opts'
  ----------------------------------------
ERROR: Command errored out with exit status 1: /home/user/src/pinnwand/venv/bin/python3.8 /home/user/src/pinnwand/venv/lib/python3.8/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmpbj9ri7zx Check the logs for full command output.
supakeen commented 4 years ago

@zzzeek it seems that I can "fix" the abovementioned 3.8 issue by removing my poetry.lock file and creating a clean virtual environment. I will put some more time in comparing my local environment (even when created cleanly for 3.6/3.7) and the Travis environment because my installs on those versions succeed locally.

This does not stop the Travis build from erroring: https://travis-ci.org/github/supakeen/pinnwand/jobs/664737934

The steps I've taken to resolve the new problem were.

  1. Remove my virtual environment.
  2. Remove my poetry.lock file.
  3. Recreate a virtual environment with python3.8 -m venv venv.
  4. Install the latest versions of pip, setuptools, and poetry with venv/bin/pip install -U pip setuptools wheel poetry.
  5. Running poetry install.

This succesfully installs all my latest dependencies. Here's the poetry.lock diff generated by this. I don't see any relevant packages being updated:

diff --git a/poetry.lock b/poetry.lock
index cb58102..a5b1cf5 100644
--- a/poetry.lock
+++ b/poetry.lock
@@ -181,7 +181,7 @@ marker = "python_version < \"3.7\""
 name = "importlib-resources"
 optional = false
 python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,>=2.7"
-version = "1.3.1"
+version = "1.4.0"

 [package.dependencies]
 [package.dependencies.importlib-metadata]
@@ -193,7 +193,7 @@ python = "<3.8"
 version = ">=0.4"

 [package.extras]
-docs = ["sphinx", "docutils (0.12)", "rst.linker"]
+docs = ["sphinx", "rst.linker", "jaraco.packaging"]

 [[package]]
 category = "dev"
@@ -355,7 +355,7 @@ description = "Better HTML formatter for Pygments"
 name = "pygments-better-html"
 optional = false
 python-versions = "*"
-version = "0.1.0"
+version = "0.1.3"

 [package.dependencies]
 Pygments = ">=2.0.0"
@@ -415,7 +415,7 @@ description = "YAML parser and emitter for Python"
 name = "pyyaml"
 optional = false
 python-versions = ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*"
-version = "5.3"
+version = "5.3.1"

 [[package]]
 category = "dev"
@@ -499,7 +499,7 @@ description = "Virtual Python Environment builder"
 name = "virtualenv"
 optional = false
 python-versions = "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,>=2.7"
-version = "20.0.10"
+version = "20.0.13"
 [package.dependencies]
 appdirs = ">=1.4.3,<2"
@@ -643,8 +643,8 @@ importlib-metadata = [
     {file = "importlib_metadata-1.5.0.tar.gz", hash = "sha256:06f5b3a99029c7134207dd882428a66992a9de2bef7c2b699b5641f9886c3302"},
 ]
 importlib-resources = [
-    {file = "importlib_resources-1.3.1-py2.py3-none-any.whl", hash = "sha256:1dff36d42d94bd523eeb847c25c7dd327cb56686d74a26dfcc8d67c504922d59"},
-    {file = "importlib_resources-1.3.1.tar.gz", hash = "sha256:7f0e1b2b5f3981e39c52da0f99b2955353c5a139c314994d1126c2551ace9bdf"},
+    {file = "importlib_resources-1.4.0-py2.py3-none-any.whl", hash = "sha256:dd98ceeef3f5ad2ef4cc287b8586da4ebad15877f351e9688987ad663a0a29b8"},
+    {file = "importlib_resources-1.4.0.tar.gz", hash = "sha256:4019b6a9082d8ada9def02bece4a76b131518866790d58fdda0b5f8c603b36c2"},
 ]
 mccabe = [
     {file = "mccabe-0.6.1-py2.py3-none-any.whl", hash = "sha256:ab8a6258860da4b6677da4bd2fe5dc2c659cff31b3ee4f7f5d64e79735b80d42"},
@@ -714,8 +714,8 @@ pygments = [
     {file = "Pygments-2.6.1.tar.gz", hash = "sha256:647344a061c249a3b74e230c739f434d7ea4d8b1d5f3721bc0f3558049b38f44"},
 ]
 pygments-better-html = [
-    {file = "pygments_better_html-0.1.0-py3-none-any.whl", hash = "sha256:5706140aa42f58446c67d4d910afe4a0f3dee723b9ecee717252ac1e0e30a38a"},
-    {file = "pygments_better_html-0.1.0.tar.gz", hash = "sha256:16ea025060453add08cbbc9b9c3dc30434fb7957b6f21c674787239d60cd3a79"},
+    {file = "pygments_better_html-0.1.3-py3-none-any.whl", hash = "sha256:5c5955696ebf4e3e855f063b1f95a379ea4ff818c030b364a8a7dccec91f8325"},
+    {file = "pygments_better_html-0.1.3.tar.gz", hash = "sha256:9bddcfdbb08b414c1c34c3bfc56b9825d32277e123b9072995960ad90948be80"},
 ]
 pyparsing = [
     {file = "pyparsing-2.4.6-py2.py3-none-any.whl", hash = "sha256:c342dccb5250c08d45fd6f8b4a559613ca603b57498511740e65cd11a2e7dcec"},
@@ -730,17 +730,17 @@ pytest-cov = [
     {file = "pytest_cov-2.8.1-py2.py3-none-any.whl", hash = "sha256:cdbdef4f870408ebdbfeb44e63e07eb18bb4619fae852f6e760645fa36172626"},
 ]
 pyyaml = [
-    {file = "PyYAML-5.3-cp27-cp27m-win32.whl", hash = "sha256:940532b111b1952befd7db542c370887a8611660d2b9becff75d39355303d82d"},
-    {file = "PyYAML-5.3-cp27-cp27m-win_amd64.whl", hash = "sha256:059b2ee3194d718896c0ad077dd8c043e5e909d9180f387ce42012662a4946d6"},
-    {file = "PyYAML-5.3-cp35-cp35m-win32.whl", hash = "sha256:4fee71aa5bc6ed9d5f116327c04273e25ae31a3020386916905767ec4fc5317e"},
-    {file = "PyYAML-5.3-cp35-cp35m-win_amd64.whl", hash = "sha256:dbbb2379c19ed6042e8f11f2a2c66d39cceb8aeace421bfc29d085d93eda3689"},
-    {file = "PyYAML-5.3-cp36-cp36m-win32.whl", hash = "sha256:e3a057b7a64f1222b56e47bcff5e4b94c4f61faac04c7c4ecb1985e18caa3994"},
-    {file = "PyYAML-5.3-cp36-cp36m-win_amd64.whl", hash = "sha256:74782fbd4d4f87ff04159e986886931456a1894c61229be9eaf4de6f6e44b99e"},
-    {file = "PyYAML-5.3-cp37-cp37m-win32.whl", hash = "sha256:24521fa2890642614558b492b473bee0ac1f8057a7263156b02e8b14c88ce6f5"},
-    {file = "PyYAML-5.3-cp37-cp37m-win_amd64.whl", hash = "sha256:1cf708e2ac57f3aabc87405f04b86354f66799c8e62c28c5fc5f88b5521b2dbf"},
-    {file = "PyYAML-5.3-cp38-cp38-win32.whl", hash = "sha256:70024e02197337533eef7b85b068212420f950319cc8c580261963aefc75f811"},
-    {file = "PyYAML-5.3-cp38-cp38-win_amd64.whl", hash = "sha256:cb1f2f5e426dc9f07a7681419fe39cee823bb74f723f36f70399123f439e9b20"},
-    {file = "PyYAML-5.3.tar.gz", hash = "sha256:e9f45bd5b92c7974e59bcd2dcc8631a6b6cc380a904725fce7bc08872e691615"},
+    {file = "PyYAML-5.3.1-cp27-cp27m-win32.whl", hash = "sha256:74809a57b329d6cc0fdccee6318f44b9b8649961fa73144a98735b0aaf029f1f"},
+    {file = "PyYAML-5.3.1-cp27-cp27m-win_amd64.whl", hash = "sha256:240097ff019d7c70a4922b6869d8a86407758333f02203e0fc6ff79c5dcede76"},
+    {file = "PyYAML-5.3.1-cp35-cp35m-win32.whl", hash = "sha256:4f4b913ca1a7319b33cfb1369e91e50354d6f07a135f3b901aca02aa95940bd2"},
+    {file = "PyYAML-5.3.1-cp35-cp35m-win_amd64.whl", hash = "sha256:cc8955cfbfc7a115fa81d85284ee61147059a753344bc51098f3ccd69b0d7e0c"},
+    {file = "PyYAML-5.3.1-cp36-cp36m-win32.whl", hash = "sha256:7739fc0fa8205b3ee8808aea45e968bc90082c10aef6ea95e855e10abf4a37b2"},
+    {file = "PyYAML-5.3.1-cp36-cp36m-win_amd64.whl", hash = "sha256:69f00dca373f240f842b2931fb2c7e14ddbacd1397d57157a9b005a6a9942648"},
+    {file = "PyYAML-5.3.1-cp37-cp37m-win32.whl", hash = "sha256:d13155f591e6fcc1ec3b30685d50bf0711574e2c0dfffd7644babf8b5102ca1a"},
+    {file = "PyYAML-5.3.1-cp37-cp37m-win_amd64.whl", hash = "sha256:73f099454b799e05e5ab51423c7bcf361c58d3206fa7b0d555426b1f4d9a3eaf"},
+    {file = "PyYAML-5.3.1-cp38-cp38-win32.whl", hash = "sha256:06a0d7ba600ce0b2d2fe2e78453a470b5a6e000a985dd4a4e54e436cc36b0e97"},
+    {file = "PyYAML-5.3.1-cp38-cp38-win_amd64.whl", hash = "sha256:95f71d2af0ff4227885f7a6605c37fd53d3a106fcab511b8860ecca9fcf400ee"},
+    {file = "PyYAML-5.3.1.tar.gz", hash = "sha256:b8eac752c5e14d3eca0e6dd9199cd627518cb5ec06add0de9d32baeee6fe645d"},
 ]
 regex = [
     {file = "regex-2020.2.20-cp27-cp27m-win32.whl", hash = "sha256:99272d6b6a68c7ae4391908fc15f6b8c9a6c345a46b632d7fdb7ef6c883a2bbb"},
@@ -821,8 +821,8 @@ typing-extensions = [
     {file = "typing_extensions-3.7.4.1.tar.gz", hash = "sha256:091ecc894d5e908ac75209f10d5b4f118fbdb2eb1ede6a63544054bb1edb41f2"},
 ]
 virtualenv = [
-    {file = "virtualenv-20.0.10-py2.py3-none-any.whl", hash = "sha256:10750cac3b5a9e6eed54d0f1f8222c550dc47f84609c95cbc504d44a58a048b8"},
-    {file = "virtualenv-20.0.10.tar.gz", hash = "sha256:8512e83f1d90f8e481024d58512ac9c053bf16f54d9138520a0929396820dd78"},
+    {file = "virtualenv-20.0.13-py2.py3-none-any.whl", hash = "sha256:87831f1070534b636fea2241dd66f3afe37ac9041bcca6d0af3215cdcfbf7d82"},
+    {file = "virtualenv-20.0.13.tar.gz", hash = "sha256:f3128d882383c503003130389bf892856341c1da12c881ae24d6358c82561b55"},
 ]
 wcwidth = [
     {file = "wcwidth-0.1.8-py2.py3-none-any.whl", hash = "sha256:8fd29383f539be45b20bd4df0dc29c20ba48654a41e661925e612311e9f3c603"},
rvanlaar commented 4 years ago

It's weird.

I'm using the python:3.5.2 docker container. Upgraded pip to 20.0.2 and setuptools to 46.0.0. wheel is already 0.34.2

Removed the poetry.lock file, sqlalchemy and re-added it. Now it's working.

For bitbucket-pipelines I also had to clear my cache. It's working now.

github-actions[bot] commented 6 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.