Dependency walk failed - Githubissues

SunnyR commented 1 year ago

Poetry version: 1.3.2
Python version: 3.8
OS version and name: wsl ubuntu 20.04
pyproject.toml: https://gist.github.com/SunnyR/eedd0f0d974b4c046a660f354eb65dee

[x] I am on the latest stable Poetry version, installed using a recommended method.
[x] I have searched the issues of this repo and believe that this is not a duplicate.
[x] I have consulted the FAQ and blog for any relevant entries or release notes.
[x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

When using poetry export with dev dependencies (following command), i get stack trace below with error: poetry export -f requirements.txt -o ./dev_dependencies --without-hashes --with dev

Stack trace:

  12  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/cleo/application.py:327 in run
       325│ 
       326│             try:
     → 327│                 exit_code = self._run(io)
       328│             except BrokenPipeError:
       329│                 # If we are piped to another process, it may close early and send a

  11  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/console/application.py:190 in _run
       188│         self._load_plugins(io)
       189│ 
     → 190│         exit_code: int = super()._run(io)
       191│         return exit_code
       192│ 

  10  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/cleo/application.py:431 in _run
       429│             io.input.interactive(interactive)
       430│ 
     → 431│         exit_code = self._run_command(command, io)
       432│         self._running_command = None
       433│ 

   9  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/cleo/application.py:473 in _run_command
       471│ 
       472│         if error is not None:
     → 473│             raise error
       474│ 
       475│         return terminate_event.exit_code

   8  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/cleo/application.py:457 in _run_command
       455│ 
       456│             if command_event.command_should_run():
     → 457│                 exit_code = command.run(io)
       458│             else:
       459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

   7  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/cleo/commands/base_command.py:119 in run
       117│         io.input.validate()
       118│ 
     → 119│         status_code = self.execute(io)
       120│ 
       121│         if status_code is None:

   6  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/cleo/commands/command.py:62 in execute
        60│ 
        61│         try:
     →  62│             return self.handle()
        63│         except KeyboardInterrupt:
        64│             return 1

   5  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry_plugin_export/command.py:107 in handle
       105│         exporter.with_credentials(self.option("with-credentials"))
       106│         exporter.with_urls(not self.option("without-urls"))
     → 107│         exporter.export(fmt, Path.cwd(), output or self.io)
       108│ 

   4  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry_plugin_export/exporter.py:86 in export
        84│             raise ValueError(f"Invalid export format: {fmt}")
        85│ 
     →  86│         getattr(self, self.EXPORT_METHODS[fmt])(cwd, output)
        87│ 
        88│     def _export_generic_txt(

   3  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry_plugin_export/exporter.py:101 in _export_generic_txt
        99│         )
       100│ 
     → 101│         for dependency_package in get_project_dependency_packages(
       102│             self._poetry.locker,
       103│             project_requires=root.all_requires,

   2  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry_plugin_export/walker.py:103 in get_project_dependency_packages
       101│         selected.append(dependency)
       102│ 
     → 103│     for package, dependency in get_project_dependencies(
       104│         project_requires=selected,
       105│         locked_packages=repository.packages,

   1  ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry_plugin_export/walker.py:129 in get_project_dependencies
       127│         )
       128│ 
     → 129│     nested_dependencies = walk_dependencies(
       130│         dependencies=project_requires,
       131│         packages_by_name=packages_by_name,

  RuntimeError

  Dependency walk failed at click (>=6.5)

  at ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry_plugin_export/walker.py:155 in walk_dependencies
      151│             requirement, packages_by_name, nested_dependencies
      152│         )
      153│ 
      154│         if not locked_package:
    → 155│             raise RuntimeError(f"Dependency walk failed at {requirement}")
      156│ 
      157│         if requirement.extras:
      158│             locked_package = locked_package.with_features(requirement.extras)
      159│

neersighted commented 1 year ago

You have not provided a pyproject.toml as requested in the issue template -- can you please provide that information so that people can help you debug?

finswimmer commented 1 year ago

I guess this is a duplicate of https://github.com/python-poetry/poetry-plugin-export/issues/173

SunnyR commented 1 year ago

@neersighted updated the issue description with pyproject.toml

@finswimmer i'm already using a python constraint so i don't believe this is a duplicate. Let me know.

dimbleby commented 1 year ago

It's hard to know what to make of this. So far as I can see poetry simply doesn't support source distributions with file suffix .tgz, which is all that is available at https://pypi.org/project/amqplib/1.0.2/#files, therefore the given pyproject.toml can't even lock.

So I don't know how this becomes an export problem...

SunnyR commented 1 year ago

I've updated the pyproject.toml file with packages that are the culprit - click and black.

Also, this issue might be related - https://github.com/psf/black/issues/2964, but i'm still not sure why export is not working?

SunnyR commented 1 year ago

Tweaked my pyproject.toml a bit more and was able to resolve the issue by specifying black versions for each python version.

Marking closed.

neersighted commented 1 year ago

I don't know that this should be closed -- we shouldn't fail to export a valid lock file generated from a valid pyproject.

mhamiltonj commented 1 year ago

I'm having the same issue, not sure if it's the same cause. I think in our case it is related to #28. Our poetry.lock conatins duplicate entries for scipy (versions 1.8.1 and 1.10.0). This doesn't seem to be a problem, the poetry instally 1.10 and the project works fine. But export fails with Dependency walk failed at scipy (>=1.3,<1.9). I have no idea where the <1.9 constraint is coming from.

In our case a workaround is adding a direct scipy = ">=1.10" dependency in our pyproject.toml, but this seems artificial, since scipy is a subdependency, not a direct dependency for us.

marcorichetta commented 1 year ago

Same as @mhamiltonj In our case, the issue is https://github.com/statsmodels/statsmodels/issues/8543 Workaround: poetry add "statsmodels@^0.13.2,!=0.13.3,!=0.13.4,!=0.13.5"

neersighted commented 1 year ago

Adding a direct dependency is not exactly a workaround; you should add constraints for anything you care to perturb/alter, not just anything you import directly. That being said, it is a workaround in this specific case as you are trying to perturb the lock file for us with this plugin, and not the dependency tree as a whole.

colindean commented 1 year ago

I encountered this Dependency walk failed at scipy (>=1.3,<1.9) error today.

I am able to reproduce this with this pyproject.toml and poetry.lock.

$ poetry debug info

Poetry
Version: 1.4.0
Python:  3.9.15

Virtualenv
Python:         3.9.15
Implementation: CPython
Path:           /Users/colin/Library/Caches/pypoetry/virtualenvs/poetry-export-bug-gh176-qC8Y9brs-py3.9
Executable:     /Users/colin/Library/Caches/pypoetry/virtualenvs/poetry-export-bug-gh176-qC8Y9brs-py3.9/bin/python
Valid:          True

System
Platform:   darwin
OS:         posix
Python:     3.9.15
Path:       /Users/colin/.pyenv/versions/3.9.15
Executable: /Users/colin/.pyenv/versions/3.9.15/bin/python3.9

N.b. it's got two scipy references in it: https://gist.github.com/colindean/e1b7326a567c44b5708cc822f9c7f561#file-poetry-lock-L1444-L1478

I think it might have something to do with the statsmodels package, which is coming into this through pyod. Specifically statsmodels spec for scipy:

scipy = [
    {version = ">=1.3", markers = "python_version > \"3.9\" and python_version < \"3.12\" or platform_system != \"Windows\" and python_version < \"3.12\" or platform_machine != \"x86\" and python_version < \"3.12\""},
    {version = ">=1.3,<1.9", markers = "python_version == \"3.9\" and platform_system == \"Windows\" and platform_machine == \"x86\""},
]

novas0x2a commented 1 year ago

poetry export also appears to fail on poetry itself:

$ poetry debug info

Poetry
Version: 1.4.0
Python:  3.10.6

Virtualenv
Python:         3.10.6
Implementation: CPython
Path:           /home/mlundy/.cache/pypoetry/virtualenvs/poetry-0f7nDIKJ-py3.10
Executable:     /home/mlundy/.cache/pypoetry/virtualenvs/poetry-0f7nDIKJ-py3.10/bin/python
Valid:          True

System
Platform:   linux
OS:         posix
Python:     3.10.6
Path:       /usr
Executable: /usr/bin/python3.10

$ poetry self show plugins

  • poetry-plugin-export (1.3.0) Poetry plugin to export the dependencies to various formats
      1 application plugin

      Dependencies
        - poetry (>=1.3.0,<2.0.0)
        - poetry-core (>=1.3.0,<2.0.0)

$ GIT_PAGER=cat git show --no-patch
commit 0e72a55c43a993ec0258facec23416c9212964ba (HEAD, tag: 1.4.0)
Author: Randy Döring <30527984+radoering@users.noreply.github.com>
Date:   Sat Feb 18 04:19:46 2023

    release: bump version to 1.4.0

$ poetry export -vvv
Loading configuration file /home/mlundy/.config/pypoetry/config.toml
Source (poetry-lockfile): 1 packages found for poetry-core 1.5.1
Source (poetry-lockfile): 1 packages found for poetry-plugin-export >=1.3.0,<2.0.0
Source (poetry-lockfile): 1 packages found for backports-cached-property >=1.0.2,<2.0.0
Source (poetry-lockfile): 1 packages found for build >=0.10.0,<0.11.0
Source (poetry-lockfile): 1 packages found for cachecontrol >=0.12.9,<0.13.0
Source (poetry-lockfile): 1 packages found for cleo >=2.0.0,<3.0.0
Source (poetry-lockfile): 1 packages found for crashtest >=0.4.1,<0.5.0
Source (poetry-lockfile): 1 packages found for dulwich >=0.21.2,<0.22.0
Source (poetry-lockfile): 1 packages found for filelock >=3.8.0,<4.0.0
Source (poetry-lockfile): 1 packages found for html5lib >=1.0,<2.0
Source (poetry-lockfile): 1 packages found for importlib-metadata >=4.4
Source (poetry-lockfile): 1 packages found for installer >=0.6.0,<0.7.0
Source (poetry-lockfile): 1 packages found for jsonschema >=4.10.0,<5.0.0
Source (poetry-lockfile): 1 packages found for keyring >=23.9.0,<24.0.0
Source (poetry-lockfile): 1 packages found for lockfile >=0.12.2,<0.13.0
Source (poetry-lockfile): 1 packages found for packaging >=20.4
Source (poetry-lockfile): 1 packages found for pexpect >=4.7.0,<5.0.0
Source (poetry-lockfile): 1 packages found for pkginfo >=1.9.4,<2.0.0
Source (poetry-lockfile): 1 packages found for platformdirs >=2.5.2,<3.0.0
Source (poetry-lockfile): 1 packages found for pyproject-hooks >=1.0.0,<2.0.0
Source (poetry-lockfile): 1 packages found for requests >=2.18,<3.0
Source (poetry-lockfile): 1 packages found for requests-toolbelt >=0.9.1,<0.11.0
Source (poetry-lockfile): 1 packages found for shellingham >=1.5,<2.0
Source (poetry-lockfile): 1 packages found for tomli >=2.0.1,<3.0.0
Source (poetry-lockfile): 1 packages found for tomlkit >=0.11.1,<0.11.2 || >0.11.2,<0.11.3 || >0.11.3,<1.0.0
Source (poetry-lockfile): 1 packages found for trove-classifiers >=2022.5.19
Source (poetry-lockfile): 2 packages found for virtualenv >=20.4.3,<20.4.5 || >20.4.5,<20.4.6 || >20.4.6,<21.0.0
Source (poetry-lockfile): 1 packages found for virtualenv <20.16.6
Source (poetry-lockfile): 1 packages found for xattr >=0.10.0,<0.11.0
Source (poetry-lockfile): 1 packages found for urllib3 >=1.26.0,<2.0.0

  Stack trace:

  12  ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/cleo/application.py:327 in run
       325│ 
       326│             try:
     → 327│                 exit_code = self._run(io)
       328│             except BrokenPipeError:
       329│                 # If we are piped to another process, it may close early and send a

  11  ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/poetry/console/application.py:190 in _run
       188│         self._load_plugins(io)
       189│ 
     → 190│         exit_code: int = super()._run(io)
       191│         return exit_code
       192│ 

  10  ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/cleo/application.py:431 in _run
       429│             io.input.interactive(interactive)
       430│ 
     → 431│         exit_code = self._run_command(command, io)
       432│         self._running_command = None
       433│ 

   9  ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/cleo/application.py:473 in _run_command
       471│ 
       472│         if error is not None:
     → 473│             raise error
       474│ 
       475│         return terminate_event.exit_code

   8  ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/cleo/application.py:457 in _run_command
       455│ 
       456│             if command_event.command_should_run():
     → 457│                 exit_code = command.run(io)
       458│             else:
       459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

   7  ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/cleo/commands/base_command.py:119 in run
       117│         io.input.validate()
       118│ 
     → 119│         status_code = self.execute(io)
       120│ 
       121│         if status_code is None:

   6  ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/cleo/commands/command.py:62 in execute
        60│ 
        61│         try:
     →  62│             return self.handle()
        63│         except KeyboardInterrupt:
        64│             return 1

   5  ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/poetry_plugin_export/command.py:107 in handle
       105│         exporter.with_credentials(self.option("with-credentials"))
       106│         exporter.with_urls(not self.option("without-urls"))
     → 107│         exporter.export(fmt, Path.cwd(), output or self.io)
       108│ 
       109│         return 0

   4  ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/poetry_plugin_export/exporter.py:80 in export
        78│             raise ValueError(f"Invalid export format: {fmt}")
        79│ 
     →  80│         getattr(self, self.EXPORT_METHODS[fmt])(cwd, output)
        81│ 
        82│     def _export_generic_txt(

   3  ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/poetry_plugin_export/exporter.py:95 in _export_generic_txt
        93│         )
        94│ 
     →  95│         for dependency_package in get_project_dependency_packages(
        96│             self._poetry.locker,
        97│             project_requires=root.all_requires,

   2  ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/poetry_plugin_export/walker.py:97 in get_project_dependency_packages
        95│         selected.append(dependency)
        96│ 
     →  97│     for package, dependency in get_project_dependencies(
        98│         project_requires=selected,
        99│         locked_packages=repository.packages,

   1  ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/poetry_plugin_export/walker.py:125 in get_project_dependencies
       123│         )
       124│ 
     → 125│     nested_dependencies = walk_dependencies(
       126│         dependencies=project_requires,
       127│         packages_by_name=packages_by_name,

  RuntimeError

  Dependency walk failed at virtualenv (<20.16.6)

  at ~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/poetry_plugin_export/walker.py:155 in walk_dependencies
      151│             requirement, packages_by_name, nested_dependencies
      152│         )
      153│ 
      154│         if not locked_package:
    → 155│             raise RuntimeError(f"Dependency walk failed at {requirement}")
      156│ 
      157│         if requirement.extras:
      158│             locked_package = locked_package.with_features(requirement.extras)
      159│

Agalin commented 1 year ago

Just as @novas0x2a said - even poetry itself fails as it defines separate virtualenv dependency range for Windows and Python 3.9:

virtualenv = [
    {version = ">=20.4.3,<20.4.5 || >20.4.5,<20.4.6 || >20.4.6,<21.0.0", markers = "sys_platform != \"win32\" or python_version != \"3.9\""},
    {version = ">=20.4.3,<20.4.5 || >20.4.5,<20.4.6 || >20.4.6,<20.16.6", markers = "sys_platform == \"win32\" and python_version == \"3.9\""},
]

novas0x2a commented 1 year ago

linking to a closely-related issue: #168

SunnyR commented 1 year ago

Also, the culprit seems to be this block of code - https://github.com/python-poetry/poetry-plugin-export/blob/main/src/poetry_plugin_export/walker.py#L235. @finswimmer

gsemet commented 10 months ago

I see this error on several newer packages:

zipp 3.16.2+
typing-extensions 4.8.0

when they are part of the dependency graph, poetry fails with "Dependency walk failed" error.

david-nano commented 8 months ago

Anything came up since? I'm using poetry==1.7.1

Edit: I've tried to add specific version of scipy and found that my python version (3.8) isn't supported for the latest scipy, which require minimum of 3.9

alanwilter commented 4 months ago

Hi, I need help, I have a similar issue, using poetry 1.8.2 and poetry-plugin-export (1.7.1).

poetry export --without-hashes --format=requirements.txt | sed 's#file:///.*/heyex-pipe/#file:///var/task/#g' > requirements.txt
Dependency walk failed at urllib3 (>=1.21.1,<3)

From my pyproject.toml:

[tool.poetry.dependencies]
python = ">=3.9,<3.11"
typed-argument-parser = "^1.7.2"
opencv-python = "4.6.0.66"
pillow = "^9.3.0"
tqdm = "^4.64.1"
scikit-image = "^0.19.3"
numpy = "^1.23.5"
tornado = "^6.2"
parse = "^1.19.0"
more-itertools = "^9.0.0"
matplotlib = "^3.7.0"
tensorflow = "2.13.1"
humanize = "^4.8.0"
torch = "2.0.1"
boto3 = "^1.34.0"

[tool.poetry.group.dev.dependencies]
pre-commit = "^2.20.0"
pytest-cov = "^4.0.0"
ipdb = "^0.13.11"
pdbpp = "^0.10.3"
boto3-stubs = { extras = ["lambda", "s3"], version = "^1.34.0" }
types-pillow = "^9.4.0.12"
types-tqdm = "^4.64.7.13"
pytest-xdist = "^3.2.0"
keras = "^2.11.0"
onnxruntime-gpu = "^1.13.1"
pip = "^23.1.2"
wheel = "^0.40.0"
ruff = "^0.1.3"
mypy = "^1.6.1"
types-requests = "2.31.0.6"
mypy-boto3-lambda = "^1.34.0"
mypy-boto3-s3 = "^1.34.0"

Never mind, the issue is the way botocore deals with dependencies. Upgraded to Python 3.10 and it seems to work now.

python-poetry / poetry-plugin-export

Dependency walk failed #176

Issue