Python restricted dependency not working

[carlotardivo]

• Poetry version: 1.2.2

• Python version: 3.9.13

• OS version and name: macOS Catalina, Version 0.15.7

• [x] I am on the latest stable Poetry version, installed using a recommended method.

• [x] I have searched the issues of this repo and believe that this is not a duplicate.

• [x] I have consulted the FAQ and blog for any relevant entries or release notes.

• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

Hi, I'm currently working on a multirepo context and I have a pyproject.toml file that has some other repos between its dependencies. I tried to recreate the situation with three empty repos lib_a, lib_b, lib_c where lib_a depends on the other two. To reproduce the error just clone this repo: https://github.com/carlotardivo/poetry-issue-6895 and execute a poetry lock -vvv from lib_a root directory.

These are the three pyproject.toml files:

lib_a:

[tool.poetry]
name = "lib_a"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
readme = "README.md"
packages = [{include = "liba"}]

[tool.poetry.dependencies]
python = ">=3.7.13,<3.10"
lib_b={path='../lib_b'}
lib_c={path='../lib_c', optional= true, python = ">=3.9,<3.10"}

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

lib_b

[tool.poetry]
name = "lib_b"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
readme = "README.md"
packages = [{include = "libb"}]

[tool.poetry.dependencies]
python = ">=3.7.13,<3.10"
scipy=[
    {version='1.5.*', python='~3.7.13'},
    {version='1.8.0', python='>=3.8'}
]

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

lib_c:

[tool.poetry]
name = "lib_c"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
readme = "README.md"
packages = [{include = "libc"}]

[tool.poetry.dependencies]
python = ">=3.9,<3.10"
scipy = { version = "^1.7.0", python = ">=3.9,<3.10" }

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Expected result

Basically what I'd expect, according to these docs (python restrcted dependencies) is to be able to create a lock file that allows for the creation of an environment with

• lib_b (with scipy 1.5.*) when python ~3.7.13
• lib_b (with scipy 1.8.0) and lib_c installed only when python >=3.9,<3.10

Actual result

The lock file generation breaks with the following error, that seems related to the fact that scipy 1.5 is looked also for python >=3.9,<3.10 (while only ^1.7.0 should in this python range):

(poetry-experiment-py3.9) bash-3.2$ poetry lock -vvv
Loading configuration file /<my-user-local-path>/Library/Preferences/pypoetry/config.toml
Loading configuration file /<my-user-local-path>/Library/Preferences/pypoetry/auth.toml
Using virtualenv: /<my-user-local-path>/Library/Caches/pypoetry/virtualenvs/poetry-experiment-7exzTMzg-py3.9
Project environment contains an empty path in sys_path, ignoring.
Updating dependencies
Resolving dependencies...
   1: fact: lib-a is 0.1.0
   1: derived: lib-a
   1: fact: lib-a depends on lib_b (0.1.0)
   1: fact: lib-a depends on lib_c (0.1.0)
   1: selecting lib-a (0.1.0)
   1: derived: lib_c (0.1.0) @ ../lib_c
   1: derived: lib_b (0.1.0) @ ../lib_b
   1: fact: lib-c (0.1.0) depends on scipy (^1.7.0)
   1: selecting lib-c (0.1.0 /<my-user-local-path>/Desktop/poetry_experiment/lib_c)
   1: derived: scipy (>=1.7.0,<2.0.0)
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
Creating new session for pypi.org
[urllib3.connectionpool] Starting new HTTPS connection (1): pypi.org:443
[urllib3.connectionpool] https://pypi.org:443 "GET /simple/scipy/ HTTP/1.1" 304 0
[filelock] Attempting to acquire lock 4437848368 on /<my-user-local-path>/Library/Caches/pypoetry/cache/repositories/PyPI/_http/c/d/5/d/e/cd5de44b899cbe1869765c21b55cffc53c0c0ffa9d3c6f1fd40a42a2.lock
[filelock] Lock 4437848368 acquired on /<my-user-local-path>/Library/Caches/pypoetry/cache/repositories/PyPI/_http/c/d/5/d/e/cd5de44b899cbe1869765c21b55cffc53c0c0ffa9d3c6f1fd40a42a2.lock
[filelock] Attempting to release lock 4437848368 on /<my-user-local-path>/Library/Caches/pypoetry/cache/repositories/PyPI/_http/c/d/5/d/e/cd5de44b899cbe1869765c21b55cffc53c0c0ffa9d3c6f1fd40a42a2.lock
[filelock] Lock 4437848368 released on /<my-user-local-path>/Library/Caches/pypoetry/cache/repositories/PyPI/_http/c/d/5/d/e/cd5de44b899cbe1869765c21b55cffc53c0c0ffa9d3c6f1fd40a42a2.lock
Source (PyPI): 10 packages found for scipy >=1.7.0,<2.0.0
   0: Duplicate dependencies for scipy
   0: Different requirements found for scipy (1.5.*) with markers python_full_version >= "3.7.13" and python_full_version < "3.8.0" and scipy (1.8.0) with markers python_version >= "3.8".
   1: Version solving took 0.233 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('lib-b', '0.1.0', source_type='directory', source_url='/<my-user-local-path>/Desktop/poetry_experiment/lib_b'): {'scipy': <Dependency scipy (>=1.5.0,<1.6.0)>}}).
   1: fact: lib-a is 0.1.0
   1: derived: lib-a
   1: fact: lib-a depends on lib_b (0.1.0)
   1: fact: lib-a depends on lib_c (0.1.0)
   1: selecting lib-a (0.1.0)
   1: derived: lib_c (0.1.0) @ ../lib_c
   1: derived: lib_b (0.1.0) @ ../lib_b
   1: fact: lib-c (0.1.0) depends on scipy (^1.7.0)
   1: selecting lib-c (0.1.0 /<my-user-local-path>/Desktop/poetry_experiment/lib_c)
   1: derived: scipy (>=1.7.0,<2.0.0)
Source (PyPI): 10 packages found for scipy >=1.7.0,<2.0.0
   1: fact: lib-b (0.1.0) depends on scipy (1.5.*)
   1: conflict: lib-b (0.1.0) depends on scipy (1.5.*)
   1: derived: scipy (>=1.5.0,<1.6.0)
   1: conflict: lib-c (0.1.0) depends on scipy (^1.7.0)
   1: ! not scipy (>=1.7.0,<2.0.0) is satisfied by scipy (>=1.5.0,<1.6.0)
   1: ! which is caused by "lib-b (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_b depends on scipy (1.5.*)"
   1: ! thus: lib-c (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_c is incompatible with lib-b (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_b
   1: ! lib-b (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_b is satisfied by lib_b (0.1.0) @ ../lib_b
   1: ! which is caused by "lib-a depends on lib_b (0.1.0) @ ../lib_b"
   1: ! thus: lib-c is forbidden
   1: ! lib-c (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_c is satisfied by lib_c (0.1.0) @ ../lib_c
   1: ! which is caused by "lib-a depends on lib_c (0.1.0) @ ../lib_c"
   1: ! thus: version solving failed
   1: Version solving took 0.016 seconds.
   1: Tried 1 solutions.

  Stack trace:

  4  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/solver.py:151 in _solve
      149│ 
      150│         try:
    → 151│             result = resolve_version(
      152│                 self._package, self._provider, locked=locked, use_latest=use_latest
      153│             )

  3  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/mixology/__init__.py:24 in resolve_version
       22│     solver = VersionSolver(root, provider, locked=locked, use_latest=use_latest)
       23│ 
    →  24│     return solver.solve()
       25│ 

  2  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/mixology/version_solver.py:127 in solve
      125│             while next is not None:
      126│                 self._propagate(next)
    → 127│                 next = self._choose_package_version()
      128│ 
      129│             return self._result()

  1  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/mixology/version_solver.py:446 in _choose_package_version
      444│             package = locked
      445│ 
    → 446│         package = self._provider.complete_package(package)
      447│ 
      448│         conflict = False

  OverrideNeeded

  ({Package('lib-b', '0.1.0', source_type='directory', source_url='/<my-user-local-path>/Desktop/poetry_experiment/lib_b'): {'scipy': <Dependency scipy (>=1.5.0,<1.6.0)>}}, {Package('lib-b', '0.1.0', source_type='directory', source_url='/<my-user-local-path>/Desktop/poetry_experiment/lib_b'): {'scipy': <Dependency scipy (==1.8.0)>}})

  at ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/provider.py:764 in complete_package
      760│                     current_overrides.update({dependency_package: package_overrides})
      761│                     overrides.append(current_overrides)
      762│ 
      763│             if overrides:
    → 764│                 raise OverrideNeeded(*overrides)
      765│ 
      766│         # Modifying dependencies as needed
      767│         clean_dependencies = []
      768│         for dep in dependencies:

The following error occurred when trying to handle this error:

  Stack trace:

  4  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/solver.py:151 in _solve
      149│ 
      150│         try:
    → 151│             result = resolve_version(
      152│                 self._package, self._provider, locked=locked, use_latest=use_latest
      153│             )

  3  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/mixology/__init__.py:24 in resolve_version
       22│     solver = VersionSolver(root, provider, locked=locked, use_latest=use_latest)
       23│ 
    →  24│     return solver.solve()
       25│ 

  2  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/mixology/version_solver.py:126 in solve
      124│             next: str | None = self._root.name
      125│             while next is not None:
    → 126│                 self._propagate(next)
      127│                 next = self._choose_package_version()
      128│ 

  1  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/mixology/version_solver.py:165 in _propagate
      163│                     # where that incompatibility will allow us to derive new assignments
      164│                     # that avoid the conflict.
    → 165│                     root_cause = self._resolve_conflict(incompatibility)
      166│ 
      167│                     # Back jumping erases all the assignments we did at the previous

  SolveFailure

  Because lib-c (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_c depends on scipy (^1.7.0)
   and lib-b (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_b depends on scipy (1.5.*), lib-c (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_c is incompatible with lib-b (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_b.
  So, because lib-a depends on both lib_b (0.1.0) @ ../lib_b and lib_c (0.1.0) @ ../lib_c, version solving failed.

  at ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/mixology/version_solver.py:364 in _resolve_conflict
      360│             )
      361│             self._log(f'! which is caused by "{most_recent_satisfier.cause}"')
      362│             self._log(f"! thus: {incompatibility}")
      363│ 
    → 364│         raise SolveFailure(incompatibility)
      365│ 
      366│     def _choose_package_version(self) -> str | None:
      367│         """
      368│         Tries to select a version of a required package.

The following error occurred when trying to handle this error:

  Stack trace:

  13  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/application.py:329 in run
       327│ 
       328│             try:
     → 329│                 exit_code = self._run(io)
       330│             except Exception as e:
       331│                 if not self._catch_exceptions:

  12  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/console/application.py:185 in _run
       183│         self._load_plugins(io)
       184│ 
     → 185│         exit_code: int = super()._run(io)
       186│         return exit_code
       187│ 

  11  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/application.py:423 in _run
       421│             io.input.set_stream(stream)
       422│ 
     → 423│         exit_code = self._run_command(command, io)
       424│         self._running_command = None
       425│ 

  10  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/application.py:465 in _run_command
       463│ 
       464│         if error is not None:
     → 465│             raise error
       466│ 
       467│         return event.exit_code

   9  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/application.py:449 in _run_command
       447│ 
       448│             if event.command_should_run():
     → 449│                 exit_code = command.run(io)
       450│             else:
       451│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

   8  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/commands/base_command.py:119 in run
       117│         io.input.validate()
       118│ 
     → 119│         status_code = self.execute(io)
       120│ 
       121│         if status_code is None:

   7  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/cleo/commands/command.py:83 in execute
        81│ 
        82│         try:
     →  83│             return self.handle()
        84│         except KeyboardInterrupt:
        85│             return 1

   6  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/console/commands/lock.py:54 in handle
        52│         self.installer.lock(update=not self.option("no-update"))
        53│ 
     →  54│         return self.installer.run()
        55│ 

   5  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/installation/installer.py:115 in run
       113│             self._execute_operations = False
       114│ 
     → 115│         return self._do_install()
       116│ 
       117│     def dry_run(self, dry_run: bool = True) -> Installer:

   4  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/installation/installer.py:248 in _do_install
       246│                 source_root=self._env.path.joinpath("src")
       247│             ):
     → 248│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       249│         else:
       250│             self._io.write_line("Installing dependencies from lock file")

   3  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/solver.py:73 in solve
        71│         with self._provider.progress():
        72│             start = time.time()
     →  73│             packages, depths = self._solve(use_latest=use_latest)
        74│             end = time.time()
        75│ 

   2  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/solver.py:157 in _solve
       155│             packages = result.packages
       156│         except OverrideNeeded as e:
     → 157│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
       158│         except SolveFailure as e:
       159│             raise SolverProblemError(e)

   1  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/solver.py:116 in solve_in_compatibility_mode
       114│             )
       115│             self._provider.set_overrides(override)
     → 116│             _packages, _depths = self._solve(use_latest=use_latest)
       117│             for index, package in enumerate(_packages):
       118│                 if package not in packages:

  SolverProblemError

  Because lib-c (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_c depends on scipy (^1.7.0)
   and lib-b (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_b depends on scipy (1.5.*), lib-c (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_c is incompatible with lib-b (0.1.0) @ file:///<my-user-local-path>/Desktop/poetry_experiment/lib_b.
  So, because lib-a depends on both lib_b (0.1.0) @ ../lib_b and lib_c (0.1.0) @ ../lib_c, version solving failed.

  at ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/solver.py:159 in _solve
      155│             packages = result.packages
      156│         except OverrideNeeded as e:
      157│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      158│         except SolveFailure as e:
    → 159│             raise SolverProblemError(e)
      160│ 
      161│         combined_nodes = depth_first_search(PackageNode(self._package, packages))
      162│         results = dict(aggregate_package_nodes(nodes) for nodes in combined_nodes)
      163│ 

I'm I missing something?

Thank you very much.

[neersighted]

Please create a repo that reproduces this as there's enough complexity that a search-and-replace of (likely partially tested) reproducer pyproject.tomls is fraught. That being said, I immediately notice you have optional = true for a dependency -- that is a feature used by extras only, and should not be used with the marker-based selection you are using.

[gianfa]

@neersighted Does this mean that the use of extra groups is not compatible with the use of markers?

At this point I wonder if there is a hierarchy of use among groups, extras, and markers in the dependencies, but more importantly whether there are exclusion constraints ("this is not allowed with that"). By any chance are there clear references on this?

[carlotardivo]

I @neersighted, thanks for the quick answer. That optional = True was left by mistake (in the original use case the toml file had an actual extra dependency), however the problem remains the same after deleting it (in any case I'm quite interested on having an answer to @gianfa questions too).

If it is good for you I'm going to leave the optional = True in the issue, and I also added a link to a repo that can be used to reproduce the error.

PS: in lib_c the python = ">=3.9,<3.10" marker on the scipy dependency should be redundant, but I don't think it can be the source of the problem.

[neersighted]

They're not incompatible; but without seeing a complete real (sanitized real, as opposed to synthetic recreation, as many users do not fully test their minimal reproductions) pyproject.toml(s), it's hard to know if it's not some use of extras not included in the reproduction that breaks things. We have that now, so I'm confident that was merely a red herring.

[dimbleby]

probably duplicates #5506

[carlotardivo]

I @neersighted, sorry but I cannot fully understand what do you mean with complete real example. I linked a mock repo that can be easily cloned and where the error can be reproduced. Has anybody tried this? Did you have the same unexpected output as mine? If so, I think this should be labelled as a bug.

[neersighted]

We have that now

We have a real clonable repro; I have not had time to investigate, but thank you for providing it.

If so, I think this should be labelled as a bug.

This is labeled as a bug, and as requiring triage/someone to reproduce and confirm the details.

[jylind]

Hi, adding my question here as it looks like a duplicate or something really close. Using Poetry 1.2.2 too.

[tool.poetry]
name = "RobotRun"
version = "1.0.0"
description = "Test Automation Framework"
authors = ["Tester <tester@mail.com>"]

[tool.poetry.dependencies]
python = ">=3.8, <3.12"
robotframework = [
    {version = "3.1.2", python = "~3.8"},
    {version = "~6", python = "^3.10"}
]

[tool.poetry.group.dev]
optional = true

[tool.poetry.group.dev.dependencies]
robotframework-tidy = { version = "^3", python = ">=3.10"}

[build-system]
requires = ["poetry-core>=1.2"]
build-backend = "poetry.core.masonry.api"

With above config shouldn't the pass lock file creation be successful as robotframework-tidy is only installed if Python is 3.10 or higher, when we are also having robotframework 6? Now it fails with: So, because robotrun depends on both robotframework (3.1.2) and robotframework-tidy (^3), version solving failed.

When RF 3.1.2 is changed to 4 (and resolve passes). the poetry install work as expected and only installs robotframework-tidy when Python 3.10 is used.

[carlotardivo]

Hi, is there any update on this? Is there any plan to attack this issue in the nearby future? This unexpected behavior is becoming quite blocking for our organization.

[ndevenish]

What feels like another very similar reproduction; Package napari sets python_requires = >=3.8 in their setup.cfg, even though it isn't compatible with python 3.11; so I want the prerelease if using python 3.11.

[tool.poetry]
name = "test_resolve"
version = "0.1.0"
description = ""
authors = ["Some Person <some@example.com>"]

[tool.poetry.dependencies]
python = "^3.10"
napari = [
    {version = "^0.4.18rc2", python = "^3.11"},
    {version = "^0.4.17", python = "<3.11"},
]

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

however, a poetry env use python3.10 && poetry install still installs 0.4.18rc2 on python 3.10. (on Poetry 1.5.1). This seems like exactly the scenario described by https://python-poetry.org/docs/dependency-specification/#multiple-constraints-dependencies.

[dimbleby]

the constraint ^0.4.17 allows version 0.4.18rc2, if you don't want that then write a tighter constraint

[ndevenish]

Ah! That works! But shouldn't that be filtered out by the fact that I didn't allow prereleases?

[ndevenish]

There is definitely some weirdness going on. It looks like selecting the prerelease explicitly for one of the restricted dependencies implicitly allows resolution of prereleases for all of the others, e.g.

napari = [
    {version = "^0.4.17", python = "^3.11"},
    {version = "^0.4.17", python = "<3.11"},
]

doesn't match 0.4.18rc2 on either python 3.10 or 3.11, but changing the 3.11 specifier to explicit prerelease does on both.

Perhaps this implicit allow-prerelease was intended - I couldn't find it mentioned in the documentation.

Even attempting to force no-prereleases failed (resolved the rc2 on 3.10)

napari = [
    {version = "^0.4.18rc2", python = "^3.11"},
    {version = "^0.4.17", allow-prereleases = false, python = "<3.11"},
]

and FWIW the only combination I found to get what I wanted was:

napari = [
    {version = "^0.4.18rc2", python = "^3.11"},
    {version = "<0.4.18a0", python = "<3.11"},
]