search

jazzband / pip-tools

A set of tools to keep your pinned Python dependencies fresh.
https://pip-tools.rtfd.io
BSD 3-Clause "New" or "Revised" License
7.69k stars 610 forks source link

pip-compile fails with conflict for multiple references to the same direct reference of a package #1978

Closed manuel-koch closed 9 months ago

manuel-koch commented 1 year ago

Environment Versions

  1. MacOs Ventura
  2. Python version: $ python -V: Python 3.10.6
  3. pip version: $ pip --version: pip 23.2.1 from /python/site-packages/pip (python 3.10)
  4. pip-tools version: $ pip-compile --version: pip-compile, version 7.3.0

Preface

The dependencies as modeled in my program and the depending libs are as follows

i.e. pyproject.toml of my-lib-b looks like

[build-system]
requires = ["hatchling", "hatch-vcs"]
build-backend = "hatchling.build"

[project]
....
dependencies = [
  ....
  "my-lib-a @ git+ssh://git@bitbucket.org/my-lib-a.git@v4.1.1",
]
dynamic = ["version"]

[tool.hatch.version]
source = "vcs"

[tool.hatch.metadata]
allow-direct-references = true

[tool.hatch.build.hooks.vcs]
version-file = "my_lib_b/_version.py"

i.e. requirements.in of my program looks like

....
-e git+ssh://git@bitbucket.org/my-lib-a.git@v4.1.1#egg=my-lib-a
-e git+ssh://git@bitbucket.org/my-lib-b.git@My_Branch#egg=my-lib-b
....

Error when running pip-compile

Running pip-compile

$ pip-compile --verbose --resolver=backtracking /project/requirements.in

yields the following error

  ERROR: Cannot install -r /project/requirements.in (line 36) and my-lib-a 4.1.1 (from git+ssh://****@bitbucket.org/my-lib-a.git@v4.1.1#egg=my-lib-a) because these package versions have conflicting dependencies.
  The conflict is caused by:
      The user requested my-lib-a 4.1.1 (from git+ssh://****@bitbucket.org/my-lib-a.git@v4.1.1#egg=my-lib-a)
      my-lib-b 0.0.2.dev1+g100972e depends on my-lib-a 4.1.1 (from git+ssh://****@bitbucket.org/my-lib-a.git@v4.1.1)

I would expect that requiring the same git commit ( and hence library version is the same ) in a program and a nested dependency is Ok.

Is this a general problem, i.e. it is not supported to have direct references of the same package in a program and a nested dependency ? Is this a bug in pip-compile ?

Full console output looks like

Using pip-tools configuration defaults found in 'pyproject.toml'.
Using indexes:
  https://pypi.org/simple

                          ROUND 1                           
  Obtaining my-lib-a from git+ssh://****@bitbucket.org/my-lib-a.git@v4.1.1#egg=my-lib-a (from -r /project/requirements.in (line 35))
    Cloning ssh://****@bitbucket.org/my-lib-a.git (to revision v4.1.1) to /python/src/my-lib-a
    Running command git clone --filter=blob:none --quiet 'ssh://****@bitbucket.org/my-lib-a.git' /python/src/my-lib-a
    warning: filtering not recognized by server, ignoring
    warning: filtering not recognized by server, ignoring
    Running command git checkout -q 011d2303542909412da0e88a03dc3ddf9e5b9f3d
    Resolved ssh://****@bitbucket.org/my-lib-a.git to commit 011d2303542909412da0e88a03dc3ddf9e5b9f3d
    Installing build dependencies ... done
    Checking if build backend supports build_editable ... done
    Getting requirements to build wheel ... done
    Preparing metadata (pyproject.toml) ... done
  Obtaining my-lib-b from git+ssh://****@bitbucket.org/my-lib-b.git@My_Branch#egg=my-lib-b (from -r /project/requirements.in (line 36))
    Cloning ssh://****@bitbucket.org/my-lib-b.git (to revision My_Branch) to /python
    Running command git clone --filter=blob:none --quiet 'ssh://****@bitbucket.org/my-lib-b.git' /python
    warning: filtering not recognized by server, ignoring
    warning: filtering not recognized by server, ignoring
    Running command git checkout -b My_Branch --track origin/My_Branch
    Switched to a new branch 'My_Branch'
    branch 'My_Branch' set up to track 'origin/My_Branch'.
    Resolved ssh://****@bitbucket.org/my-lib-b.git to commit 100972e7b82a77989558a171b7ebf272d3b9932b
    Installing build dependencies ... done
    Checking if build backend supports build_editable ... done
    Getting requirements to build wheel ... done
    Preparing metadata (pyproject.toml) ... done
  ....
  <A LOT OF OTHER DEPENDENCIES>
  ....
  Collecting my-lib-a@ git+ssh://git@bitbucket.org/my-lib-a.git@v4.1.1 (from my-lib-b->-r /project/requirements.in (line 36))
    Cloning ssh://****@bitbucket.org/my-lib-a.git (to revision v4.1.1) to /private/var/folders/4l/pq2frs651d1ccv5rtxxmwjww0000gn/T/pip-resolve-m3ids16i/my-lib-a_9c76dffd54ea43919d161ed073d34ae1
    Running command git clone --filter=blob:none --quiet 'ssh://****@bitbucket.org/my-lib-a.git' /private/var/folders/4l/pq2frs651d1ccv5rtxxmwjww0000gn/T/pip-resolve-m3ids16i/my-lib-a_9c76dffd54ea43919d161ed073d34ae1
    warning: filtering not recognized by server, ignoring
    warning: filtering not recognized by server, ignoring
    Running command git checkout -q 011d2303542909412da0e88a03dc3ddf9e5b9f3d
    Resolved ssh://****@bitbucket.org/my-lib-a.git to commit 011d2303542909412da0e88a03dc3ddf9e5b9f3d
    Installing build dependencies ... done
    Getting requirements to build wheel ... done
    Preparing metadata (pyproject.toml) ... done
  Collecting nptyping<3,>=2 (from my-lib-b->-r /project/requirements.in (line 36))
    Using cached nptyping-2.5.0-py3-none-any.whl (37 kB)
  INFO: pip is looking at multiple versions of my-lib-b to determine which version is compatible with other requirements. This could take a while.
  ERROR: Cannot install -r /project/requirements.in (line 36) and my-lib-a 4.1.1 (from git+ssh://****@bitbucket.org/my-lib-a.git@v4.1.1#egg=my-lib-a) because these package versions have conflicting dependencies.

  The conflict is caused by:
      The user requested my-lib-a 4.1.1 (from git+ssh://****@bitbucket.org/my-lib-a.git@v4.1.1#egg=my-lib-a)
      my-lib-b 0.0.2.dev1+g100972e depends on my-lib-a 4.1.1 (from git+ssh://****@bitbucket.org/my-lib-a.git@v4.1.1)

  To fix this you could try to:
  1. loosen the range of package versions you've specified
  2. remove package versions to allow pip attempt to solve the dependency conflict

Traceback (most recent call last):
  File "/python/site-packages/pip/_vendor/resolvelib/resolvers.py", line 316, in _backjump
    name, candidate = broken_state.mapping.popitem()
KeyError: 'dictionary is empty'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/python/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
    result = self._result = resolver.resolve(
  File "/python/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/python/site-packages/pip/_vendor/resolvelib/resolvers.py", line 434, in resolve
    success = self._backjump(causes)
  File "/python/site-packages/pip/_vendor/resolvelib/resolvers.py", line 318, in _backjump
    raise ResolutionImpossible(causes)
pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=ExplicitRequirement(EditableCandidate('git+ssh://****@bitbucket.org/my-lib-a.git@v4.1.1#egg=my-lib-a')), parent=None), RequirementInformation(requirement=ExplicitRequirement(LinkCandidate('git+ssh://****@bitbucket.org/my-lib-a.git@v4.1.1')), parent=EditableCandidate('git+ssh://****@bitbucket.org/my-lib-b.git@My_Branch#egg=my-lib-b'))]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/python/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/python/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/python/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/python/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/python/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/python/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/python/site-packages/piptools/scripts/compile.py", line 659, in cli
    results = resolver.resolve(max_rounds=max_rounds)
  File "/python/site-packages/piptools/resolver.py", line 604, in resolve
    is_resolved = self._do_resolve(
  File "/python/site-packages/piptools/resolver.py", line 636, in _do_resolve
    resolver.resolve(
  File "/python/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 101, in resolve
    raise error from e
pip._internal.exceptions.DistributionNotFound: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
atugushev commented 1 year ago

Could you please try to run pip install -r /project/requirements.in --dry-run?

manuel-koch commented 1 year ago

Running plain pip install on a fresh python environment:

$ pip install -r requirements.in --dry-run

results in more-or-less the same strange error

Obtaining my-lib-a from git+ssh://****@bitbucket.org/my-lib-a.git@v4.1.1#egg=my-lib-a (from -r requirements.in (line 35))
  Cloning ssh://****@bitbucket.org/my-lib-a.git (to revision v4.1.1) to /python/src/my-lib-a
  Running command git clone --filter=blob:none --quiet 'ssh://****@bitbucket.org/my-lib-a.git' /python/src/my-lib-a
  warning: filtering not recognized by server, ignoring
  warning: filtering not recognized by server, ignoring
  Running command git checkout -q 011d2303542909412da0e88a03dc3ddf9e5b9f3d
  Resolved ssh://****@bitbucket.org/my-lib-a.git to commit 011d2303542909412da0e88a03dc3ddf9e5b9f3d
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Obtaining my-lib-b from git+ssh://****@bitbucket.org/my-lib-b.git@My_Branch#egg=my-lib-b (from -r requirements.in (line 36))
  Cloning ssh://****@bitbucket.org/my-lib-b.git (to revision My_Branch) to /python/src/my-lib-b
  Running command git clone --filter=blob:none --quiet 'ssh://****@bitbucket.org/my-lib-b.git' /python/src/my-lib-b
  warning: filtering not recognized by server, ignoring
  warning: filtering not recognized by server, ignoring
  Running command git checkout -b My_Branch --track origin/My_Branch
  Switched to a new branch 'My_Branch'
  branch 'My_Branch' set up to track 'origin/My_Branch'.
  Resolved ssh://****@bitbucket.org/my-lib-b.git to commit 100972e7b82a77989558a171b7ebf272d3b9932b
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
....
<A LOT OF OTHER DEPENDENCIES>
....
Collecting my-lib-a@ git+ssh://git@bitbucket.org/my-lib-a.git@v4.1.1 (from my-lib-b->-r requirements.in (line 36))
  Cloning ssh://****@bitbucket.org/my-lib-a.git (to revision v4.1.1) to /private/var/folders/4l/pq2frs651d1ccv5rtxxmwjww0000gn/T/pip-install-yv5vm01q/my-lib-a_c49dcd3e766147d19847ac0547687e67
  Running command git clone --filter=blob:none --quiet 'ssh://****@bitbucket.org/my-lib-a.git' /private/var/folders/4l/pq2frs651d1ccv5rtxxmwjww0000gn/T/pip-install-yv5vm01q/my-lib-a_c49dcd3e766147d19847ac0547687e67
  warning: filtering not recognized by server, ignoring
  warning: filtering not recognized by server, ignoring
  Running command git checkout -q 011d2303542909412da0e88a03dc3ddf9e5b9f3d
  Resolved ssh://****@bitbucket.org/my-lib-a.git to commit 011d2303542909412da0e88a03dc3ddf9e5b9f3d
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting nptyping<3,>=2 (from my-lib-b->-r requirements.in (line 36))
  Using cached nptyping-2.5.0-py3-none-any.whl (37 kB)
INFO: pip is looking at multiple versions of my-lib-b to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install -r requirements.in (line 36) and my-lib-a 4.1.1 (from git+ssh://****@bitbucket.org/my-lib-a.git@v4.1.1#egg=my-lib-a) because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested my-lib-a 4.1.1 (from git+ssh://****@bitbucket.org/my-lib-a.git@v4.1.1#egg=my-lib-a)
    my-lib-b 0.0.2.dev1+g100972e depends on my-lib-a 4.1.1 (from git+ssh://****@bitbucket.org/my-lib-a.git@v4.1.1)

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
manuel-koch commented 1 year ago

Is pip maybe confused because of the two slightly different URLs: w/ or w/o the "egg" postfix ?

atugushev commented 1 year ago

@manuel-koch, thank you! This error originates from pip, so you might consider raising this issue in their tracker. I would also recommend searching through their tracker; you may find some related issues. Nonetheless, thanks for the report!

manuel-koch commented 1 year ago

Follow up from the child ticket I've created for pip.

As mentioned here my issue could be solved by not using the "-e" pip-option in my requirements.in but instead use the "normal" way of referring package via VCS URL:

....
my-lib-a @ git+ssh://git@bitbucket.org/my-lib-a.git@v4.1.1
my-lib-b @ git+ssh://git@bitbucket.org/my-lib-b.git@My_Branch
....
webknjaz commented 9 months ago

I supposed this can be closed now.