search

ddelange / pipgrip

Lightweight pip dependency resolver with deptree preview functionality based on the PubGrub algorithm
Other
180 stars 14 forks source link

Error when package requires itself #95

Closed jdchn closed 1 year ago

jdchn commented 2 years ago

What you were trying to do (and why)

Analyze the dependencies for a requirements.txt file containing jax.

The error can be isolated to the jax dependency etils[epath].

What happened (including command output)

Command output

```console $ pipgrip -vv --tree jax INFO: discovering jax INFO: fact: _root_ is root INFO: derived: root INFO: fact: root depends on jax (*) INFO: selecting _root_ (0.0.0) INFO: derived: jax (*) INFO: fact: jax (0.3.16) depends on absl-py (*) INFO: fact: jax (0.3.16) depends on etils[epath] (*) INFO: fact: jax (0.3.16) depends on numpy (>=1.20) INFO: fact: jax (0.3.16) depends on opt-einsum (*) INFO: fact: jax (0.3.16) depends on scipy (>=1.5) INFO: fact: jax (0.3.16) depends on typing-extensions (*) INFO: selecting jax (0.3.16) INFO: derived: typing-extensions (*) INFO: derived: scipy (>=1.5) INFO: derived: opt-einsum (*) INFO: derived: numpy (>=1.20) INFO: derived: etils[epath] (*) INFO: derived: absl-py (*) INFO: discovering typing-extensions INFO: discovering scipy>=1.5 INFO: discovering opt-einsum INFO: discovering numpy>=1.20 INFO: discovering etils[epath] INFO: discovering absl-py Traceback (most recent call last): File "/home/vagrant/.local/bin/pipgrip", line 8, in sys.exit(main()) File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/cli.py", line 434, in main solution = solver.solve() File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/libs/mixology/version_solver.py", line 69, in solve if not self._run(): File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/libs/mixology/version_solver.py", line 85, in _run next_package = self._choose_package_version() File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/libs/mixology/version_solver.py", line 363, in _choose_package_version for incompatibility in self._source.incompatibilities_for( File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/libs/mixology/package_source.py", line 112, in incompatibilities_for incompatibility = Incompatibility( File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/libs/mixology/incompatibility.py", line 61, in __init__ assert by_ref[ref] is not None AssertionError ``` ```console $ pipgrip -vvv --tree etils[epath] DEBUG: environment: {'implementation_name': 'cpython', 'implementation_version': '3.8.10', 'os_name': 'posix', 'platform_machine': 'x86_64', 'platform_release': '5.15.0-41-generic', 'platform_system': 'Linux', 'platform_version': '#44~20.04.1-Ubuntu SMP Fri Jun 24 13:27:29 UTC 2022', 'python_full_version': '3.8.10', 'platform_python_implementation': 'CPython', 'python_version': '3.8', 'sys_platform': 'linux'} DEBUG: pip version: [20, 0, 2] DEBUG: pipgrip version: 0.8.5 INFO: discovering etils[epath] DEBUG: Downloading/building wheel for etils[epath] into cache_dir /home/vagrant/.cache/pip/wheels/pipgrip DEBUG: ['/usr/bin/python3', '-m', 'pip', 'wheel', '--no-deps', '--disable-pip-version-check', '--wheel-dir', '/home/vagrant/.cache/pip/wheels/pipgrip', '--progress-bar=off', 'etils[epath]'] DEBUG: {'etils[epath]': '/home/vagrant/.cache/pip/wheels/pipgrip/etils-0.7.1-py3-none-any.whl'} DEBUG: Searching metadata in /home/vagrant/.cache/pip/wheels/pipgrip/etils-0.7.1-py3-none-any.whl DEBUG: dropped conditional dep etils[array-types] ; extra == "all" DEBUG: dropped conditional dep etils[ecolab] ; extra == "all" DEBUG: dropped conditional dep etils[edc] ; extra == "all" DEBUG: dropped conditional dep etils[enp] ; extra == "all" DEBUG: dropped conditional dep etils[epath] ; extra == "all" DEBUG: dropped conditional dep etils[epy] ; extra == "all" DEBUG: dropped conditional dep etils[etqdm] ; extra == "all" DEBUG: dropped conditional dep etils[etree] ; extra == "all" DEBUG: dropped conditional dep etils[etree-dm] ; extra == "all" DEBUG: dropped conditional dep etils[etree-jax] ; extra == "all" DEBUG: dropped conditional dep etils[etree-tf] ; extra == "all" DEBUG: dropped conditional dep etils[enp] ; extra == "array-types" DEBUG: dropped conditional dep pytest ; extra == "dev" DEBUG: dropped conditional dep pytest-subtests ; extra == "dev" DEBUG: dropped conditional dep pytest-xdist ; extra == "dev" DEBUG: dropped conditional dep pylint>=2.6.0 ; extra == "dev" DEBUG: dropped conditional dep yapf ; extra == "dev" DEBUG: dropped conditional dep chex ; extra == "dev" DEBUG: dropped conditional dep jupyter ; extra == "ecolab" DEBUG: dropped conditional dep numpy ; extra == "ecolab" DEBUG: dropped conditional dep mediapy ; extra == "ecolab" DEBUG: dropped conditional dep etils[enp] ; extra == "ecolab" DEBUG: dropped conditional dep etils[epy] ; extra == "ecolab" DEBUG: dropped conditional dep etils[epy] ; extra == "edc" DEBUG: dropped conditional dep numpy ; extra == "enp" DEBUG: dropped conditional dep etils[epy] ; extra == "enp" DEBUG: included conditional dep importlib_resources ; extra == "epath" DEBUG: included conditional dep typing_extensions ; extra == "epath" DEBUG: included conditional dep zipp ; extra == "epath" DEBUG: included conditional dep etils[epy] ; extra == "epath" DEBUG: dropped conditional dep typing_extensions ; extra == "epy" DEBUG: dropped conditional dep absl-py ; extra == "etqdm" DEBUG: dropped conditional dep tqdm ; extra == "etqdm" DEBUG: dropped conditional dep etils[epy] ; extra == "etqdm" DEBUG: dropped conditional dep etils[array_types] ; extra == "etree" DEBUG: dropped conditional dep etils[epy] ; extra == "etree" DEBUG: dropped conditional dep etils[enp] ; extra == "etree" DEBUG: dropped conditional dep etils[etqdm] ; extra == "etree" DEBUG: dropped conditional dep dm-tree ; extra == "etree-dm" DEBUG: dropped conditional dep etils[etree] ; extra == "etree-dm" DEBUG: dropped conditional dep jax[cpu] ; extra == "etree-jax" DEBUG: dropped conditional dep etils[etree] ; extra == "etree-jax" DEBUG: dropped conditional dep tf-nightly ; extra == "etree-tf" DEBUG: dropped conditional dep etils[etree] ; extra == "etree-tf" DEBUG: Finding possible versions for etils[epath] DEBUG: ['/usr/bin/python3', '-m', 'pip', 'wheel', '--no-deps', '--disable-pip-version-check', '--progress-bar=off', 'etils[epath]==42.42.post424242'] DEBUG: {'etils[epath]': ['0.2.0', '0.3.0', '0.3.1', '0.3.2', '0.3.3', '0.4.0', '0.5.0', '0.5.1', '0.6.0', '0.7.0', '0.7.1']} INFO: fact: _root_ is root INFO: derived: root INFO: fact: root depends on etils[epath] (*) INFO: selecting _root_ (0.0.0) INFO: derived: etils[epath] (*) Traceback (most recent call last): File "/home/vagrant/.local/bin/pipgrip", line 8, in sys.exit(main()) File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/cli.py", line 434, in main solution = solver.solve() File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/libs/mixology/version_solver.py", line 69, in solve if not self._run(): File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/libs/mixology/version_solver.py", line 85, in _run next_package = self._choose_package_version() File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/libs/mixology/version_solver.py", line 363, in _choose_package_version for incompatibility in self._source.incompatibilities_for( File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/libs/mixology/package_source.py", line 112, in incompatibilities_for incompatibility = Incompatibility( File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/libs/mixology/incompatibility.py", line 61, in __init__ assert by_ref[ref] is not None AssertionError ```

What you expected to happen

Expected to see dependency tree for jax.

Step-by-step reproduction instructions

In an Ubuntu 20.04 or RHEL UBI 8 environment (VM or container) with Python 3.8...

  1. Install pipgrip.
  2. Run pipgrip -vv --tree etils[epath].
ddelange commented 2 years ago

Hi @jdchn 👋

First of all, thanks for the reproducible bug report!

This is a hard crash in the vendored and adapted mixology library:

# If we have two terms that refer to the same package but have a null
# intersection, they're mutually exclusive, making this incompatibility
# irrelevant, since we already know that mutually exclusive version
# ranges are incompatible. We should never derive an irrelevant
# incompatibility.
assert by_ref[ref] is not None

From a first glance, this one might be triggered because the package requires itself, something I haven't seen in the wild before (or even knew was allowed).

Let me think how to properly handle this cyclic dependency, it might not be a trivial fix.

ddelange commented 2 years ago

It looks like poetry does not support the pattern, whereas pip happily installs it. I strive for pip compliance, so this needs a fix for sure.

pipgrip currently combines multiple extras alphabetically, e.g. here you could expect to encounter somewhere in pipgrip's output: smart_open[azure,http,gcs,s3]. In the example below that should also be the case, although you'd have to select some other etils extras in order to observe this behaviour.

What etils is doing is next level. It will make for a beautiful test case... I would expect to find etils listed as dependency of itself in the tree (even though it's only a "trick" used by their devs to express the needed dependencies depending on scenario), it would be incorrect to expect pipgrip to "flatten" and resolve this into a list of (only) external dependencies.

Not strictly true:

etils[epath] (0.7.1)
|_ numpy (42)
|_ typing_extensions (42)
...

Expected:

etils[epath] (0.7.1)
|_ etils[enp] (0.7.1)
   |_ numpy (42)
   |_ etils[epy] (0.7.1)
      |_ typing_extensions (42)
...

without warnings of cyclic dependencies. It only gets cyclic if the same extra is encountered down the tree, e.g. if epy extra would ask for epath extra:

etils[epath] (0.7.1)
|_ etils[enp] (0.7.1)
   |_ numpy (42)
   |_ etils[epy] (0.7.1)
      |_ typing_extensions (42)
      |_ etils[epath] (0.7.1, cyclic)
...

This gets iffy, because there might additionally be version specifiers for etils added to this scenario, in which case there is potential for a "etils>=0.7 depends on etils==0.6.0, thats a no-no" exit. That leads me to the conclusion that this needs a fix in the vendored package, as full version solving capabilities need to be retained.

jdchn commented 2 years ago

Let me think how to properly handle this cyclic dependency, it might not be a trivial fix.

I don't feel confident enough in myself to provide any useful comments :|

It does seem like etils is a playing some "interesting" games with packaging to avoid the hassle of multiple projects — See etils#185.

jdchn commented 2 years ago

It may be a little trickier than it seems. It will be tempting to simply treat self-references as normal dependencies, but there should be some caveats:

The above is probably what would be expected, but I don't know what Pip will actually do.

I think there's also another related issue:

Command output

```console Traceback (most recent call last): File "/home/vagrant/.local/bin/pipgrip", line 8, in sys.exit(main()) File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/home/vagrant/.local/lib/python3.8/site-packages/pipgrip/cli.py", line 460, in main raise RuntimeError( RuntimeError: Unexpected partial solution encountered, not all packages have decisions ```

ddelange commented 2 years ago

Thanks for reporting that one too! Separate bug, yet related 🤔

github-actions[bot] commented 1 year ago

Released 0.9.0