search

fractal-napari-plugins-collection / napari-ome-zarr-navigator

BSD 3-Clause "New" or "Revised" License
6 stars 0 forks source link

Make napari-ome-zarr-navigator work in pixi projects #29

Open jluethi opened 1 month ago

jluethi commented 1 month ago

Currently, I can't install napari-ome-zarr-navigator in the same pixi project as napari due to conflicts in sub-dependencies of each.

pixi init napari-fractal
cd napari-fractal 
pixi add python==3.10
pixi add --pypi napari-ome-zarr-navigator
✔ Added napari-ome-zarr-navigator >=0.1.2, <0.2
pixi add napari
  × failed to solve the pypi requirements of 'default' 'osx-arm64'
  ├─▶ failed to resolve pypi dependencies
  ╰─▶ Because fractal-tasks-core==1.3.2 depends on pydantic>2,<=2.8.2 and pydantic==2.9.2, we can conclude that fractal-tasks-
      core==1.3.2 cannot be used.
      And because napari-ome-zarr-navigator==0.1.2 depends on fractal-tasks-core==1.3.2, we can conclude that napari-ome-zarr-
      navigator==0.1.2 cannot be used.
      And because only napari-ome-zarr-navigator<=0.1.2 is available and you require napari-ome-zarr-navigator>=0.1.2, we can
      conclude that your requirements are unsatisfiable.

This is after fixing the numpy dependency issues: https://github.com/fractal-napari-plugins-collection/napari-ome-zarr-navigator/pull/27

napari-ome-zarr-navigator depends on fractal-tasks-core for NGFF library functionality. Fracal-tasks-core depends on pydantic <=2.8.2 When adding napari, something appears to find a conflict within fractal-tasks-core itself. It's not that napari can't coexist with pydantic==2.8.0. But one of the shared dependencies is apparently creating issues here. And some dependency that we don't pin in fractal-tasks-core appears to depend on pydantic==2.9.2 exactly. While this seems like over-aggressive dependency pinning here, it makes the installation a big pain.

In practice, none of what napari-ome-zarr-navigator uses from fractal-tasks-core actually depends on pydantic deep enough where this would matter, because installing it e.g. in conda environments where those dependencies aren't checked doesn't lead to any issues. But we 1) want to work with pixi & 2) want to avoid dependency issues in general.

Potential ways to solve this:

  1. Switch to using ngio instead of fractal-tasks-core as a dependency in napari-ome-zarr-navigator: https://github.com/fractal-analytics-platform/ngio
  2. Relax pydantic constraints in fractal-tasks-core (see discussion here: https://github.com/fractal-analytics-platform/fractal-tasks-core/issues/835)
  3. Figure out which dependency leads to issue and pin that one

Orders of effort: If there's a simple way to track part 3 down, that would be the easiest. I'm currently failing to find it. Maybe @lorenzocerrone would have an idea?

If not, doing the refactor to switch to ngio is anyway useful, but some work. Changing fractal-tasks-core dependencies for pydantic would be the hardest for sure

tcompa commented 4 days ago

I can reproduce this

For the record, this still happens right now - but the current error mentions the very latest pydantic (2.10.1). It is a bit weird that some dependency really pins the latest version of pydantic, I'm wondering if this is actually the case or it's something somehow introduced by pixi itself.

$ pixi init napari-fractal
✔ Created /tmp/pixitest/napari-fractal/pixi.toml

$ cd napari-fractal/

$ pixi add python==3.10
✔ Added python==3.10

$ pixi add --pypi napari-ome-zarr-navigator
✔ Added napari-ome-zarr-navigator >=0.1.2, <0.2

Added these as pypi-dependencies.
$ pixi add napari
  × failed to solve the pypi requirements of 'default' 'linux-64'
  ├─▶ failed to resolve pypi dependencies
  ╰─▶ Because fractal-tasks-core==1.3.2 depends on pydantic>2,<=2.8.2 and pydantic==2.10.1, we can conclude that fractal-tasks-core==1.3.2 cannot be used.
      And because napari-ome-zarr-navigator==0.1.2 depends on fractal-tasks-core==1.3.2, we can conclude that napari-ome-zarr-navigator==0.1.2 cannot be used.
      And because only napari-ome-zarr-navigator<=0.1.2 is available and you require napari-ome-zarr-navigator>=0.1.2, we can conclude that your requirements are unsatisfiable.

It seems like it has to do with pixi, not with pydantic

I don't think this has to do with pydantic, which seems clear from the next example (after a fresh pixi init):

$ pixi add python==3.10
✔ Added python==3.10

$ pixi add pydantic==2.8.2
✔ Added pydantic==2.8.2

$ pixi add --pypi napari-ome-zarr-navigator
✔ Added napari-ome-zarr-navigator >=0.1.2, <0.2
Added these as pypi-dependencies.

$ pixi add napari
  × failed to solve the pypi requirements of 'default' 'linux-64'
  ├─▶ failed to resolve pypi dependencies
  ╰─▶ Because fractal-tasks-core==1.3.2 depends on docstring-parser>=0.15,<0.16 and docstring-parser==0.16, we can conclude that fractal-tasks-core==1.3.2 cannot be used.
      And because napari-ome-zarr-navigator==0.1.2 depends on fractal-tasks-core==1.3.2, we can conclude that napari-ome-zarr-navigator==0.1.2 cannot be used.
      And because only napari-ome-zarr-navigator<=0.1.2 is available and you require napari-ome-zarr-navigator>=0.1.2, we can conclude that your requirements are unsatisfiable.

I don't know much about pixi, but it seems like:

Possible solution: use a single add

Is this a valid solution?

$ pixi add python==3.10 
✔ Added python==3.10

$ pixi add napari --pypi napari-ome-zarr-navigator
✔ Added napari >=0.5.4, <0.6
✔ Added napari-ome-zarr-navigator >=0.1.2, <0.2
Added these as pypi-dependencies.

The difference with respect with the original one is that napari is now taken from pypi, and not from conda. If this is OK, then I guess it could be a reasonable workaround (unless we want to understand better how pixi works).

Note that the resulting pixi.toml file looks like

[project]
authors = ["..."]
channels = ["conda-forge"]
description = "Add a short description here"
name = "napari-fractal"
platforms = ["linux-64"]
version = "0.1.0"

[tasks]

[dependencies]
python = "==3.10"

[pypi-dependencies]
napari = ">=0.5.4, <0.6"
napari-ome-zarr-navigator = ">=0.1.2, <0.2"

and the napari sources are all from pypi:

$ grep napari pixi.lock  | grep http

      - pypi: https://files.pythonhosted.org/packages/7e/c8/6bcaf27758397c8f1da906113bc99a9a4bbbc062835c2bff4fb2c3406753/napari-0.5.4-py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/c4/f4/7714f4c5a7c01f71312208fbbb0abd1b887d2119b4010747cf6319fd8c76/napari_console-0.1.1-py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/0d/81/8088d4d695ff7b93af4b1224448ec3270990c37d4e77bdb454c0078384e7/napari_ome_zarr-0.6.1-py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/51/20/68139041c42833feca4b1ad025bdb82609815f3a6f84ce73c980fa0665b5/napari_ome_zarr_navigator-0.1.2-py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/c9/60/eaf45290008cfb7cc1e99f6427540704fafc5c65a3a25ff0024085cd2e0d/napari_plugin_engine-0.2.0-py3-none-any.whl
      - pypi: https://files.pythonhosted.org/packages/02/e7/8cf145b19dda544a4a4eebfb4461313aa933cf6874532b7ae696fab65a90/napari_svg-0.2.0-py3-none-any.whl
  url: https://files.pythonhosted.org/packages/7e/c8/6bcaf27758397c8f1da906113bc99a9a4bbbc062835c2bff4fb2c3406753/napari-0.5.4-py3-none-any.whl
  url: https://files.pythonhosted.org/packages/c4/f4/7714f4c5a7c01f71312208fbbb0abd1b887d2119b4010747cf6319fd8c76/napari_console-0.1.1-py3-none-any.whl
  url: https://files.pythonhosted.org/packages/0d/81/8088d4d695ff7b93af4b1224448ec3270990c37d4e77bdb454c0078384e7/napari_ome_zarr-0.6.1-py3-none-any.whl
  url: https://files.pythonhosted.org/packages/51/20/68139041c42833feca4b1ad025bdb82609815f3a6f84ce73c980fa0665b5/napari_ome_zarr_navigator-0.1.2-py3-none-any.whl
  url: https://files.pythonhosted.org/packages/c9/60/eaf45290008cfb7cc1e99f6427540704fafc5c65a3a25ff0024085cd2e0d/napari_plugin_engine-0.2.0-py3-none-any.whl
  url: https://files.pythonhosted.org/packages/02/e7/8cf145b19dda544a4a4eebfb4461313aa933cf6874532b7ae696fab65a90/napari_svg-0.2.0-py3-none-any.whl
jluethi commented 4 days ago

Thanks a lot for digging into this @tcompa !

This now works for me to run this napari plugins within pixi. When napari is installed from pypi, we need to manually add pyqt as well. That's one of the conda benefits I think.

Thus, the current way to make this work is:

pixi init napari-fractal
cd napari-fractal
pixi add python==3.10
pixi add napari --pypi napari-ome-zarr-navigator
pixi add pyqt
pixi run napari

Which leads to the following pixi.toml:

[project]
authors = ["jluethi <joel.luethi@uzh.ch>"]
channels = ["conda-forge"]
description = "Add a short description here"
name = "napari-fractal"
platforms = ["osx-arm64"]
version = "0.1.0"

[tasks]

[dependencies]
python = "==3.10"
pyqt = ">=5.15.9,<6"

[pypi-dependencies]
napari = ">=0.5.4, <0.6"
napari-ome-zarr-navigator = ">=0.1.2, <0.2"

Very weird that pixi behaves in that way with pydantic dependencies. Any idea what are we doing wrong here @lorenzocerrone & @imagejan ?

imagejan commented 3 days ago

Just a few comments:

$ pixi add napari --pypi napari-ome-zarr-navigator

.. is the same as:

$ pixi add --pypi napari napari-ome-zarr-navigator

.. so both will be installed via PyPI (i.e. the --pypi flag affects the whole command).

Pixi solves the conda dependencies first, and then can run into incompatibilities if a PyPI package has different version requirements of a dependency that was already solved in the conda step. (They apply some mapping of pypi->conda packages to solve the transient dependencies via conda if possible.)

Possible explanation

When you pixi add napari (via conda), it will have the latest docstring_parser==0.16 as a dependency (via magicgui). Trying to install napari-ome-zarr-navigator will fail because it depends on docstring-parser=^0.15 (via fractal-tasks-core).

Alternative solution

Pinning to docstring_parser<0.16 in the conda dependencies helps:

pixi init
pixi add python==3.10 pydantic==2.8.2 docstring_parser<0.16 napari pyqt
pixi add --pypi napari-ome-zarr-navigator

Note: I noticed that fractal-tasks-core on conda-forge is at version 1.0.1 (see https://anaconda.org/conda-forge/fractal-tasks-core/files). Any plans to change that? I guess it would also simplify the issue here.

tcompa commented 3 days ago

Note: I noticed that fractal-tasks-core on conda-forge is at version 1.0.1 (see https://anaconda.org/conda-forge/fractal-tasks-core/files). Any plans to change that? I guess it would also simplify the issue here.

This is unfortunately blocked, due to https://github.com/fractal-analytics-platform/fractal-tasks-core/issues/611. Support for pip-like extras on conda is very limited (or at least that was my understanding a while ago when I looked it up - but I'd be happy to learn that I missed something..). If this is the case, then publishing on conda-forge becomes less relevant (because all tasks of that package require a fractal-tasks extra being set).

imagejan commented 3 days ago

Thanks @tcompa, I commented on the linked issue.

Related question: I can see why fractal tasks can optionally depend on cellpose or scikit-image to get things done. But why is there a dependency on stackview? Aren't tasks supposed to run headlessly?

tcompa commented 3 days ago

Related question: I can see why fractal tasks can optionally depend on cellpose or scikit-image to get things done. But why is there a dependency on stackview? Aren't tasks supposed to run headlessly?

We already indirectly dependede on stackview, due to napari-workflows, but we also included it as a direct dependency so that we could constrain its version, since version 0.9.0 was broken.

Note that subsequent patch releases did fix the issue (see https://github.com/haesleinhuepf/napari-workflows/issues/47), but we eventually ended up keeping the version constraint stackview<0.9.0 in place as a conservative choice.

imagejan commented 3 days ago

We already indirectly dependede on stackview, due to napari-workflows, but we also included it as a direct dependency so that we could constrain its version, since version 0.9.0 was broken.

Alright, thanks for the insight. So this is actually an issue of napari-workflows that depends on stackview although it explicitly states that "it serves as backend"... 😕

jluethi commented 3 days ago

Thanks a lot for the further explanations @imagejan ! Re stackview dependency: The mid-term goal is to move the napari workflows wrapper into its own task package to avoid having to deal with such dependency version issues in tasks-core.

And longer term, library functionality of fractal-tasks-core should either be available in ngio (all OME-Zarr related operations) or in a to-be-extracted package that handles manifest creation etc. => it will become less likely that people depend on tasks-core, but either on ngio or the task building package