Add optional dependencies

[fabcor-maxiv]

GitHub: fixes https://github.com/mxcube/mxcubecore/issues/885

[fabcor-maxiv]

This is kind of an experiment. I do not know if it will go well or not, but I believe it is worth trying a bit at least.

So far I managed to add quite a bit of optional dependencies. Some are still missing.

[fabcor-maxiv]

Adding bliss is problematic, Poetry can not solve the dependencies. I will see if I can find a way to help Poetry with that.

[fabcor-maxiv]

With the following change:

diff --git a/pyproject.toml b/pyproject.toml
index 7e21f97f..c1d5fd04 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -26,7 +26,7 @@ classifiers = [
 [tool.poetry.dependencies]
 python = ">=3.8,<3.11"

-#bliss = { version = "*", optional = true , python = "<3.10" }
+bliss = { version = "*", optional = true , python = "<3.10" }
 devtools = { version = "^0.12.2", optional = true }
 pyepics = {version = "^3.5.2", optional = true}
 pymba = {version = "^0.3.7", optional = true}
@@ -80,7 +80,7 @@ furo = "^2023.9.10"
 # tine <https://tine.desy.de/>
 # v4l2

-#bliss = ["bliss"]
+bliss = ["bliss"]
 epics = ["pyepics"]
 tango = ["pytango", "sardana"

Poetry (under both Python 3.8 and 3.9) ends up failing on trying to install fisx.

$ poetry install --all-extras
Installing dependencies from lock file

Package operations: 56 installs, 6 updates, 0 removals

  - Updating pycparser (2.21 /home/conda/feedstock_root/build_artifacts/pycparser_1636257122734/work -> 2.21)
  - Downgrading cffi (1.16.0 /home/conda/feedstock_root/build_artifacts/cffi_1696001742886/work -> 1.15.1)
  - Updating cryptography (42.0.5 /home/conda/feedstock_root/build_artifacts/cryptography-split_1708780255643/work -> 42.0.5)
  - Installing h5py (3.9.0)
  - Downgrading crashtest (0.4.1 /home/conda/feedstock_root/build_artifacts/crashtest_1667467006729/work -> 0.3.1)
  - Installing hdf5plugin (4.3.0)
  - Installing pylev (1.4.0)
  - Installing types-pyopenssl (24.0.0.20240311)
  - Installing aioredis (2.0.1)
  - Downgrading cleo (2.1.0 /home/conda/feedstock_root/build_artifacts/cleo_1698693584261/work -> 1.0.0a4)
  - Installing fabio (2023.10.0)
  - Installing hiredis (2.3.2)
  - Installing pptree (3.1)
  - Installing python-ulid (1.1.0)
  - Installing types-redis (4.6.0.20240311)
  - Installing types-six (1.16.21.20240311)
  - Installing beartype (0.16.4)
  - Installing fisx (1.3.1): Failed

  OverrideNeeded

  ({Package('oldest-supported-numpy', '2023.12.21'): {'numpy': <Dependency numpy (==1.22.2)>}}, {Package('oldest-supported-numpy', '2023.12.21'): {'numpy': <Dependency numpy (==1.21.0)>}}, {Package('oldest-supported-numpy', '2023.12.21'): {'numpy': <Dependency numpy (==1.19.3)>}}, {Package('oldest-supported-numpy', '2023.12.21'): {'numpy': <Dependency numpy (==1.25.0)>}}, {Package('oldest-supported-numpy', '2023.12.21'): {'numpy': <Dependency numpy (==1.23.3)>}}, {Package('oldest-supported-numpy', '2023.12.21'): {'numpy': <Dependency numpy (<empty>)>}})

  at ~/.local/share/mamba/envs/mxcubecore/lib/python3.9/site-packages/poetry/puzzle/provider.py:653 in complete_package
      649│                     current_overrides.update({package: package_overrides})
      650│                     overrides.append(current_overrides)
      651│ 
      652│             if overrides:
    → 653│                 raise OverrideNeeded(*overrides)
      654│ 
      655│         # Modifying dependencies as needed
      656│         clean_dependencies = []
      657│         for dep in dependencies:

The following error occurred when trying to handle this error:

  OverrideNeeded

  {Package('oldest-supported-numpy', '2023.12.21'): {'numpy': <Dependency numpy (==1.22.2)>}, Package('oldest-supported-numpy', '2023.12.12'): {'numpy': <Dependency numpy (==1.22.2)>}}

  at ~/.local/share/mamba/envs/mxcubecore/lib/python3.9/site-packages/poetry/puzzle/provider.py:653 in complete_package
      649│                     current_overrides.update({package: package_overrides})
      650│                     overrides.append(current_overrides)
      651│ 
      652│             if overrides:
    → 653│                 raise OverrideNeeded(*overrides)
      654│ 
      655│         # Modifying dependencies as needed
      656│         clean_dependencies = []
      657│         for dep in dependencies:

The following error occurred when trying to handle this error:

  OverrideNeeded

  {Package('oldest-supported-numpy', '2023.12.21'): {'numpy': <Dependency numpy (==1.22.2)>}, Package('oldest-supported-numpy', '2023.12.12'): {'numpy': <Dependency numpy (==1.22.2)>}, Package('oldest-supported-numpy', '2023.10.25'): {'numpy': <Dependency numpy (==1.22.2)>}}

  at ~/.local/share/mamba/envs/mxcubecore/lib/python3.9/site-packages/poetry/puzzle/provider.py:653 in complete_package
      649│                     current_overrides.update({package: package_overrides})
      650│                     overrides.append(current_overrides)
      651│ 
      652│             if overrides:
    → 653│                 raise OverrideNeeded(*overrides)
      654│ 
      655│         # Modifying dependencies as needed
      656│         clean_dependencies = []
      657│         for dep in dependencies:

The following error occurred when trying to handle this error:

  OverrideNeeded

  {Package('oldest-supported-numpy', '2023.12.21'): {'numpy': <Dependency numpy (==1.22.2)>}, Package('oldest-supported-numpy', '2023.12.12'): {'numpy': <Dependency numpy (==1.22.2)>}, Package('oldest-supported-numpy', '2023.10.25'): {'numpy': <Dependency numpy (==1.22.2)>}, Package('oldest-supported-numpy', '2023.8.3'): {'numpy': <Dependency numpy (==1.22.2)>}}

  at ~/.local/share/mamba/envs/mxcubecore/lib/python3.9/site-packages/poetry/puzzle/provider.py:653 in complete_package
      649│                     current_overrides.update({package: package_overrides})
      650│                     overrides.append(current_overrides)
      651│ 
      652│             if overrides:
    → 653│                 raise OverrideNeeded(*overrides)
      654│ 
      655│         # Modifying dependencies as needed
      656│         clean_dependencies = []
      657│         for dep in dependencies:

The following error occurred when trying to handle this error:

  IncompatibleConstraintsError

  Incompatible constraints in requirements of oldest-supported-numpy (2022.11.19):
  numpy (==1.22.2) ; platform_machine == "loongarch64" and python_version < "3.11"
  numpy (==1.23.3) ; python_version == "3.9" and platform_system == "OS400"
  numpy ; python_version >= "3.9" and platform_python_implementation == "PyPy"

  at ~/.local/share/mamba/envs/mxcubecore/lib/python3.9/site-packages/poetry/puzzle/provider.py:925 in _resolve_overlapping_markers
      921│                     specific_source_dependency = dep
      922│                 constraint = constraint.intersect(dep.constraint)
      923│             if constraint.is_empty():
      924│                 # conflict in overlapping area
    → 925│                 raise IncompatibleConstraintsError(package, *used_dependencies)
      926│ 
      927│             if not any(uses):
      928│                 # This is an edge case where the dependency is not required
      929│                 # for the resulting marker. However, we have to consider it anyway

Cannot install fisx.

  - Updating msgpack (1.0.7 /home/conda/feedstock_root/build_artifacts/msgpack-python_1700926506810/work -> 1.0.8)
  - Installing parso (0.8.3)
  - Installing pytz (2023.3.post1)
  - Installing redis-om (0.0.27)
  - Installing silx (1.1.2)
  - Installing stompest (2.3.0)
  - Installing wcwidth (0.2.13)

[marcus-oscarsson]

Should we keep this one open ?

[fabcor-maxiv]

Should we keep this one open ?

Depends if it is a worthwhile task or not...

I think it would be good to have all dependencies listed. I do not know how others handle deployment if not all dependencies are listed. How do facilities know what needs to be installed? Where do they keep their list of dependencies?

This is something I am interested in. This is something we discussed a bit in our breakout room at the code camp, but did not manage to get to the bottom of things. Maybe the questionnaire we worked on, could help shed some light on this. But on the other hand, if everyone is happy with things as they are, then maybe I can focus on MAX IV dependencies only and ignore the rest.