`local` and `reana` extras running into PyYAML 5.4.x Cython issues

[matthewfeickert]

python -m pip install --upgrade '.[reana,local]' 'pyyaml>=6.0'

...
ERROR: Cannot install pyyaml>=6.0, reana-commons[snakemake,yadage]==0.8.0, reana-commons[snakemake,yadage]==0.8.1, reana-commons[snakemake,yadage]==0.8.2, reana-commons[snakemake,yadage]==0.8.3, reana-commons[snakemake,yadage]==0.8.4, reana-commons[snakemake,yadage]==0.8.5, recast-atlas, recast-atlas==0.3.0 and recast-atlas[local,reana]==0.3.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested pyyaml>=6.0
    recast-atlas[local,reana] 0.3.0 depends on pyyaml>=5.1
    yadage 0.21.0 depends on pyyaml
    packtivity 0.16.2 depends on pyyaml
    yadage-schemas 0.10.7 depends on pyyaml
    recast-atlas 0.3.0 depends on pyyaml>=5.1
    reana-commons[snakemake,yadage] 0.8.5 depends on PyYAML<6.0 and >=5.1
    The user requested pyyaml>=6.0
    recast-atlas[local,reana] 0.3.0 depends on pyyaml>=5.1
    yadage 0.21.0 depends on pyyaml
    packtivity 0.16.2 depends on pyyaml
    yadage-schemas 0.10.7 depends on pyyaml
    recast-atlas 0.3.0 depends on pyyaml>=5.1
    reana-commons[snakemake,yadage] 0.8.4 depends on PyYAML<6.0 and >=5.1
    The user requested pyyaml>=6.0
    recast-atlas[local,reana] 0.3.0 depends on pyyaml>=5.1
    yadage 0.21.0 depends on pyyaml
    packtivity 0.16.2 depends on pyyaml
    yadage-schemas 0.10.7 depends on pyyaml
    recast-atlas 0.3.0 depends on pyyaml>=5.1
    reana-commons[snakemake,yadage] 0.8.3 depends on PyYAML<6.0 and >=5.1
    The user requested pyyaml>=6.0
    recast-atlas[local,reana] 0.3.0 depends on pyyaml>=5.1
    yadage 0.21.0 depends on pyyaml
    packtivity 0.16.2 depends on pyyaml
    yadage-schemas 0.10.7 depends on pyyaml
    recast-atlas 0.3.0 depends on pyyaml>=5.1
    reana-commons[snakemake,yadage] 0.8.2 depends on PyYAML<6.0 and >=5.1
    The user requested pyyaml>=6.0
    recast-atlas[local,reana] 0.3.0 depends on pyyaml>=5.1
    yadage 0.21.0 depends on pyyaml
    packtivity 0.16.2 depends on pyyaml
    yadage-schemas 0.10.7 depends on pyyaml
    recast-atlas 0.3.0 depends on pyyaml>=5.1
    reana-commons[snakemake,yadage] 0.8.1 depends on PyYAML<6.0 and >=5.1
    The user requested pyyaml>=6.0
    recast-atlas[local,reana] 0.3.0 depends on pyyaml>=5.1
    yadage 0.21.0 depends on pyyaml
    packtivity 0.16.2 depends on pyyaml
    yadage-schemas 0.10.7 depends on pyyaml
    recast-atlas 0.3.0 depends on pyyaml>=5.1
    reana-commons[snakemake,yadage] 0.8.0 depends on PyYAML<6.0 and >=5.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

https://github.com/recast-hep/recast-atlas/blob/b660442ff2654db8d9bf9efe79eff4894eb6bf02/setup.py#L5-L12

https://github.com/recast-hep/recast-atlas/blob/b660442ff2654db8d9bf9efe79eff4894eb6bf02/setup.py#L14

[matthewfeickert]

The issue is that if you try something like

python -m pip install --upgrade 'yadage>=0.21.0' 'reana-commons[snakemake,yadage]>=0.8.0' 'PyYAML>6.0'

then as

The conflict is caused by:
    The user requested yadage>=0.21.0
    reana-commons[snakemake,yadage] 0.9.2.1 depends on yadage~=0.20.1; extra == "yadage"

the constraint that reana-commons v0.9.2.1 applies of yadage~=0.20.1 is incompatible with yadage>=0.21.0 and then backtracing fails.

c.f. https://github.com/reanahub/reana-commons/issues/409

[matthewfeickert]

python -m pip install --upgrade '.[reana,local]'
# python -m pip install --upgrade 'yadage>=0.21.0' 'reana-client>=0.8.0' 'PyYAML>6.0'

has the reana extra

https://github.com/recast-hep/recast-atlas/blob/b660442ff2654db8d9bf9efe79eff4894eb6bf02/setup.py#L14

and reana-client v0.8.0 requires

reana-commons[yadage,snakemake]>=0.8.0,<0.9.0

where reana-commons v0.8.5

has

install_requires = [
    "bravado>=10.2,<10.4",
    "checksumdir>=1.1.4,<1.2",
    "click>=7.0",
    "fs>=2.0",
    "jsonschema[format]>=3.0.1,<4.0.0",
    "kombu>=4.6",
    "mock>=3.0,<4",
    "PyYAML>=5.1,<6.0",
    "Werkzeug>=0.14.1",
]

---

For reana-commons v0.9.2 this persists

install_requires = [
    "bravado>=10.2,<10.4",
    "checksumdir>=1.1.4,<1.2",
    "click>=7.0",
    "fs>=2.0",
    "jsonschema[format]>=3.0.1,<4.0.0",
    "kombu>=4.6",
    "mock>=3.0,<4",
    "PyYAML>=5.1,<6.0",
    "Werkzeug>=0.14.1",
    "wcmatch>=8.3,<8.5",
]

but for 0.9.3a6 there's a break given https://github.com/reanahub/reana-commons/pull/402

install_requires = [
    "bravado>=10.2,<10.4",
    "checksumdir>=1.1.4,<1.2",
    "click>=7.0",
    "fs>=2.0",
    "jsonschema[format]>=3.0.1,<4.0.0",
    "kombu>=4.6",
    "mock>=3.0,<4",
    "PyYAML>=5.1,<7.0",
    "Werkzeug>=0.14.1",
    "wcmatch>=8.3,<8.5",
]

but

python -m pip install --upgrade --pre 'yadage>=0.21.0' 'reana-client>=0.8.0' 'PyYAML>6.0'

reana-commons v0.9.3a6 fails with

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [3 lines of output]
      error in reana-commons setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Expected end or semicolon (after version specifier)
          wcmatch>=8.3<8.5
                 ~~~~~^
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

which was fixed in https://github.com/reanahub/reana-commons/pull/386 but the backtracking in the solve goes further into the past than reana-commons v0.9.3a6 and doesn't pick it up.

[matthewfeickert]

So

python -m pip install --upgrade 'yadage>=0.21.0' 'reana-commons>=0.9.3a6' 'PyYAML>6.0'

is solvable, but

python -m pip install --upgrade 'reana-client>=0.8.5' 'reana-commons>=0.9.3a6' 'PyYAML>6.0'

is not.