`reana-client` `v0.7.5` `setup.py` on PyPI has conflicts that make `pip install` take too long / fail

[matthewfeickert]

The setup.py install_requires for reana-client v0.7.5 distributed on PyPI are creating dependency conflicts that pip's dependency resolver is struggling to resolve

https://github.com/reanahub/reana-client/blob/146020961a9483dfe7b8dde5a77f5314f1c3ee24/setup.py#L44-L61

The result is that running pip install reana-client in a clean virtual environment will attempt to install many successive wheels in an attempt to find a suitable environment. I gave up after waiting for more than 5 minutes as it is clear this behavior is a bug.

Minimal Reproducible Example

$ docker run --rm -ti python:3.8-bullseye /bin/bash
root@2bf508238222:/# python -m venv example && . example/bin/activate  # so pip won't yell at us about installing as root
(example) root@2bf508238222:/# python --version --version
Python 3.8.12 (default, Sep  3 2021, 20:28:07) 
[GCC 10.2.1 20210110]
(example) root@2bf508238222:/# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
(example) root@2bf508238222:/# python -m pip --quiet install --upgrade pip setuptools wheel
(example) root@2bf508238222:/# pip list
Package    Version
---------- -------
pip        21.2.4
setuptools 58.0.4
wheel      0.37.0
(example) root@2bf508238222:/# python -m pip install reana-client
...  # Failure case encountered

(cc @lukasheinrich as this affects https://github.com/recast-hep/recast-atlas/)

This is fixed on the current master

https://github.com/reanahub/reana-client/blob/f48a2e65950214ed530c1f454886acd512009d80/setup.py#L40-L48

$ docker run --rm -ti python:3.8-bullseye /bin/bash
root@c12020cc3d2d:/# python -m venv example && . example/bin/activate
(example) root@c12020cc3d2d:/# python -m pip --quiet install --upgrade pip setuptools wheel
(example) root@c12020cc3d2d:/# python -m pip install --upgrade "git+https://github.com/reanahub/reana-client.git@f48a2e65950214ed530c1f454886acd512009d80#egg=reana-client"
(example) root@c12020cc3d2d:/# pip show reana-client
Name: reana-client
Version: 0.8.0a2
Summary: REANA client
Home-page: https://github.com/reanahub/reana-client
Author: REANA
Author-email: info@reana.io
License: UNKNOWN
Location: /example/lib/python3.8/site-packages
Requires: tablib, click, PyYAML, reana-commons, werkzeug, jsonpointer, cwltool
Required-by: 

so this is just a problem with the latest PyPI release.

Would you consider making a patch release for this?

[matthewfeickert]

The same behavior is seen with the alpha versions out for v0.8.0 on PyPI at the moment https://pypi.org/project/reana-client/0.8.0a2/

[mvidalgarcia]

setuptools==58.0.0 has introduced some breaking changes that might be creating issues with some old library versions. Could you trying installing setuptools<58 instead?

[matthewfeickert]

setuptools==58.0.0 has introduced some breaking changes that might be creating issues with some old library versions. Could you trying installing setuptools<58 instead?

That works

$ docker run --rm -ti python:3.8-bullseye /bin/bash
root@c53b4d67a07e:/# python -m venv example && . example/bin/activate
(example) root@c53b4d67a07e:/# python -m pip --quiet install --upgrade pip "setuptools<58" wheel
(example) root@c53b4d67a07e:/# python -m pip install reana-client
(example) root@c53b4d67a07e:/# pip show reana-client
Name: reana-client
Version: 0.7.5
Summary: REANA client
Home-page: https://github.com/reanahub/reana-client
Author: REANA
Author-email: info@reana.io
License: UNKNOWN
Location: /example/lib/python3.8/site-packages
Requires: reana-commons, strict-rfc3339, PyYAML, cwl-utils, six, werkzeug, tablib, rfc3987, pyOpenSSL, webcolors, yadage-schemas, jsonpointer, cwltool, click
Required-by: 

Are the v0.8.0a1 and v0.8.0a2 pre-releases on PyPI now indicative of a planned v0.8.0 release (or release candidate) before the end of 2021? If not, would it be possible to get a patch release that just swept in the relevant changes to setup.py?

[mvidalgarcia]

Are the v0.8.0a1 and v0.8.0a2 pre-releases on PyPI now indicative of a planned v0.8.0 release (or release candidate)

Definitely.

before the end of 2021?

@tiborsimko (project manager) should be able to answer that. I hope so..

would it be possible to get a patch release that just swept in the relevant changes to setup.py?

I'm not sure if this is feasible, as IIRC, this problem comes from cwltool -> schema-salad -> rdflib-jsonld dependency, so we'd need to upgrade cwltool as well in maint-0.7 branches which I think it'd involved a lot of work. @audrium can elaborate more on this topic as he was in charge of this upgrade.

[audrium]

I'm not sure if this is feasible, as IIRC, this problem comes from cwltool -> schema-salad -> rdflib-jsonld dependency, so we'd need to upgrade cwltool as well in maint-0.7 branches which I think it'd involved a lot of work. @audrium can elaborate more on this topic as he was in charge of this upgrade.

It really seems that cwltool dependency is causing this issue. I tried to do:

python -m pip --quiet install --upgrade pip setuptools wheel
pip install "cwltool==1.0.20191022103248"

and got the same error. I guess the old version of cwltool have a strongly pinned schema-salad dependency which will not work with the latest setuptools. It will probably be easier for us to finally release v0.8.0 than to upgrade cwltool in maint-0.7 branches

[matthewfeickert]

It will probably be easier for us to finally release v0.8.0 than to upgrade cwltool in maint-0.7 branches

Okay this all sounds reasonable. Thanks for the followup and feedback @mvidalgarcia and @audrium. I'll go ahead and close this for now as @lukasheinrich and I can just temporarily enforce setuptools<58 for the recast-atlas[reana] extra until reana-client v0.8.0 is out.

[matthewfeickert]

Nice to see that for Python 3.8 reana-client v0.8.0a4 is able to install without issues with

$ python -m pip install --upgrade pip setuptools wheel
$ python -m pip install --pre reana-client

:+1:

[matthewfeickert]

@tiborsimko @mvidalgarcia @audrium :wave: Out of curiosity, as there is now another release candidate reana-client v0.8.0a5 on PyPI, do you have any updated projections on when you plan to release reana-client v0.8.0? Even before or after end of 2021 would be helpful planning info.

I hope I don't come across like I'm pressuring you for a new release before the team is ready — I just was curious given the new RC. :+1:

Never mind I see that PR #580 is open for the release. :tada: