Suported Python versions duration?

[matthewfeickert]

:wave: Hi. I'd like to bump the yadage version

https://github.com/reanahub/reana-commons/blob/0fa088a4e912742cdc80eac758762b6d01650cef/setup.py#L35

to v0.21.0 so that it enforces a new lower bound on packtivity that properly handles jqlang v1.6 and v1.7 (c.f. https://github.com/yadage/yadage/pull/132). However, yadage v0.21.0 is Python 3.8+ now that Python 3.7 is officially EOL, but reana-commons still supports Python 3.6

https://github.com/reanahub/reana-commons/blob/0fa088a4e912742cdc80eac758762b6d01650cef/setup.py#L95

https://github.com/reanahub/reana-commons/blob/0fa088a4e912742cdc80eac758762b6d01650cef/.github/workflows/ci.yml#L119-L120

I think from discussions in the past that reana tries to support EOL Python for some time given operations requirements (maybe I have this wrong?), but do you have a projected timeline on when a dependency with a requires-python of >=3.8 could be used?

[tiborsimko]

We try to be very liberal with reana-client dependencies so that the client can run on even very old users platforms.

That said, we offer AppImage that people on GNU/Linux can use, and the reana-client-go might be available in the next months too (basically only create and validate commands are missing there)

So if the Yadage version upgrade is desired, then we can certainly work on that, and bump the Python version minimal dependency, since we have offer other means for people with old systems to keep things still working.

BTW, we should also look at the cluster side whether some Yadage internals used by REANA might have changed in order to estimate any necessary changes for the reana-workflow-engine-yadage component. I assume that the new Yadage version runs well all the past Yadage workflow specs?

[matthewfeickert]

We try to be very liberal with reana-client dependencies so that the client can run on even very old users platforms.

... So if the Yadage version upgrade is desired, then we can certainly work on that, and bump the Python version minimal dependency, since we have offer other means for people with old systems to keep things still working.

I definitley understand the motivation here and applaud the commitmnent to supporting users that the REANA team provides. Trying to support as many possible Python versions as possible while also placing upper bounds on dependencies though means that dependencies that take an alternative approach of intentionally dropping EOL Pythons to provide better support for supported Python versions (and then going back and making patch releases for the EOL Python series of releases as necessary) can't get any new releases in without breaking the EOL Python versions.

To be fully clear, the new yadage version is not required for things to keep working (on Python 3.7+, Python 3.6 is defintiley broken) given yadage is pinned down to the patch release

https://github.com/reanahub/reana-commons/blob/0fa088a4e912742cdc80eac758762b6d01650cef/setup.py#L35

and yadage v0.20.1 just has packtivity as a dependency (without bounds) and so will pick up the packtivity v0.15.1 patch release that I backported to cover the Python 3.7 supported versions. The motivation to be able to bump things to yadage v0.21.0 (the Python 3.8+ version) is that I placed a lower bound on the required packtivity version to make sure that a dependency resolver (like pip's) couldn't somehow backsolve into an older packtivity release that would be broken by modern jq/jqlang.

BTW, we should also look at the cluster side whether some Yadage internals used by REANA might have changed in order to estimate any necessary changes for the reana-workflow-engine-yadage component. I assume that the new Yadage version runs well all the past Yadage workflow specs?

Good point. This should be checked. The good news is that basically the only change between yadage v0.21.0 and v0.20.1 was https://github.com/yadage/yadage/pull/111 to handle networkx v2.0, and the minor version bump is to address the CPython change and lower bound being placed on the dependnecies. In GitHub things look radically different given there was a resturcturing to modern packaging approaches but after that dust settled the hasn't been changes to the source code. So everything has continued to pass tests.

[tiborsimko]

The good news is that basically the only change between yadage v0.21.0 and v0.20.1 was yadage/yadage#111 to handle networkx v2.0

Good, so if Yadage 20 and 21 are pretty much nicely compatible internally, then we could do something like:

    "yadage": [
        "adage>=0.11,<0.12 ; python_version>='3.8'",
        "adage~=0.10.1 ; python_version<'3.8'",
        "yadage-schemas>=0.11,<0.12 ; python_version>='3.8'",
        "yadage-schemas~=0.10.6 ; python_version<'3.8'",
        "yadage>=0.21.0,<0.22.0 ; python_version>='3.8'",
        "yadage~=0.20.8.0 ; python_version<'3.8'",
    ],

Would this address what you are looking for?

I can test locally whether this would work fine, both client-wise and cluster-wise.

P.S. Note that I'd like to keep some upper pins for Yadage dependencies in order to the respect the semantic versioning; the detailed pins are there only because it happened that a past minor upgrade broke stuff. However, if you plan to keep the backwards compatibility for any future Yadage 21.x minor release, then the above setup would give enough freedom already, since it would automatically trust and install any latest Yadage 21 minor release there is on PyPI. And when you make a new major Yadage 22 release one day, with possible breaking changes, we would explicitly test for any incompatible internals that r-w-e-yadage might be using inside REANA, and allow the new version only when everything works well. WDYT?

[tiborsimko]

I can test locally whether this would work fine, both client-wise and cluster-wise.

I tried the above approach and unfortunately the Yadage 0.21.0 does not work on the server-side; I'm seeing publisher-related problems:

2023-09-21 14:33:48,865 | reana-workflow-engine-yadage | MainThread | DEBUG | Publishing workflow running status...
2023-09-21 14:33:48,868 | amqp | MainThread | DEBUG | Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@reana-message-broker-0.reana-message-broker.default.svc.cluster.local', 'copyright': 'Copyright (c) 2007-2022 VMware, Inc. or its affiliates.', 'information': 'Licensed under the MPL 2.0. Website: https://rabbitmq.com', 'platform': 'Erlang/OTP 24.3.4.1', 'product': 'RabbitMQ', 'version': '3.8.34'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
2023-09-21 14:33:48,869 | amqp | MainThread | DEBUG | using channel_id: 1
2023-09-21 14:33:48,869 | amqp | MainThread | DEBUG | Channel open
2023-09-21 14:33:48,870 | root | MainThread | DEBUG | Publisher: message sent: {'workflow_uuid': 'a45b4f3c-cc5d-44df-9df3-4ca23da86483', 'logs': '', 'status': 1, 'message': None}
2023-09-21 14:33:48,870 | yadage.utils | MainThread | DEBUG | setting up backend fromenv with opts {}
2023-09-21 14:33:48,871 | root | MainThread | DEBUG | stat: path should be string, bytes, os.PathLike or integer, not NoneType
2023-09-21 14:33:48,872 | root | MainThread | DEBUG | Publisher: message sent: {'workflow_uuid': 'a45b4f3c-cc5d-44df-9df3-4ca23da86483', 'logs': 'Workflow exited unexpectedly.\nstat: path should be string, bytes, os.PathLike or integer, not NoneType', 'status': 3, 'message': None}
2023-09-21 14:33:48,872 | root | MainThread | DEBUG | Publisher: closing queue connection

This is exactly the same problem as in https://github.com/reanahub/reana-workflow-engine-yadage/issues/222 which apparently depends on https://github.com/yadage/yadage/issues/116 which is still open? So we may need to stick to using Yadage 0.20.1 on the server-side until this is fully investigated...

[mr-c]

Hello. The CWL reference runner also dropped support for Python 3.6 & Python 3.7 this year. Would you accept a PR to enable a later version of cwltool for Python 3.8+ users?

[tiborsimko]

@matthewfeickert @mr-c We plan to start dropping support for both Python 3.6 and 3.7 as part of working on the forthcoming REANA release series (0.95). @matthewfeickert I hope it is OK to drop support for 3.7 also on the client side? @mr-c Once we have done that, we could look into upgrading the cwltool indeed (although I assume that it might be more complicated as we discussed in https://github.com/reanahub/reana-commons/pull/427).

[matthewfeickert]

I hope it is OK to drop support for 3.7 also on the client side?

@tiborsimko That's great! I've been trying to keep RECAST following the Python EOL schedule and only supporting Pythons that are not EOL. Most ATLAS users are on Python 3.8 or Python 3.9 at the oldest (most are working on Python 3.10+).

$ eol python
┌───────┬────────────┬─────────┬────────────────┬────────────┬────────────┐
│ cycle │  release   │ latest  │ latest release │  support   │    eol     │
├───────┼────────────┼─────────┼────────────────┼────────────┼────────────┤
│ 3.12  │ 2023-10-02 │ 3.12.2  │   2024-02-06   │ 2025-04-02 │ 2028-10-31 │
│ 3.11  │ 2022-10-24 │ 3.11.8  │   2024-02-06   │ 2024-04-01 │ 2027-10-31 │
│ 3.10  │ 2021-10-04 │ 3.10.14 │   2024-03-19   │ 2023-04-05 │ 2026-10-31 │
│ 3.9   │ 2020-10-05 │ 3.9.19  │   2024-03-19   │ 2022-05-17 │ 2025-10-31 │
│ 3.8   │ 2019-10-14 │ 3.8.19  │   2024-03-19   │ 2021-05-03 │ 2024-10-31 │
│ 3.7   │ 2018-06-26 │ 3.7.17  │   2023-06-05   │ 2020-06-27 │ 2023-06-27 │
│ 3.6   │ 2016-12-22 │ 3.6.15  │   2021-09-03   │ 2018-12-24 │ 2021-12-23 │
│ 3.5   │ 2015-09-12 │ 3.5.10  │   2020-09-05   │   False    │ 2020-09-30 │
│ 3.4   │ 2014-03-15 │ 3.4.10  │   2019-03-18   │   False    │ 2019-03-18 │
│ 3.3   │ 2012-09-29 │ 3.3.7   │   2017-09-19   │   False    │ 2017-09-29 │
│ 3.2   │ 2011-02-20 │ 3.2.6   │   2014-10-12   │   False    │ 2016-02-20 │
│ 3.1   │ 2009-06-26 │ 3.1.5   │   2012-04-06   │   False    │ 2012-04-09 │
│ 3.0   │ 2008-12-03 │ 3.0.1   │   2009-02-12   │   False    │ 2009-06-27 │
│ 2.7   │ 2010-07-03 │ 2.7.18  │   2020-04-19   │   False    │ 2020-01-01 │
│ 2.6   │ 2008-10-01 │ 2.6.9   │   2013-10-29   │   False    │ 2013-10-29 │
└───────┴────────────┴─────────┴────────────────┴────────────┴────────────┘