python 3.11: cannot submit snakemake workflows

[tiborsimko]

Seen whilst testing the researcher-oriented installation method for REANA 0.9.1-alpha.1 using Python 3.11 on the client side:

$ reana-client run -w snakemake -f ./reana-snakemake.yaml
==> Creating a workflow...
Traceback (most recent call last):
  File "/home/tibor/.virtualenvs/reana-0.9.0/lib/python3.11/site-packages/snakemake/__init__.py", line 699, in snakemake
    success = workflow.execute(
              ^^^^^^^^^^^^^^^^^
  File "/home/tibor/.virtualenvs/reana-0.9.0/lib/python3.11/site-packages/snakemake/workflow.py", line 960, in execute
    self.scheduler = JobScheduler(
                     ^^^^^^^^^^^^^
  File "/home/tibor/.virtualenvs/reana-0.9.0/lib/python3.11/site-packages/snakemake/scheduler.py", line 98, in __init__
    from ratelimiter import RateLimiter
  File "/home/tibor/.virtualenvs/reana-0.9.0/lib/python3.11/site-packages/ratelimiter.py", line 36, in <module>
    class RateLimiter(object):
  File "/home/tibor/.virtualenvs/reana-0.9.0/lib/python3.11/site-packages/ratelimiter.py", line 127, in RateLimiter
    __aexit__ = asyncio.coroutine(__exit__)
                ^^^^^^^^^^^^^^^^^
AttributeError: module 'asyncio' has no attribute 'coroutine'
==> ERROR: Cannot create workflow snakemake:
Snakemake specification is invalid.

[giuseppe-steduto]

This happens because snakemake uses the ratelimiter dependency, which in turn uses the coroutine decorator of the asyncio module that was removed since python 3.11.

ratelimiter still has an open issue about that and snakemake switched the dependency to throttler in its 7.18.2 release. In REANA, we currently use snakemake==6.8.0. Upgrading snakemake would work, but it may not be effortless. On the other hand, the fix for ratelimiter is rather easy and there are multiple forks of the project that address this issue in different ways (take a look at this or this, for example) and that apparently would solve our problem. For example, here's what I tried:

1. Create a new virtualenv with Python 3.11
2. pip install reana-client
3. Set the REANA_SERVER_URL and REANA_ACCESS_TOKEN env variables
4. Run reana-client run -w snakemake -f ./reana-snakemake.yaml and verify that there is indeed the issue with coroutine
5. "Patch" ratelimiter by replacing it with one of the forked projects, such as pip install -e git+git://github.com/classroomtechtools/ratelimiter@master#egg=ratelimiter
6. Run again reana-client run -w snakemake -f ./reana-snakemake.yaml and verify that the issue is gone.

One problem though is that the ratelimiter repository may not be actively maintained, therefore it might be hard to have the fixed python package published. Also, we would of course need more testing to make sure that fixing ratelimiter works as expected.

What do you think?