TypeError: Multiple inheritance with NamedTuple is not supported

[kinow]

Hi!

On Ubuntu LTS, with miniconda and Python 3.9.13, I cannot run WfExS. It fails with the following traceback:

(venv) kinow@ranma:~/Development/python/workspace/WfExS-backend$ python WfExS-backend.py --full-help
Traceback (most recent call last):
  File "/home/kinow/Development/python/workspace/WfExS-backend/WfExS-backend.py", line 39, in <module>
    from wfexs_backend.wfexs_backend import WfExSBackend
  File "/home/kinow/Development/python/workspace/WfExS-backend/wfexs_backend/wfexs_backend.py", line 57, in <module>
    from .common import AbstractWfExSException
  File "/home/kinow/Development/python/workspace/WfExS-backend/wfexs_backend/common.py", line 288, in <module>
    class GeneratedContent(AbstractGeneratedContent, NamedTuple):
  File "/home/kinow/Development/python/miniconda3/lib/python3.9/typing.py", line 1929, in _namedtuple_mro_entries
    raise TypeError("Multiple inheritance with NamedTuple is not supported")
TypeError: Multiple inheritance with NamedTuple is not supported

It looks like this could be related to the following issue:

• https://bugs.python.org/issue36517
• https://github.com/python/cpython/pull/19363

I think it was first released with 3.9.0-alpha6. Given this is a change in Python, I guess WfExS will have to update the code eventually to support Py 3.9+. This patch fixes the initial command, but I am not sure if it doesn't break something else :+1:

diff --git a/wfexs_backend/common.py b/wfexs_backend/common.py
index 56878fd..a51dd7f 100644
--- a/wfexs_backend/common.py
+++ b/wfexs_backend/common.py
@@ -285,7 +285,7 @@ class ExpectedOutput(NamedTuple):
 class AbstractGeneratedContent(object):
     pass

-class GeneratedContent(AbstractGeneratedContent, NamedTuple):
+class GeneratedContent(AbstractGeneratedContent):
     """
     local: Local absolute path of the content which was generated. It
       is an absolute path in the outputs directory of the execution.
@@ -302,7 +302,7 @@ class GeneratedContent(AbstractGeneratedContent, NamedTuple):
     secondaryFiles: Optional[Sequence[AbstractGeneratedContent]] = None

-class GeneratedDirectoryContent(AbstractGeneratedContent, NamedTuple):
+class GeneratedDirectoryContent(AbstractGeneratedContent):
     """
     local: Local absolute path of the content which was generated. It
       is an absolute path in the outputs directory of the execution.

[kinow]

My version:

(venv) kinow@ranma:~/Development/python/workspace/WfExS-backend$ git log -n 1
commit f260476e782edb52d168703d2c666daf56d48de3 (HEAD -> main, tag: 0.5.4, upstream/main, upstream/HEAD)
Author: José María Fernández <jose.m.fernandez@bsc.es>
Date:   Wed Jun 1 20:46:48 2022 +0200

    Removed indirect GPLv3+ dependency which was fetched as an optional jsonschema dependency.

    Also, updated the version tag and copyright year in headers.
(venv) kinow@ranma:~/Development/python/workspace/WfExS-backend$ python WfExS-backend.py -V
WfExS-backend.py version 0.5.4 (f260476e782edb52d168703d2c666daf56d48de3)
(venv) kinow@ranma:~/Development/python/workspace/WfExS-backend$ 

[jmfernandez]

We have just released version 0.5.5, where these internal data structures have been switched from NamedTuples to dataclasses. We have also added some automated checks through GitHub actions, so we can realize sooner this issues when newer python versions are released.

Could you give it a try, please?

[kinow]

I did the following:

 cd Development/python/workspace/WfExS-backend/
 git fetch --all
 git status
 git diff
 git reset --hard
 git status
 git rebase upstream/main
 git status
 source venv/bin/activate
 pip install -U pip
 pip install -U .

Which synced my branch and re-installed the current main branch. Then running the same command that I used when I reported this issue:

(venv) kinow@ranma:~/Development/python/workspace/WfExS-backend$ python WfExS-backend.py --full-help
usage: WfExS-backend.py [-h] [--log-file LOGFILENAME] [-q] [-v] [-d] [-L LOCALCONFIGFILENAME] [--cache-dir CACHEDIR] [-V] [--full-help]
                        {init,cache,staged-workdir,export,config-validate,stage,mount-workdir,export-stage,offline-execute,execute,export-results,export-crate} ...

WfExS (workflow execution service) backend 0.5.5-5-g2b1a528 (2b1a528751100c428833d4286586095fe459668c)

optional arguments:
  -h, --help            show this help message and exit
  --log-file LOGFILENAME
                        Store messages in a file instead of using standard error and standard output (default: None)
  -q, --quiet           Only show engine warnings and errors (default: None)
  -v, --verbose         Show verbose (informational) messages (default: None)
  -d, --debug           Show debug messages (use with care, as it can disclose passphrases and passwords) (default: None)
  -L LOCALCONFIGFILENAME, --local-config LOCALCONFIGFILENAME
                        Local installation configuration file (default: /home/kinow/Development/python/workspace/WfExS-backend/wfexs_config.yml)
  --cache-dir CACHEDIR  Caching directory (default: None)
  -V, --version         show program's version number and exit
  --full-help           It returns full help (default: False)

commands:
  Command to run. It must be one of these

  {init,cache,staged-workdir,export,config-validate,stage,mount-workdir,export-stage,offline-execute,execute,export-results,export-crate}
    init                Init local setup
    cache               Cache handling subcommands
    staged-workdir      Staged working directories handling subcommands
    export              Staged working directories export subcommands
    config-validate     Validate the configuration files to be used for staging and execution
    stage               Prepare the staging (working) directory for workflow execution, fetching dependencies and contents
    mount-workdir       Mount the encrypted staging directory on secure staging scenarios
    export-stage        Export the staging directory as an RO-Crate
    offline-execute     Execute an already prepared workflow in the staging directory
    execute             Execute the stage + offline-execute + export steps
    export-results      Export the results to a remote location, gathering their public ids
    export-crate        Export an already executed workflow in the staging directory as an RO-Crate

Subparser 'init'
usage: WfExS-backend.py init [-h]

optional arguments:
  -h, --help  show this help message and exit

Subparser 'cache'
usage: WfExS-backend.py cache [-h] [-r] [--cascade] [-g] {ls,inject,fetch,rm,validate} {input,ro-crate,ga4gh-trs,workflow} [cache_command_args ...]

positional arguments:
  {ls,inject,fetch,rm,validate}
                        Cache command to perform

                        ls          List the cache entries
                        inject      Inject a new entry in the cache
                        fetch       Fetch a new cache entry, giving as input both the URI and optionally both a security context file and a security context name
                        rm          Remove an entry from the cache
                        validate    Validate the consistency of the cache
  {input,ro-crate,ga4gh-trs,workflow}
                        Cache type to perform the cache command

                        input       Cached or injected inputs
                        ro-crate    Cached RO-Crates (usually from WorkflowHub)
                        ga4gh-trs   Cached files from tools described at GA4GH TRS repositories
                        workflow    Cached workflows, which come from a git repository
  cache_command_args    Optional cache element names (default: None)

optional arguments:
  -h, --help            show this help message and exit
  -r                    Try doing the operation recursively (i.e. both metadata and data) (default: False)
  --cascade             Try doing the operation in cascade (including the URIs which resolve to other URIs) (default: False)
  -g, --glob            Given cache element names are globs (default: False)

Subparser 'staged-workdir'
usage: WfExS-backend.py staged-workdir [-h] [-g] {offline-exec,ls,mount,rm,shell,status} [staged_workdir_command_args ...]

positional arguments:
  {offline-exec,ls,mount,rm,shell,status}
                        Staged working directory command to perform

                        offline-exec    Offline execute the staged instances which match the input pattern
                        ls              List the staged instances
                            It shows the instance id, nickname,
                            encryption and whether they are damaged
                        mount           Mount the staged instances which match the input pattern
                        rm              Removes the staged instances which match the input pattern
                        shell           Launches a command in the workdir
                            First parameter is either the staged instance id or the nickname.
                            It launches the command specified after the id.
                            If there is no additional parameters, it launches a shell
                            in the mounted working directory of the instance
                        status          Shows staged instances status
  staged_workdir_command_args
                        Optional staged working directory element names (default: None)

optional arguments:
  -h, --help            show this help message and exit
  -g, --glob            Given staged workflow names are globs (default: False)

Subparser 'export'
usage: WfExS-backend.py export [-h] [-Z SECURITYCONTEXTSCONFIGFILENAME] [-E EXPORTSCONFIGFILENAME] -J WORKFLOWWORKINGDIRECTORY {ls,run} [export_contents_command_args ...]

positional arguments:
  {ls,run}              Export operations from staged working directory to perform

                        ls              List the public identifiers obtained from previous export actions
                        run             Run the different export actions, pushing the exported content and gathering the obtained permanent / public identifiers
  export_contents_command_args
                        Optional export names (default: None)

optional arguments:
  -h, --help            show this help message and exit
  -Z SECURITYCONTEXTSCONFIGFILENAME, --creds-config SECURITYCONTEXTSCONFIGFILENAME
                        Configuration file, describing security contexts, which hold credentials and similar (default: None)
  -E EXPORTSCONFIGFILENAME, --exports-config EXPORTSCONFIGFILENAME
                        Configuration file, describing exports which can be done (default: None)
  -J WORKFLOWWORKINGDIRECTORY, --staged-job-dir WORKFLOWWORKINGDIRECTORY
                        Already staged job directory (default: None)

Subparser 'config-validate'
usage: WfExS-backend.py config-validate [-h] -W WORKFLOWCONFIGFILENAME [-Z SECURITYCONTEXTSCONFIGFILENAME] [-n NICKNAME_PREFIX]

optional arguments:
  -h, --help            show this help message and exit
  -W WORKFLOWCONFIGFILENAME, --workflow-config WORKFLOWCONFIGFILENAME
                        Configuration file, describing workflow and inputs (default: None)
  -Z SECURITYCONTEXTSCONFIGFILENAME, --creds-config SECURITYCONTEXTSCONFIGFILENAME
                        Configuration file, describing security contexts, which hold credentials and similar (default: None)
  -n NICKNAME_PREFIX, --nickname-prefix NICKNAME_PREFIX
                        Nickname prefix to be used on staged workdir creation (default: None)

Subparser 'stage'
usage: WfExS-backend.py stage [-h] -W WORKFLOWCONFIGFILENAME [-Z SECURITYCONTEXTSCONFIGFILENAME] [-n NICKNAME_PREFIX]

optional arguments:
  -h, --help            show this help message and exit
  -W WORKFLOWCONFIGFILENAME, --workflow-config WORKFLOWCONFIGFILENAME
                        Configuration file, describing workflow and inputs (default: None)
  -Z SECURITYCONTEXTSCONFIGFILENAME, --creds-config SECURITYCONTEXTSCONFIGFILENAME
                        Configuration file, describing security contexts, which hold credentials and similar (default: None)
  -n NICKNAME_PREFIX, --nickname-prefix NICKNAME_PREFIX
                        Nickname prefix to be used on staged workdir creation (default: None)

Subparser 'mount-workdir'
usage: WfExS-backend.py mount-workdir [-h] -J WORKFLOWWORKINGDIRECTORY

optional arguments:
  -h, --help            show this help message and exit
  -J WORKFLOWWORKINGDIRECTORY, --staged-job-dir WORKFLOWWORKINGDIRECTORY
                        Already staged job directory (default: None)

Subparser 'export-stage'
usage: WfExS-backend.py export-stage [-h] -J WORKFLOWWORKINGDIRECTORY [--full]

optional arguments:
  -h, --help            show this help message and exit
  -J WORKFLOWWORKINGDIRECTORY, --staged-job-dir WORKFLOWWORKINGDIRECTORY
                        Already staged job directory (default: None)
  --full                Should the RO-Crate contain a copy of the inputs (and outputs)? (default: False)

Subparser 'offline-execute'
usage: WfExS-backend.py offline-execute [-h] -J WORKFLOWWORKINGDIRECTORY

optional arguments:
  -h, --help            show this help message and exit
  -J WORKFLOWWORKINGDIRECTORY, --staged-job-dir WORKFLOWWORKINGDIRECTORY
                        Already staged job directory (default: None)

Subparser 'execute'
usage: WfExS-backend.py execute [-h] -W WORKFLOWCONFIGFILENAME [-Z SECURITYCONTEXTSCONFIGFILENAME] [-E EXPORTSCONFIGFILENAME] [-n NICKNAME_PREFIX] [--full]

optional arguments:
  -h, --help            show this help message and exit
  -W WORKFLOWCONFIGFILENAME, --workflow-config WORKFLOWCONFIGFILENAME
                        Configuration file, describing workflow and inputs (default: None)
  -Z SECURITYCONTEXTSCONFIGFILENAME, --creds-config SECURITYCONTEXTSCONFIGFILENAME
                        Configuration file, describing security contexts, which hold credentials and similar (default: None)
  -E EXPORTSCONFIGFILENAME, --exports-config EXPORTSCONFIGFILENAME
                        Configuration file, describing exports which can be done (default: None)
  -n NICKNAME_PREFIX, --nickname-prefix NICKNAME_PREFIX
                        Nickname prefix to be used on staged workdir creation (default: None)
  --full                Should the RO-Crate contain a copy of the inputs (and outputs)? (default: False)

Subparser 'export-results'
usage: WfExS-backend.py export-results [-h] -J WORKFLOWWORKINGDIRECTORY

optional arguments:
  -h, --help            show this help message and exit
  -J WORKFLOWWORKINGDIRECTORY, --staged-job-dir WORKFLOWWORKINGDIRECTORY
                        Already staged job directory (default: None)

Subparser 'export-crate'
usage: WfExS-backend.py export-crate [-h] -J WORKFLOWWORKINGDIRECTORY [--full]

optional arguments:
  -h, --help            show this help message and exit
  -J WORKFLOWWORKINGDIRECTORY, --staged-job-dir WORKFLOWWORKINGDIRECTORY
                        Already staged job directory (default: None)
  --full   

So no more traceback! :tada:

Closing now. Thanks @jmfernandez !