Incompatibility between cwltools and singularity for virtus2

[bioinfo-fiend]

I've been trying to run Virtus2 on an HPC environment, I have tried with different versions of cwltool and singularity and come across constantly the same error. I have a specific conda/python virtual environment in which I have all the needed requirements to run the virtus2 algorithm. I was running against my own specific files but opted for the "basic" tutorial.

I am currently running the out-of-the-box method provided on the Virtus2 GitHub page.

cwltool --debug --singularity /hpc/software/algorithms/VIRTUS2/workflow/createindex.cwl
/hpc/software/algorithms/VIRTUS2/workflow/createindex.job.yaml

This is the error I run into:

ValueError: invalid literal for int() with base 10: 'a'
ERROR Workflow or tool uses unsupported feature:
Singularity is required to run this tool: invalid literal for int() with base 10: 'a'

My current python/conda environment:

• apptainer/singularity version 1.1.3-1.el7 (latest version of singularity)
• cwltool version 3.1.20210922203925
• python 3.5 to 3.9

[yyoshiaki]

Can you share the full traceback?

Also, do other cwl programs work on your system? Here is an example of a test.

cwltool --singularity https://raw.githubusercontent.com/common-workflow-language/common-workflow-language/main/v1.0/v1.0/v1.0/cat3-tool-mediumcut.cwl https://github.com/common-workflow-language/common-workflow-language/blob/main/v1.0/v1.0/cat-job.json

https://github.com/common-workflow-language/cwltool#using-udocker

If not, it would be a problem with cwltool, so please ask at cwltool's repo.

[bioinfo-fiend]

Hey @yyoshiaki thanks for responding. I reported the issue on the cwltools repo and they gave a response and I did not have any issues running their examples but continue with the same error when trying to run VRITUS2. In any case, I have provided the full traceback below using VIRTUS2 via conda:

INFO /hpc/local/CentOS7/software/miniconda-4.12.0/envs/virtus2_env/bin/cwltool 3.1.20210922203925
INFO Resolved '/hpc/software/algorithms/VIRTUS2/workflow/createindex.cwl' to 'file:///hpc/software/algorithms/VIRTUS2/workflow/createindex.cwl'
INFO:    Using cached SIF image
DEBUG Pulled Docker image docker://docker.io/node:slim  using singularity
DEBUG Running nodejs via ['singularity', 'exec', '--contain', '--ipc', '--cleanenv', '--pid', 'docker://docker.io/node:slim', 'node', '--eval']
DEBUG [workflow ] initialized from file:///hpc/software/algorithms/VIRTUS2/workflow/createindex.cwl#createindex
INFO [workflow ] start
DEBUG [workflow ] inputs {
    "dir_name_STAR_human": "STAR_index_human",
    "dir_name_STAR_virus": "STAR_index_virus",
    "output_name_genomefasta_human": "GRCh38.p13.genome.fa.gz",
    "output_name_virus": "viruses.fasta",
    "runThreadN": 40,
    "url_genomefasta_human": "ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_33/GRCh38.p13.genome.fa.gz",
    "url_virus": "https://raw.githubusercontent.com/yyoshiaki/VIRTUS2/master/data/viruses.fasta"
}
DEBUG [workflow ] job step file:///hpc/software/algorithms/VIRTUS2/workflow/createindex.cwl#createindex/star_index_virus not ready
INFO [workflow ] starting step mkdir_virus
DEBUG [step mkdir_virus] job input {
    "file:///hpc/software/algorithms/VIRTUS2/workflow/createindex.cwl#createindex/mkdir_virus/dir_name": "STAR_index_virus"
}
DEBUG [step mkdir_virus] evaluated job input to {
    "file:///hpc/software/algorithms/VIRTUS2/workflow/createindex.cwl#createindex/mkdir_virus/dir_name": "STAR_index_virus"
}
INFO [step mkdir_virus] start
DEBUG [job mkdir_virus] initializing from file:///hpc/software/algorithms/VIRTUS2/tool/mkdir/mkdir.cwl as part of step mkdir_virus
DEBUG [job mkdir_virus] {
    "dir_name": "STAR_index_virus"
}
DEBUG [job mkdir_virus] path mappings is {}
DEBUG [job mkdir_virus] command line bindings is [
    {
        "position": [
            -1000000,
            0
        ],
        "datum": "mkdir"
    },
    {
        "position": [
            -1000000,
            1
        ],
        "datum": "-p"
    },
    {
        "position": [
            0,
            "dir_name"
        ],
        "datum": "STAR_index_virus"
    }
]
DEBUG Singularity version: apptainer version 1.1.4-1.el7.
DEBUG Singularity error
Traceback (most recent call last):
  File "/hpc/local/CentOS7/software/miniconda-4.12.0/envs/virtus2_env/lib/python3.6/site-packages/cwltool/job.py", line 778, in run
    runtimeContext.tmp_outdir_prefix,
  File "/hpc/local/CentOS7/software/miniconda-4.12.0/envs/virtus2_env/lib/python3.6/site-packages/cwltool/singularity.py", line 249, in get_from_requirements
    if not self.get_image(cast(Dict[str, str], r), pull_image, force_pull):
  File "/hpc/local/CentOS7/software/miniconda-4.12.0/envs/virtus2_env/lib/python3.6/site-packages/cwltool/singularity.py", line 113, in get_image
    if is_version_3_or_newer():
  File "/hpc/local/CentOS7/software/miniconda-4.12.0/envs/virtus2_env/lib/python3.6/site-packages/cwltool/singularity.py", line 43, in is_version_3_or_newer
    return int(get_version()[0]) >= 3
ValueError: invalid literal for int() with base 10: 'a'
ERROR Workflow or tool uses unsupported feature:
Singularity is required to run this tool: invalid literal for int() with base 10: 'a'
Traceback (most recent call last):
  File "/hpc/local/CentOS7/software/miniconda-4.12.0/envs/virtus2_env/lib/python3.6/site-packages/cwltool/job.py", line 778, in run
    runtimeContext.tmp_outdir_prefix,
  File "/hpc/local/CentOS7/software/miniconda-4.12.0/envs/virtus2_env/lib/python3.6/site-packages/cwltool/singularity.py", line 249, in get_from_requirements
    if not self.get_image(cast(Dict[str, str], r), pull_image, force_pull):
  File "/hpc/local/CentOS7/software/miniconda-4.12.0/envs/virtus2_env/lib/python3.6/site-packages/cwltool/singularity.py", line 113, in get_image
    if is_version_3_or_newer():
  File "/hpc/local/CentOS7/software/miniconda-4.12.0/envs/virtus2_env/lib/python3.6/site-packages/cwltool/singularity.py", line 43, in is_version_3_or_newer
    return int(get_version()[0]) >= 3
ValueError: invalid literal for int() with base 10: 'a'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/hpc/local/CentOS7/software/miniconda-4.12.0/envs/virtus2_env/lib/python3.6/site-packages/cwltool/main.py", line 1248, in main
    tool, initialized_job_order_object, runtimeContext, logger=_logger
  File "/hpc/local/CentOS7/software/miniconda-4.12.0/envs/virtus2_env/lib/python3.6/site-packages/cwltool/executors.py", line 60, in __call__
    return self.execute(process, job_order_object, runtime_context, logger)
  File "/hpc/local/CentOS7/software/miniconda-4.12.0/envs/virtus2_env/lib/python3.6/site-packages/cwltool/executors.py", line 143, in execute
    self.run_jobs(process, job_order_object, logger, runtime_context)
  File "/hpc/local/CentOS7/software/miniconda-4.12.0/envs/virtus2_env/lib/python3.6/site-packages/cwltool/executors.py", line 250, in run_jobs
    job.run(runtime_context)
  File "/hpc/local/CentOS7/software/miniconda-4.12.0/envs/virtus2_env/lib/python3.6/site-packages/cwltool/job.py", line 823, in run
    ) from err
cwltool.errors.UnsupportedRequirement: Singularity is required to run this tool: invalid literal for int() with base 10: 'a'

[mr-c]

continuing here from https://github.com/common-workflow-language/cwltool/issues/1777

cwltool 3.1.20210922203925 doesn't support apptainer's different output for singularity version; that is fixed in a recent releases of cwltool (looks like https://github.com/common-workflow-language/cwltool/releases/tag/3.1.20220607081835 was the first release with that fix)

cwltool.sandboxjs.JavascriptException: cwltool requires Node.js engine to evaluate and validate Javascript expressions, but couldn't find it. Tried nodejs, node, docker run node:slim

For udocker, you must have a local copy of nodejs installed to be able to parse CWL expressions. With --singularity and recent cwltool, it can fallback to using Singularity/Apptainer to run a nodejs container, but that can be rather slow for complex workflows that use CWL expressions heavily.

Finally, I see that you are using conda; I would recommend making a coda environment with a newer version of Python, like python==3.10 as that will also speed up parsing and managing CWL workflows.

[mr-c]

Ah, in fact the use of Python 3.6 is probably why you got such an old cwltool version; conda-forge dropped support for it; 3.1.20211001174446 was the last cwltool release to get published for Conda: https://anaconda.org/conda-forge/cwltool/files?version=3.1.20211001174446 (compare to https://anaconda.org/conda-forge/cwltool/files?version=3.1.20211004060744 )

[mr-c]

Ah, in fact the use of Python 3.6 is probably why you got such an old cwltool version; conda-forge dropped support for it; 3.1.20211001174446 was the last cwltool release to get published for Conda: https://anaconda.org/conda-forge/cwltool/files?version=3.1.20211001174446 (compare to https://anaconda.org/conda-forge/cwltool/files?version=3.1.20211004060744 )

@mr-c Should I also use the latest version of python as well?

Conda-forge hasn't completed its support of Python 3.11, so stick to Python 3.10 (or at least Python 3.7)

mamba create -n py310_cwltool python==3.10 cwltool
# or
conda create -n py310_cwltool python==3.10 cwltool
# will get you cwltool 3.1.20221201130942 , that is from 2022-12-01 :-)

[bioinfo-fiend]

@mr-c indeed I've created a completely new conda environment and have installed cwltool with python version 3.10 and this works as expected. Would there be any major difference if using a python virtual environment (venv) versus using a conda environment?

[mr-c]

@mr-c indeed I've created a completely new conda environment and have installed cwltool with python version 3.10 and this works as expected.

I'm glad to hear that!

Would there be any major difference if using a python virtual environment (venv) versus using a conda environment?

Our primary support is for PyPI and venv. Conda is on a best effort basis. Currently the two are fairly equal, though with venv you can use the faster Python 3.11 today.

[bioinfo-fiend]

@mr-c sounds good, will take a look at it :D

[bioinfo-fiend]

@yyoshiaki when running the following default to create an index, I get the following output, not sure if this is an error.

INFO /hpc/local/CentOS7/software/miniconda-4.12.0/envs/virtus2_env/bin/cwltool 3.1.20221201130942
INFO Resolved './VIRTUS2/workflow/createindex.cwl' to 'file:///hpc/software/algorithms/VIRTUS2/workflow/createindex.cwl'
INFO [workflow ] start
INFO [workflow ] starting step rnaseq_star_index_human
INFO [step rnaseq_star_index_human] start
INFO [workflow rnaseq_star_index_human] start
INFO [workflow rnaseq_star_index_human] starting step wget_genomefasta
INFO [step wget_genomefasta] start
INFO ['singularity', 'pull', '--force', '--name', 'inutano_wget:1.20.3-r1.sif', 'docker://inutano/wget:1.20.3-r1']
INFO:    Using cached SIF image
INFO [job wget_genomefasta] /scratch/18435398/121q5j0n$ singularity \
    --quiet \
    exec \
    --contain \
    --ipc \
    --cleanenv \
    --pid \
    --home \
    /scratch/18435398/121q5j0n:/DCAOoL \
    --bind \
    /scratch/18435398/14pqeavs:/tmp \
    --pwd \
    /DCAOoL \
    /hpc/software/algorithms/inutano_wget:1.20.3-r1.sif \
    wget \
    --output-document=GRCh38.p13.genome.fa.gz \
    ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_33/GRCh38.p13.genome.fa.gz 2> /scratch/18435398/121q5j0n/wget-stderr.log
INFO [job wget_genomefasta] completed success
INFO [step wget_genomefasta] completed success
INFO [workflow rnaseq_star_index_human] starting step gunzip_genomefasta
INFO [step gunzip_genomefasta] start
INFO ['singularity', 'pull', '--force', '--name', 'alpine:3.9.sif', 'docker://alpine:3.9']
INFO:    Using cached SIF image
INFO [job gunzip_genomefasta] /scratch/18435398/q90cxw7j$ singularity \
    --quiet \
    exec \
    --contain \
    --ipc \
    --cleanenv \
    --pid \
    --home \
    /scratch/18435398/q90cxw7j:/DCAOoL \
    --bind \
    /scratch/18435398/w9uv3v85:/tmp \
    --bind \
    /scratch/18435398/121q5j0n/GRCh38.p13.genome.fa.gz:/var/lib/cwl/stg110e8044-136c-4956-96ef-6fa45374ef40/GRCh38.p13.genome.fa.gz:ro \
    --pwd \
    /DCAOoL \
    /hpc/software/algorithms/alpine:3.9.sif \
    gunzip \
    --stdout \
    /var/lib/cwl/stg110e8044-136c-4956-96ef-6fa45374ef40/GRCh38.p13.genome.fa.gz > /scratch/18435398/q90cxw7j/GRCh38.p13.genome.fa 2> /scratch/18435398/q90cxw7j/gunzip-stderr.log
WARNING [job gunzip_genomefasta] exited with status: 1
WARNING [job gunzip_genomefasta] completed permanentFail
WARNING [step gunzip_genomefasta] completed permanentFail
INFO [workflow rnaseq_star_index_human] completed permanentFail
WARNING [step rnaseq_star_index_human] completed permanentFail
INFO [workflow ] completed permanentFail
{
    "downloaded_virus": null,
    "starIndex_virus": null,
    "starIndex_human": null,
    "downloaded_genomefasta_human": {
        "location": "file:///hpc/software/algorithms/GRCh38.p13.genome.fa.gz",
        "basename": "GRCh38.p13.genome.fa.gz",
        "class": "File",
        "checksum": "sha1$2b7cdbbf55b0c507ad6f56de3afb2fe9ff8a8da5",
        "size": 889409368,
        "path": "/hpc/software/algorithms/GRCh38.p13.genome.fa.gz"
    }
}
WARNING Final process status is permanentFail

[mr-c]

@bioinfo-fiend try again with cwltool --cachedir cache (and the rest of your options); then you can examine the gunzip-stderr.log in one of the folders inside cache

[yyoshiaki]

@mr-c @bioinfo-fiend Thank you very much for everything!