[BUG] <ModuleNotFoundError: No module named 'cwltool'>

[rameshvnair]

Describe the bug Contents of cwltool.log:

Original command: ./pgap.py -D singularity -r -o /oak/stanford/scg/prj_gbsc/workspace/rvnair/pgap/mg37_results test_genomes/MG37/input.yaml

Docker command: /usr/bin/singularity exec --bind /scg/apps/software/pgap/2020-09-24.build4894/input-2020-09-24.build4894:/pgap/input:ro --bind /scg/apps/software/pgap/2020-09-24.build4894/test_genomes/MG37:/pgap/user_input --bind /scg/apps/software/pgap/2020-09-24.build4894/test_genomes/MG37/pgap_input_8ffol647.yaml:/pgap/user_input/pgap_input.yaml:ro --bind /oak/stanford/scg/prj_gbsc/workspace/rvnair/pgap/mg37_results:/pgap/output:rw --pwd /pgap docker://ncbi/pgap:2020-09-24.build4894 cwltool --timestamps --disable-color --preserve-entire-environment --outdir /pgap/output pgap/pgap.cwl /pgap/user_input/pgap_input.yaml

--- Start YAML Input ---
fasta:
  class: File
  location: ASM2732v1.annotation.nucleotide.1.fasta
submol:
  class: File
  location: pgap_submol_mjchj7bi.yaml
supplemental_data: { class: Directory, location: /pgap/input }
report_usage: true
--- End YAML Input ---

--- Start Runtime Report ---
{
    "CPU cores": 64,
    "Docker image": "ncbi/pgap:2020-09-24.build4894",
    "cpu flags": "fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc art rep_good nopl nonstop_tsc extd_apicid amd_dcm aperfmperf eagerfpu pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_l2 cpb hw_pstate retpoline_amd ssbd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca",
    "cpu model": "AMD EPYC 7301 16-Core Processor",
    "max user processes": 4096,
    "memory (GiB)": 503.8,
    "memory per CPU core (GiB)": 7.9,
    "open files": 16384,
    "tmp disk space (GiB)": 1717.4,
    "virtual memory": 18874368,
    "work disk space (GiB)": 4079.1
}
--- End Runtime Report ---

Traceback (most recent call last):
  File "/usr/local/bin/cwltool", line 7, in <module>
    from cwltool.main import run
ModuleNotFoundError: No module named 'cwltool'

Ran singularity shell on the sif image and see the same error despite the tool being present in /usr/local/bin as verified below:

$ singularity shell pgap_2020-09-24.build4894.sif
Singularity pgap_2020-09-24.build4894.sif:/scg/apps/software/pgap/2020-09-24.build4894> cwltool
Traceback (most recent call last):
  File "/usr/local/bin/cwltool", line 7, in <module>
    from cwltool.main import run
ModuleNotFoundError: No module named 'cwltool'
Singularity pgap_2020-09-24.build4894.sif:/scg/apps/software/pgap/2020-09-24.build4894> ls -l /usr/local/bin
total 89
-rwxr-xr-x 1 root root 54753 Sep 24 06:47 bagit.py
-rwxr-xr-x 1 root root   225 Sep 24 06:47 chardetect
-rwxr-xr-x 1 root root   218 Sep 24 06:47 coloredlogs
-rwxr-xr-x 1 root root   223 Sep 24 06:47 csv2rdf
-rwxr-xr-x 1 root root   976 Sep 24 06:48 cwl-runner
-rwxr-xr-x 1 root root   213 Sep 24 06:48 cwltool
...

To Reproduce $ ./pgap.py -D singularity -r -o /oak/stanford/scg/prj_gbsc/workspace/rvnair/pgap/mg37_results test_genomes/MG37/input.yaml Ran the MG37 test repeatedly 10 times using the above command and see the same cwltool not found error every time.

Expected behavior I expected the cwltool to be found in the singularity image when I verified its presence.

Software versions:

• OS: CentOS Linux release 7.7.1908 (Core)
• Latest PGAP version 2020-09-24.build4894
• singularity version 3.4.1-1.2.el7

Log Files Reran pgap.py with the --debug flag but the debug folder had no log files. I see the dollowing error at the command line and the cwltool.log contents I reported above.

$ ./pgap.py --debug -D singularity -r -o /oak/stanford/scg/prj_gbsc/workspace/rvnair/pgap/mg37_results test_genomes/MG37/input.yaml
PGAP version 2020-09-24.build4894 is up to date.
Output will be placed in: /oak/stanford/scg/prj_gbsc/workspace/rvnair/pgap/mg37_results
PGAP failed, docker exited with rc = 1
Unable to find error in log file.

[azat-badretdin]

Thanks for your detailed report, Ramesh.

I tried to reproduce the error by running locally cwltool --help inside singularity shell docker://ncbi/pgap:2020-09-24.build4894 and I was not able to reproduce it.

[azat-badretdin]

My version of singularity:

$ singularity --version
singularity version 3.7.0

[rameshvnair]

Thanks @azat-badretdin. I'll look into upgrading to the latest Singularity 3.7.0 to try to resolve this issue.

[azat-badretdin]

You are welcome, Ramesh!