FATAL: "cwltool": executable file not found in $PATH in 2022-12-13.build6494

[ponomarevsy]

Does build 2022-12-13.build6494 contain the "cwltool" as build 2019-11-25-build4172 did? Please let me know.

I am getting this error for 2022-12-13.build6494:

# singularity exec --pwd /pgap --bind $CWD/input-$DATE.$BUILD:/pgap/input:ro --bind $CWD/test_genomes/MG37:/pgap/user_input --bind $CWD/test_genomes/MG37/pgap_input.yaml:/pgap/user_input/pgap_input.yaml:ro --bind $CWD/mg37_results:/pgap/output:rw docker://ncbi/pgap:$DATE.$BUILD cwltool --timestamps --outdir /pgap/output pgap.cwl /pgap/user_input/pgap_input.yaml
INFO:    Using cached SIF image
FATAL:   "cwltool": executable file not found in $PATH

Thanks!

[azat-badretdin]

Thank you for your report, user @ponomarevsy !

When the push comes to shove in pgap.py our similar command looks like this:

Docker command: /usr/local/bin/singularity exec \
--bind /home/gp_aws/.pgap/input-2023-01-08.build4295:/pgap/input:ro \
--bind /home/gp_aws/.pgap/test_genomes-2023-01-08.build4295/MG37:/pgap/user_input  \
--bind /home/gp_aws/.pgap/test_genomes-2023-01-08.build4295/MG37/pgap_input_uvb_it5x.yaml:/pgap/user_input/pgap_input.yaml:ro \
--bind /tmp:/tmp:rw \
--bind /home/gp_aws/output:/pgap/output:rw \
--bind /home/gp_aws/output/debug/log:/log/srv \
--pwd /pgap docker://ncbi/pgap-dev:2023-01-08.build4295 \
cwltool --timestamps --debug --disable-color \
--preserve-entire-environment --outdir /pgap/output \
--tmpdir-prefix /pgap/output/debug/tmpdir/ --leave-tmpdir \
--tmp-outdir-prefix /pgap/output/debug/tmp-outdir/ \
--copy-outputs pgap/taxcheck.cwl /pgap/user_input/pgap_input.yaml

I do not see significant differences between your command and our command.

We install cwltool in the container in the venv location: /root/venv/bin/ and add it to the PATH. Could you please check if your PATH contains this?

singularity exec --pwd /pgap \
--bind $CWD/input-$DATE.$BUILD:/pgap/input:ro \
--bind $CWD/test_genomes/MG37:/pgap/user_input \
--bind $CWD/test_genomes/MG37/pgap_input.yaml:/pgap/user_input/pgap_input.yaml:ro \
--bind $CWD/mg37_results:/pgap/output:rw \
docker://ncbi/pgap:$DATE.$BUILD env | 
grep PATH

[ponomarevsy]

Thank you, @azat-badretdin. Looks like the '/root/venv/bin/' is in the PATH but, for some reason, "cwltool" is NOT there... unless I am doing something VERY wrong.

export DATE="2022-12-13" export BUILD="build6494" docker://ncbi/pgap:$DATE.$BUILD

Thus, using image "docker://ncbi/pgap:2022-12-13.build6494":

# ./run.bash
INFO:    Using cached SIF image
LD_LIBRARY_PATH=/.singularity.d/libs
MANPATH=/usr/share/man/overrides:/usr/share/man:/opt/ibutils/share/man:/usr/local/share/man:/cm/local/apps/environment-modules/current/share/man:/opt/puppetlabs/puppet/share/man:/cm/local/apps/environment-modules/current/share/man
MODULEPATH=/cm/local/modulefiles
PATH=/root/venv/bin/:/panfs/pan1.be-md.ncbi.nlm.nih.gov/gpipe/bacterial_pipeline/system/2022-12-13.build6494/arch/x86_64/bin:/panfs/pan1.be-md.ncbi.nlm.nih.gov/gpipe/bacterial_pipeline/system/third-party/bin:/panfs/pan1.be-md.ncbi.nlm.nih.gov/gpipe/bacterial_pipeline/system/third-party/hmmer/bin:/usr/local/prodigal/2.6.3/bin:/usr/local/pplacer/1.1a18/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
USER_PATH=/usr/local/sbin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/lpp/mmfs/bin:/opt/ibutils/bin:/opt/puppetlabs/bin:/opt/dell/srvadmin/bin:/opt/dell/srvadmin/sbin:/usr/lpp/mmfs/bin:/root/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin

Again, using image "docker://ncbi/pgap:2022-12-13.build6494":

# singularity exec --bind /sysapps/cluster/software/pgap/test/input-2022-12-13.build6494:/pgap/input:ro --bind /sysapps/cluster/software/pgap/test/test_genomes-2022-12-13.build6494/MG37:/pgap/user_input  --bind /sysapps/cluster/software/pgap/test/test_genomes-2022-12-13.build6494/MG37/pgap_input.yaml:/pgap/user_input/pgap_input.yaml:ro --bind /tmp:/tmp:rw --bind /sysapps/cluster/software/pgap/test/output:/pgap/output:rw --bind /sysapps/cluster/software/pgap/test/output/debug/log:/log/srv --pwd /pgap docker://ncbi/pgap:2022-12-13.build6494 cwltool --timestamps --debug --disable-color --preserve-entire-environment --outdir /pgap/output --tmpdir-prefix /pgap/output/debug/tmpdir/ --leave-tmpdir --tmp-outdir-prefix /pgap/output/debug/tmp-outdir/ --copy-outputs pgap/taxcheck.cwl /pgap/user_input/pgap_input.yaml
INFO:    Using cached SIF image
FATAL:   "cwltool": executable file not found in $PATH

I've noticed you are using a (newer) developmental version of pgap: docker://ncbi/pgap-dev:2023-01-08.build4295. Where can I download the 2023-01-08.build4295 build (input and test_genomes)? Thank you!

[azat-badretdin]

Where can I download the 2023-01-08.build4295 build (input and test_genomes)? Thank you!

We do not offer intermediate builds to public or support them because they did not pass the same level of tests we are using for normal releases.

Second, I do not thin the result will be any different from your case.

"cwltool" is NOT there

Interesting.

Could you please post a listing of that directory?

[ponomarevsy]

Never mind, I was running container as 'root' and since 'cwltool' is inside '/root/venv/bin/' on the image, singularity won't see it because it binds local home of user 'root' ('/root') on the node instead.

[azat-badretdin]

I see. The downside of being root ;-)

[ponomarevsy]

Ha-Ha! Indeed. Thanks for your help, @azat-badretdin. I think we can close this ticket.