search

yuch7 / cwlexec

A new open source tool to run CWL workflows on LSF
Other
36 stars 8 forks source link

Failed to "Fill out the scatter gather result in the script" #41

Closed mscheremetjew closed 5 years ago

mscheremetjew commented 5 years ago

command (just the usual):

cwlexec -p --workdir /home/user/<username>/output/ TranscriptsAnnotation-i5only-wf.cwl TranscriptsAnnotation-i5only-wf.test.job.yaml

cwlexec fails at the scattered functionalAnalysis step and reports the following:

[15:49:15.857] INFO  - The step (functionalAnalysis/runInterproscan) scatter of 1 jobs.
[15:49:15.857] INFO  - Started job (functionalAnalysis/runInterproscan_1) with
bsub \
-cwd \
/home/user/maxim/output/a286910d-d3e3-41a4-b707-1e0a7654e4d4/functionalAnalysis/runInterproscan/scatter1 \
-o \
%J_out \
-e \
%J_err \
-env \
all,TMPDIR=/home/user/maxim/output/a286910d-d3e3-41a4-b707-1e0a7654e4d4 \
-R \
mem > 8192 \
-n \
3 \
/bin/sh -c 'interproscan.sh --outfile /home/user/maxim/output/a286910d-d3e3-41a4-b707-1e0a7654e4d4/functionalAnalysis/runInterproscan/transcript-01.p2_transcript-01.p2.i5_annotations --disable-precalc --goterms --pathways --tempdir /home/user/maxim/output/a286910d-d3e3-41a4-b707-1e0a7654e4d4/functionalAnalysis/runInterproscan --input /home/user/maxim/output/a286910d-d3e3-41a4-b707-1e0a7654e4d4/functionalAnalysis/splitSeqs/transcript-01.p2_transcript-01.p2.fasta --applications PfamA --formats TSV'
[15:49:15.877] INFO  - Job (functionalAnalysis/runInterproscan_1) was submitted. Job <1421> is submitted to default queue <normal>.
[15:49:15.877] INFO  - Started to wait for jobs by
bwait \
-w \
done(1421)
[15:50:07.854] INFO  - Fill out the scatter gather result in the script /home/user/maxim/output/a286910d-d3e3-41a4-b707-1e0a7654e4d4/functionalAnalysis/runInterproscan/functionalAnalysis/runInterproscan
[15:50:07.855] ERROR - Failed to wait for job functionalAnalysis/runInterproscan <1415>, Failed to write file "/home/user/maxim/output/a286910d-d3e3-41a4-b707-1e0a7654e4d4/functionalAnalysis/runInterproscan/functionalAnalysis/runInterproscan": /home/user/maxim/output/a286910d-d3e3-41a4-b707-1e0a7654e4d4/functionalAnalysis/runInterproscan/functionalAnalysis/runInterproscan (No such file or directory)
[15:50:07.855] ERROR - The workflow (TranscriptsAnnotation-i5only-wf) exited with <255>.
[15:50:07.855] WARN  - killing waiting job (functionalAnalysis/runInterproscan) <1415>.
[15:50:07.855] WARN  - killing waiting job (functionalAnalysis/combineResults) <1418>.
mscheremetjew commented 5 years ago

Here are the workflow, subworkflow and tool descriptions: cwl_files.zip

qiangjia commented 5 years ago

The attached zip cannot be run, missing esl-reformat.cwl.

mscheremetjew commented 5 years ago

@qiangjia sorry forgot about this. Also I made sure you've got the job test input files. All tools are containerised except InterProScan (interproscan.sh). But that is not necessary to reproduced the issue anyway. issue_41.zip $ unzip issue_41.zip $ cd issue_41 $ cwlexec --workdir /home/user//output TranscriptsAnnotation-i5only-wf.cwl jobs/TranscriptsAnnotation-i5only-wf.test.job.yaml

skeeey commented 5 years ago

@mscheremetjew, @qiangjia has fixed this problem, if the fix works well, would you close this issue?

mscheremetjew commented 5 years ago

@skeeey sure. Are you going to release a new version, so I can test the fix or do you want us to compile your latest source and test on that?

skeeey commented 5 years ago

@mscheremetjew, our next release may be in March, I compile a package with the latest code, you can use it, or you can directly compile our code, it should be easy to compile :), you can refer to https://github.com/IBMSpectrumComputing/cwlexec#build

cwlexec-0.2.1.tar.gz

mscheremetjew commented 5 years ago

@skeeey @qiangjia Tested successfully. Many Thanks for the fix.