Open mr-c opened 7 years ago
Note: In cwltoil
, sub-workflows must fully complete before any of their outputs are available for use by any other step/job. For example, the go_summary
in the functional analysis (IPS) workflow isn't subject to further processing, but its production holds up the availability of the functional_annotations
for futher processing by the parent workflow.
To run the CWL conformance tests using cwltoil
on LSF
virtualenv env
source env/bin/activate
pip install -U pip
pip install -U setuptools wheel
pip install .[cwl]
git clone https://github.com/common-workflow-language/common-workflow-language.git
cd common-workflow-language
pip install cwltest
TMP=$PWD ./run_test.sh RUNNER=toil-cwl-runner EXTRA="--batchSystem LSF --logDebug --logFile ${PWD}/log --disableCaching --user-space-docker-cmd=udocker" -j8"
(edited to use "
double quote instead of single with EXTRA
)
(edited to set TMP to a path on the shared filesystem, needed for cwltest)
Note for @mr-c : here's what I got from toil[cwl] running a workflow on a single machine, at some point I hear complaints about disk usage, although I never specified any requirements on that:
ripley 2017-06-08 18:50:26,180 Thread-82 WARNING toil.statsAndLogging: Got message from job at time 06-08-2017 18:50:26: Job used more disk than requested. Please reconsider modifying the user script to avoid the chance of failure due to incorrectly requested resources. Job 'file:///home/hmenager/ReproHackathon/reprohackathon1/cwl/tools/fastq-dump.cwl' fastq-dump 8/A/job6HSIGE used 128.93% (2.6 GB [2768723968B] used, 2.0 GB [2147483648B] requested) at the end of its run.
The tool itself is defined there: https://github.com/IFB-ElixirFr/ReproHackathon/blob/cwl/reprohackathon1/cwl/tools/fastq-dump.cwl
Priorities:
Must keep--workdir
on a non shared filesystem like/tmp
until https://github.com/BD2KGenomics/toil/pull/1573 is merged (Might be better from a performance perspective anyway)Make sure to specifyNope, hand specify minimum memory and update those as jobs fail.--retries 1
or higher so that killed job get retried with at least the default memory (from--defaultMemory 10Gi
or similar) automaticallyResourceRequirement
s with fixedor dynamicramMin
to all tools.ResourceRequirement
s at the Workflow and WorkflowStep levelsformat
with multiple options for inputs (for now) https://github.com/BD2KGenomics/toil/issues/1692--preserve-environment
takes a space separated list of environment variables to preserve, not a comma separated list as the docs previously reported https://github.com/BD2KGenomics/toil/pull/1689TOIL_LSF_ARGS
to specify the queue in your runscript:export TOIL_LSF_ARGS="-q production-rh7"
https://github.com/BD2KGenomics/toil/pull/1640There's an error in enumerating jobs in Toil 3.7.0, fix is at https://github.com/BD2KGenomics/toil/pull/1690cwltool
's strict filename check, so be sure to strip out offending characters such as:
, example at https://github.com/ProteinsWebTeam/ebi-metagenomics-cwl/commit/767cc8f54cb26ad2b53c544c2e2054d8e7116a26 https://github.com/BD2KGenomics/toil/issues/1782cwltool
based CWL executors, you'll be happier if you set a dedicated output directory via--outdir
stdout
, so redirect that to a file for posterity (example:cwltoil […] | tee output
)--restart
is handy for resuming a previous run, but (due to the lack of cache support while using the LSF batch system) any changes to the CWL descriptions will require a clean startcat
get assigned 16 cores and 100GiB of memory :-)CWLWorkflow
,ResolveIndirect
,CWLGather
,CWLScatter
, etc.. ) https://github.com/BD2KGenomics/toil/issues/1783InitialWorkDirRequirment
and confirmCurrent working branch will the bulk of the above fixed merged: https://github.com/mr-c/toil/tree/issues/1666-fail-not-on-unsubmitted-jobsLatest Toil release has all of the above mentioned fixes merged