Testing of all the CWL's on JUNO with toil-cwl-runner and singularity with LSF

mskcc / cwl-commandlinetools

Central location for CWL CommandLineTools

Other

3 stars 3 forks source link

Testing of all the CWL's on JUNO with toil-cwl-runner and singularity with LSF #48

Closed rhshah closed 5 years ago

rhshah commented 5 years ago

We need to test the following CWLS:

[x] abra2_2.17
[x] marianas_process_loop_umi_1.8.0
[x] picard_add_or_replace_read_groups_1.96
[x] picard_fix_mate_information_1.96
[x] picard_mark_duplicates_2.8.1
[x] trim_galore_0.6.2
[x] waltz_count_reads_2.0.0
[x] waltz_pileupmatrices_2.0
[x] gatk_ApplyBQSR_4.1.2.0
[x] gatk_BaseRecalibrator_4.1.2.0
[x] bedtools_genomecov_v2.28.0_cv2
[x] bedtools_merge_v2.28.0_cv2
[x] picard_collect_alignment_summary_metrics
[x] marianas_seprate_bams_1.8.1

rhshah commented 5 years ago

Started test on: marianas_process_loop_umi_1.8.0 couple of observations:

We should label the files and folder as 1.8.1.
I added the new jar to mskcc/marianas:0.1.0

I was able to run using cwltool: Version:/home/shahr2/tools/system/miniconda/versions/4.6.14/envs/test_toil/bin/cwltool 1.0.20190228155703

cwltool --singularity --non-strict /home/shahr2/github/cwl-commandlinetools/marianas_process_loop_umi_1.8.0/marianas_process_loop_umi.cwl $PWD/process_loop_umi_inputs.yaml

But failed with toil-cwl-runner:

#Using toil-cwl-runner
toil-cwl-runner --logFile $PWD/processUMILoop_toil.log --jobStore $PWD/processUMILoop_jobstore --batchSystem lsf --workDir $PWD --outdir $PWD --writeLogs $PWD/processUMILoop_toil_log --logLevel DEBUG --stats --retryCount 2  --no-container --disableCaching --maxLogFileSize 20000000000  /home/shahr2/github/cwl-commandlinetools/marianas_process_loop_umi_1.8.0/marianas_process_loop_umi.cwl $PWD/process_loop_umi_inputs.yaml`

rhshah commented 5 years ago

Probably failed due to toil version=3.20.

Anyhow this command works:

mkdir $PWD/processUMILoop_toil_log 
toil-cwl-runner --singularity --logFile processUMILoop_toil.log --jobStore $PWD/processUMILoop_jobstore --batchSystem lsf --workDir $PWD/processUMILoop_toil_log --outdir $PWD --writeLogs $PWD/processUMILoop_toil_log --logLevel DEBUG --stats --retryCount 2 --disableCaching --maxLogFileSize 20000000000 /home/shahr2/github/cwl-commandlinetools/marianas_process_loop_umi_1.8.0/marianas_process_loop_umi.cwl $PWD/process_loop_umi_inputs.yaml > processUMILoop_toil.stdout 2> processUMILoop_toil.stderr

also need to modify ramMin, coreMin

rhshah commented 5 years ago

Couple of things that I noticed:

@shalabhsuman & @ionox0

Each of the cwls that are there in those folders we are not specifying the version name to the cwl.

For example:

marianas_process_loop_umi_1.8.1/marianas_process_loop_umi.cwl

Should be:

marianas_process_loop_umi_1.8.1/marianas_process_loop_umi_1.8.1.cwl

@ionox0 for the marianas first pass and second pass CWL we dont have the unbound memory parameters, can you please add them

rhshah commented 5 years ago

@shalabhsuman @ionox0 Another thing while writing readme can you please follow this : https://github.com/mskcc/cwl-commandlinetools/tree/feature/test_modify_cwls/picard_fix_mate_information_1.96 https://github.com/mskcc/cwl-commandlinetools/tree/feature/test_modify_cwls/picard_add_or_replace_read_groups_1.96