search

mskcc / roslin-cwl

CWLs for the Roslin Variant Pipeline
0 stars 1 forks source link

Change facets_snps in db_files from string to File; are there others? #4

Closed allanbolipata closed 5 years ago

allanbolipata commented 5 years ago

Ran into the following error:

2806268 K/m/jobAZ316L    [job snp-pileup.cwl] /scratch/tmp8urKXE/3/1/out_tmpdir1pZ54X$ singularity \$
2806269 K/m/jobAZ316L        --quiet \$
2806270 K/m/jobAZ316L        exec \$
2806271 K/m/jobAZ316L        --contain \$
2806272 K/m/jobAZ316L        --pid \$
2806273 K/m/jobAZ316L        --ipc \$
2806274 K/m/jobAZ316L        --bind \$
2806275 K/m/jobAZ316L        /scratch/tmp8urKXE/3/1/out_tmpdir1pZ54X:/wssxwE:rw \$
2806276 K/m/jobAZ316L        --bind \$
2806277 K/m/jobAZ316L        /juno/work/pi/prototypes/mskcc-cwl/test_runs/tmp:/tmp:rw \$
2806278 K/m/jobAZ316L        --bind \$
2806279 K/m/jobAZ316L        /juno/work/pi/prototypes/mskcc-cwl/test_runs/work/toil-e280d616-b743-497b-a4cf-709cdb1b3c2f-43458aa4-edce-4677-8620-a3137eede36a/tmpj0dGED/86f443bb-e090-46fa-a7c9-db9114612517/tmpfh_Mz2.tmp:/var/lib/cwl/stg67b41dc9-15a6-4261-9e7c-177bade05d55/        bar_normal.rg.md.abra.printreads.bam:ro \$
2806280 K/m/jobAZ316L        --bind \$
2806281 K/m/jobAZ316L        /juno/work/pi/prototypes/mskcc-cwl/test_runs/work/toil-e280d616-b743-497b-a4cf-709cdb1b3c2f-43458aa4-edce-4677-8620-a3137eede36a/tmpj0dGED/86f443bb-e090-46fa-a7c9-db9114612517/tmpze7Is4.tmp:/var/lib/cwl/stg67b41dc9-15a6-4261-9e7c-177bade05d55/        bar_normal.rg.md.abra.printreads.bai:ro \$
2806282 K/m/jobAZ316L        --bind \$
2806283 K/m/jobAZ316L        /juno/work/pi/prototypes/mskcc-cwl/test_runs/work/toil-e280d616-b743-497b-a4cf-709cdb1b3c2f-43458aa4-edce-4677-8620-a3137eede36a/tmpj0dGED/86f443bb-e090-46fa-a7c9-db9114612517/tmpwDAfZt.tmp:/var/lib/cwl/stg67b41dc9-15a6-4261-9e7c-177bade05d55/        bar_normal.rg.md.abra.printreads.bam.bai:ro \$
2806284 K/m/jobAZ316L        --bind \$
2806285 K/m/jobAZ316L        /juno/work/pi/prototypes/mskcc-cwl/test_runs/work/toil-e280d616-b743-497b-a4cf-709cdb1b3c2f-43458aa4-edce-4677-8620-a3137eede36a/tmpj0dGED/86f443bb-e090-46fa-a7c9-db9114612517/tmpe3NONG.tmp:/var/lib/cwl/stge6e91920-f422-4167-8def-142c27399185/        bar_tumor.rg.md.abra.printreads.bai:ro \$
2806286 K/m/jobAZ316L        --bind \$
2806287 K/m/jobAZ316L        /juno/work/pi/prototypes/mskcc-cwl/test_runs/work/toil-e280d616-b743-497b-a4cf-709cdb1b3c2f-43458aa4-edce-4677-8620-a3137eede36a/tmpj0dGED/86f443bb-e090-46fa-a7c9-db9114612517/tmpVBXJgP.tmp:/var/lib/cwl/stge6e91920-f422-4167-8def-142c27399185/        bar_tumor.rg.md.abra.printreads.bam.bai:ro \$
2806288 K/m/jobAZ316L        --bind \$
2806289 K/m/jobAZ316L        /juno/work/pi/prototypes/mskcc-cwl/test_runs/work/toil-e280d616-b743-497b-a4cf-709cdb1b3c2f-43458aa4-edce-4677-8620-a3137eede36a/tmpj0dGED/86f443bb-e090-46fa-a7c9-db9114612517/tmp8kyNMy.tmp:/var/lib/cwl/stge6e91920-f422-4167-8def-142c27399185/        bar_tumor.rg.md.abra.printreads.bam:ro \$
2806290 K/m/jobAZ316L        --pwd \$
2806291 K/m/jobAZ316L        /wssxwE \$
2806292 K/m/jobAZ316L        /juno/work/pi/prototypes/mskcc-cwl/test_runs/work/toil-e280d616-b743-497b-a4cf-709cdb1b3c2f-43458aa4-edce-4677-8620-a3137eede36a/tmpj0dGED/86f443bb-e090-46fa-a7c9-db9114612517/mskcc-roslin-variant-htstools-0.1.1.img \$
2806293 K/m/jobAZ316L        /usr/bin/snp-pileup \$
2806294 K/m/jobAZ316L        --count-orphans \$
2806295 K/m/jobAZ316L        --gzip \$
2806296 K/m/jobAZ316L        --min-read-counts \$
2806297 K/m/jobAZ316L        10,0 \$
2806298 K/m/jobAZ316L        --pseudo-snps \$
2806299 K/m/jobAZ316L        50 \$
2806300 K/m/jobAZ316L        /juno/work/ci/resources/genomes/GRCh37/facets_snps/dbsnp_137.b37__RmDupsClean__plusPseudo50__DROP_SORT.vcf.gz \$
2806301 K/m/jobAZ316L        bar_normal.rg.md.abra.printreads__bar_tumor.rg.md.abra.printreads.dat.gz \$
2806302 K/m/jobAZ316L        /var/lib/cwl/stg67b41dc9-15a6-4261-9e7c-177bade05d55/bar_normal.rg.md.abra.printreads.bam \$
2806303 K/m/jobAZ316L        /var/lib/cwl/stge6e91920-f422-4167-8def-142c27399185/bar_tumor.rg.md.abra.printreads.bam$
2806304 K/m/jobAZ316L    INFO:cwltool:[job snp-pileup.cwl] /scratch/tmp8urKXE/3/1/out_tmpdir1pZ54X$ singularity \$
2806305 K/m/jobAZ316L        --quiet \$
2806306 K/m/jobAZ316L        exec \$
2806307 K/m/jobAZ316L        --contain \$
2806308 K/m/jobAZ316L        --pid \$
2806309 K/m/jobAZ316L        --ipc \$
2806310 K/m/jobAZ316L        --bind \$
2806311 K/m/jobAZ316L        /scratch/tmp8urKXE/3/1/out_tmpdir1pZ54X:/wssxwE:rw \$
                                                                                                                                                                                                                                                              2806282,1     99%
2806310 K/m/jobAZ316L        --bind \$
2806311 K/m/jobAZ316L        /scratch/tmp8urKXE/3/1/out_tmpdir1pZ54X:/wssxwE:rw \$
2806312 K/m/jobAZ316L        --bind \$
2806313 K/m/jobAZ316L        /juno/work/pi/prototypes/mskcc-cwl/test_runs/tmp:/tmp:rw \$
2806314 K/m/jobAZ316L        --bind \$
2806315 K/m/jobAZ316L        /juno/work/pi/prototypes/mskcc-cwl/test_runs/work/toil-e280d616-b743-497b-a4cf-709cdb1b3c2f-43458aa4-edce-4677-8620-a3137eede36a/tmpj0dGED/86f443bb-e090-46fa-a7c9-db9114612517/tmpfh_Mz2.tmp:/var/lib/cwl/stg67b41dc9-15a6-4261-9e7c-177bade05d55/        bar_normal.rg.md.abra.printreads.bam:ro \$
2806316 K/m/jobAZ316L        --bind \$
2806317 K/m/jobAZ316L        /juno/work/pi/prototypes/mskcc-cwl/test_runs/work/toil-e280d616-b743-497b-a4cf-709cdb1b3c2f-43458aa4-edce-4677-8620-a3137eede36a/tmpj0dGED/86f443bb-e090-46fa-a7c9-db9114612517/tmpze7Is4.tmp:/var/lib/cwl/stg67b41dc9-15a6-4261-9e7c-177bade05d55/        bar_normal.rg.md.abra.printreads.bai:ro \$
2806318 K/m/jobAZ316L        --bind \$
2806319 K/m/jobAZ316L        /juno/work/pi/prototypes/mskcc-cwl/test_runs/work/toil-e280d616-b743-497b-a4cf-709cdb1b3c2f-43458aa4-edce-4677-8620-a3137eede36a/tmpj0dGED/86f443bb-e090-46fa-a7c9-db9114612517/tmpwDAfZt.tmp:/var/lib/cwl/stg67b41dc9-15a6-4261-9e7c-177bade05d55/        bar_normal.rg.md.abra.printreads.bam.bai:ro \$
2806320 K/m/jobAZ316L        --bind \$
2806321 K/m/jobAZ316L        /juno/work/pi/prototypes/mskcc-cwl/test_runs/work/toil-e280d616-b743-497b-a4cf-709cdb1b3c2f-43458aa4-edce-4677-8620-a3137eede36a/tmpj0dGED/86f443bb-e090-46fa-a7c9-db9114612517/tmpe3NONG.tmp:/var/lib/cwl/stge6e91920-f422-4167-8def-142c27399185/        bar_tumor.rg.md.abra.printreads.bai:ro \$
2806322 K/m/jobAZ316L        --bind \$
2806323 K/m/jobAZ316L        /juno/work/pi/prototypes/mskcc-cwl/test_runs/work/toil-e280d616-b743-497b-a4cf-709cdb1b3c2f-43458aa4-edce-4677-8620-a3137eede36a/tmpj0dGED/86f443bb-e090-46fa-a7c9-db9114612517/tmpVBXJgP.tmp:/var/lib/cwl/stge6e91920-f422-4167-8def-142c27399185/        bar_tumor.rg.md.abra.printreads.bam.bai:ro \$
2806324 K/m/jobAZ316L        --bind \$
2806325 K/m/jobAZ316L        /juno/work/pi/prototypes/mskcc-cwl/test_runs/work/toil-e280d616-b743-497b-a4cf-709cdb1b3c2f-43458aa4-edce-4677-8620-a3137eede36a/tmpj0dGED/86f443bb-e090-46fa-a7c9-db9114612517/tmp8kyNMy.tmp:/var/lib/cwl/stge6e91920-f422-4167-8def-142c27399185/        bar_tumor.rg.md.abra.printreads.bam:ro \$
2806326 K/m/jobAZ316L        --pwd \$
2806327 K/m/jobAZ316L        /wssxwE \$
2806328 K/m/jobAZ316L        /juno/work/pi/prototypes/mskcc-cwl/test_runs/work/toil-e280d616-b743-497b-a4cf-709cdb1b3c2f-43458aa4-edce-4677-8620-a3137eede36a/tmpj0dGED/86f443bb-e090-46fa-a7c9-db9114612517/mskcc-roslin-variant-htstools-0.1.1.img \$
2806329 K/m/jobAZ316L        /usr/bin/snp-pileup \$
2806330 K/m/jobAZ316L        --count-orphans \$
2806331 K/m/jobAZ316L        --gzip \$
2806332 K/m/jobAZ316L        --min-read-counts \$
2806333 K/m/jobAZ316L        10,0 \$
2806334 K/m/jobAZ316L        --pseudo-snps \$
2806335 K/m/jobAZ316L        50 \$
2806336 K/m/jobAZ316L        /juno/work/ci/resources/genomes/GRCh37/facets_snps/dbsnp_137.b37__RmDupsClean__plusPseudo50__DROP_SORT.vcf.gz \$
2806337 K/m/jobAZ316L        bar_normal.rg.md.abra.printreads__bar_tumor.rg.md.abra.printreads.dat.gz \$
2806338 K/m/jobAZ316L        /var/lib/cwl/stg67b41dc9-15a6-4261-9e7c-177bade05d55/bar_normal.rg.md.abra.printreads.bam \$
2806339 K/m/jobAZ316L        /var/lib/cwl/stge6e91920-f422-4167-8def-142c27399185/bar_tumor.rg.md.abra.printreads.bam$
2806340 K/m/jobAZ316L    [E::hts_open_format] Failed to open file /juno/work/ci/resources/genomes/GRCh37/facets_snps/dbsnp_137.b37__RmDupsClean__plusPseudo50__DROP_SORT.vcf.gz$
2806341 K/m/jobAZ316L    Failed to read VCF file: No such file or directory$
2806342 K/m/jobAZ316L    [job snp-pileup.cwl] Job error:$
2806343 K/m/jobAZ316L    Error collecting output for parameter 'out_file':$
2806344 K/m/jobAZ316L    :1:1: Did not find output file with glob pattern: '[u'bar_normal.rg.md.abra.printreads__bar_tumor.rg.md.abra.printreads.dat.gz']'$
2806345 K/m/jobAZ316L    ERROR:cwltool:[job snp-pileup.cwl] Job error:$
2806346 K/m/jobAZ316L    Error collecting output for parameter 'out_file':$
2806347 K/m/jobAZ316L    :1:1: Did not find output file with glob pattern: '[u'bar_normal.rg.md.abra.printreads__bar_tumor.rg.md.abra.printreads.dat.gz']'$
2806348 K/m/jobAZ316L    [job snp-pileup.cwl] completed permanentFail$
2806349 K/m/jobAZ316L    WARNING:cwltool:[job snp-pileup.cwl] completed permanentFail$
2806350 K/m/jobAZ316L    Traceback (most recent call last):$
2806351 K/m/jobAZ316L      File "/juno/work/pi/prototypes/toil/src/toil/worker.py", line 324, in workerScript$
2806352 K/m/jobAZ316L        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore)$
2806353 K/m/jobAZ316L      File "/juno/work/pi/prototypes/toil/src/toil/job.py", line 1351, in _runner$
2806354 K/m/jobAZ316L        returnValues = self._run(jobGraph, fileStore)$
2806355 K/m/jobAZ316L      File "/juno/work/pi/prototypes/toil/src/toil/job.py", line 1296, in _run$
2806356 K/m/jobAZ316L        return self.run(fileStore)$
2806357 K/m/jobAZ316L      File "/juno/work/pi/prototypes/toil/src/toil/cwl/cwltoil.py", line 606, in run$
2806358 K/m/jobAZ316L        raise cwltool.errors.WorkflowException(status)$
2806359 K/m/jobAZ316L    WorkflowException: permanentFail$
2806360 K/m/jobAZ316L    ERROR:toil.worker:Exiting the worker because of a failed job on host jx21$
2806361 K/m/jobAZ316L    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'file:///juno/work/pi/prototypes/mskcc-cwl/roslin-cwl/tools/htstools/0.1.1/snp-pileup.cwl' /usr/bin/snp-pileup K/m/jobAZ316L with ID K/m/jobAZ316L to 3$
2806362 K/m/jobAZ316L    WARNING:toil.jobGraph:We have increased the default memory of the failed job 'file:///juno/work/pi/prototypes/mskcc-cwl/roslin-cwl/tools/htstools/0.1.1/snp-pileup.cwl' /usr/bin/snp-pileup K/m/jobAZ316L to 8589934592 bytes

See the CWL line here: https://github.com/mskcc/roslin-cwl/blob/master/workflows/project-workflow-sv.cwl#L23

It is likely because /juno/work/ci/resources/genomes/GRCh37/facets_snps/dbsnp_137.b37__RmDupsClean__plusPseudo50__DROP_SORT.vcf.gz is a string in db_files and doesn't get mapped properly in singularity; this is similar to an issue we ran into with java_temp.

Now the workaround is again binding those paths to singularity, but I would prefer we avoid that. That being said, we may need to take this opportunity to change all potentially problematic sections in the YAML-generation step to accommodate this needed change.

For example:

db_files:$
  bait_intervals: {class: File, path: /juno/work/ci/resources/roslin_resources/targets/IMPACT468/b37/picard_baits.interval_list}$
  conpair_markers: /usr/bin/conpair/data/markers/GRCh37.autosomes.phase3_shapeit2_mvncall_integrated.20130502.SNV.genotype.sselect_v4_MAF_0.4_LD_0.8.txt$
  conpair_markers_bed: /usr/bin/conpair/data/markers/GRCh37.autosomes.phase3_shapeit2_mvncall_integrated.20130502.SNV.genotype.sselect_v4_MAF_0.4_LD_0.8.bed$
  custom_enst: /usr/bin/vcf2maf/data/isoform_overrides_at_mskcc$
  delly_exclude: {class: File, path: /juno/work/ci/resources/genomes/GRCh37/delly/human.hg19.excl.tsv}$
  facets_snps: /juno/work/ci/resources/genomes/GRCh37/facets_snps/dbsnp_137.b37__RmDupsClean__plusPseudo50__DROP_SORT.vcf.gz$
  fp_genotypes: {class: File, path: /juno/work/ci/resources/roslin_resources/targets/IMPACT468/b37/FP_tiling_genotypes.txt}$
  fp_intervals: {class: File, path: /juno/work/ci/resources/roslin_resources/targets/IMPACT468/b37/FP_tiling_intervals.list}$
  grouping_file: {class: File, path: /juno/work/ci/roslin-test/variant/857/variant/2.6.0/workspace/roslin-3.10/examples/PairWorkflow/Proj_ROSLIN_DEV_sample_grouping.txt}$
  hotspot_list: /usr/bin/ngs-filters/data/hotspot-list-union-v1-v2.txt$
  hotspot_list_maf: {class: File, path: /juno/work/ci/resources/roslin-qc/hotspot-list-union-v1-v2.maf}$
  hotspot_vcf: /usr/bin/basicfiltering/data/hotspot-list-union-v1-v2.vcf$
  mapping_file: {class: File, path: /juno/work/ci/roslin-test/variant/857/variant/2.6.0/workspace/roslin-3.10/examples/PairWorkflow/Proj_ROSLIN_DEV_sample_mapping.txt}$
  pairing_file: {class: File, path: /juno/work/ci/roslin-test/variant/857/variant/2.6.0/workspace/roslin-3.10/examples/PairWorkflow/Proj_ROSLIN_DEV_sample_pairing.txt}$
  refseq: {class: File, path: /juno/work/ci/resources/request_files/refseq/refGene_b37.sorted.txt}$
  request_file: {class: File, path: /juno/work/ci/roslin-test/variant/857/variant/2.6.0/workspace/roslin-3.10/examples/PairWorkflow/Proj_ROSLIN_DEV_request.txt}$
  target_intervals: {class: File, path: /juno/work/ci/resources/roslin_resources/targets/IMPACT468/b37/picard_targets.interval_list}$
  vep_data: /juno/work/ci/resources/vep/cache$
  vep_path: /usr/bin/vep$

vep_data might need to be changed (maybe as a Directory?); vep_data, custom_enst, hotspot_list, hotspot_vcf, conpair_markers, and conpair_markers_bed are actually in their respective containers.

Just need to change!

allanbolipata commented 5 years ago

I'm working on this here https://github.com/mskcc/roslin-cwl/tree/hotfix/change_strings_to_file_or_dir