search

ICGC-TCGA-PanCancer / CGP-Somatic-Docker

Dockerised and simplified version of SeqWare-CGP-SomaticCore
GNU Affero General Public License v3.0
14 stars 9 forks source link

test data - 2 #5

Closed denis-yuen closed 8 years ago

denis-yuen commented 8 years ago

Currently attempting to execute the workflow with the test data on the branch https://github.com/ICGC-TCGA-PanCancer/CGP-Somatic-Docker/tree/feature/cwl1

Command is 

dockstore tool launch --entry Dockstore.cwl --local-entry --json test2.json

which becomes 

[job Dockstore.cwl] /tmp/tmp510v1N$ docker \                                                                                                                                                             [1536/1663]
    run \
    -i \
    --volume=/home/ubuntu/CGP-Somatic-Docker/./datastore/launcher-86c12006-c63e-4ebe-880f-f248b0e76b97/inputs/ef2387e1-00c2-497e-9bae-19d833273d70/9f8f6f03df6d100c90a761164800a42c.bam:/var/lib/cwl/stg86779b72-335
2-4adf-8478-98e3853b440f/9f8f6f03df6d100c90a761164800a42c.bam:ro \
    --volume=/home/ubuntu/CGP-Somatic-Docker/./datastore/launcher-86c12006-c63e-4ebe-880f-f248b0e76b97/inputs/196da89e-6d8a-49af-9ab2-70d302816e29/GRCh37d5_battenberg.tar.gz:/var/lib/cwl/stg6f32c8f1-139e-43cd-b94
7-427fdd812590/GRCh37d5_battenberg.tar.gz:ro \
    --volume=/home/ubuntu/CGP-Somatic-Docker/./datastore/launcher-86c12006-c63e-4ebe-880f-f248b0e76b97/inputs/ef2387e1-00c2-497e-9bae-19d833273d70/9f8f6f03df6d100c90a761164800a42c.bam.bai:/var/lib/cwl/stg86779b72
-3352-4adf-8478-98e3853b440f/9f8f6f03df6d100c90a761164800a42c.bam.bai:ro \
    --volume=/home/ubuntu/CGP-Somatic-Docker/./datastore/launcher-86c12006-c63e-4ebe-880f-f248b0e76b97/inputs/ca7276bf-10fa-45b8-9634-1b6d04270b99/ad4aeedc224c6be5e21e26c038a43df4.bam.bai:/var/lib/cwl/stg78f59412
-5cf5-4473-b80e-1720be3192c4/ad4aeedc224c6be5e21e26c038a43df4.bam.bai:ro \
    --volume=/home/ubuntu/CGP-Somatic-Docker/./datastore/launcher-86c12006-c63e-4ebe-880f-f248b0e76b97/inputs/ca7276bf-10fa-45b8-9634-1b6d04270b99/ad4aeedc224c6be5e21e26c038a43df4.bam:/var/lib/cwl/stg78f59412-5cf
5-4473-b80e-1720be3192c4/ad4aeedc224c6be5e21e26c038a43df4.bam:ro \
    --volume=/home/ubuntu/CGP-Somatic-Docker/./datastore/launcher-86c12006-c63e-4ebe-880f-f248b0e76b97/inputs/e1b2a106-3fe8-4e2b-9159-33c5ce9c95d6/GRCh37d5_CGP_refBundle.tar.gz:/var/lib/cwl/stg4d74f530-1cd7-4de3-
835f-7de7820a2dab/GRCh37d5_CGP_refBundle.tar.gz:ro \
    --volume=/tmp/tmp510v1N:/var/spool/cwl:rw \
    --volume=/home/ubuntu/CGP-Somatic-Docker/datastore/launcher-86c12006-c63e-4ebe-880f-f248b0e76b97/workinguyrgqy:/tmp:rw \
    --workdir=/var/spool/cwl \
    --read-only=true \
    --user=1000 \
    --rm \
    --env=TMPDIR=/tmp \
    --env=HOME=/var/spool/cwl \
    quay.io/pancancer/pcawg-sanger-cgp-workflow:feature_cwl1 \
    python \
    /home/seqware/CGP-Somatic-Docker/scripts/run_seqware_workflow.py \
    --tumor \
    /var/lib/cwl/stg78f59412-5cf5-4473-b80e-1720be3192c4/ad4aeedc224c6be5e21e26c038a43df4.bam \
    --normal \
    /var/lib/cwl/stg86779b72-3352-4adf-8478-98e3853b440f/9f8f6f03df6d100c90a761164800a42c.bam \
    --refFrom \
    /var/lib/cwl/stg4d74f530-1cd7-4de3-835f-7de7820a2dab/GRCh37d5_CGP_refBundle.tar.gz \
    --bbFrom \
    /var/lib/cwl/stg6f32c8f1-139e-43cd-b947-427fdd812590/GRCh37d5_battenberg.tar.gz

Looking at https://github.com/ICGC-TCGA-PanCancer/CGP-Somatic-Docker/blob/develop/Dockstore_cwl.json

corresponds to

https://github.com/ICGC-TCGA-PanCancer/CGP-Somatic-Docker/blob/feature/cwl1/test2.json

Currently fails with the following messages

[2016/09/02 11:59:54] | Row #7 , Attempt #1 out of 1 : s58_ASCAT_ascat_62,s58_BRASS_cover_101,s58_cgpPindel_pindel_71,s58_renameSampleFile_213,s58_renameSampleFile_214
[2016/09/02 11:59:54] |         Submitting 26480M batch with: s58_ASCAT_ascat_62,s58_BRASS_cover_101,s58_cgpPindel_pindel_71,s58_renameSampleFile_213,s58_renameSampleFile_214
[2016/09/02 11:59:54] |         Running command: bash /datastore/oozie-6ddf193c-472f-42ee-a74a-6cc6ce66e7f2/generated-scripts/s58_renameSampleFile_213-runner.sh
[2016/09/02 11:59:54] |         Running command: bash /datastore/oozie-6ddf193c-472f-42ee-a74a-6cc6ce66e7f2/generated-scripts/s58_BRASS_cover_101-runner.sh
[2016/09/02 11:59:54] |         Running command: bash /datastore/oozie-6ddf193c-472f-42ee-a74a-6cc6ce66e7f2/generated-scripts/s58_renameSampleFile_214-runner.sh
[2016/09/02 11:59:54] |         Running command: bash /datastore/oozie-6ddf193c-472f-42ee-a74a-6cc6ce66e7f2/generated-scripts/s58_ASCAT_ascat_62-runner.sh
[2016/09/02 11:59:54] |         Running command: bash /datastore/oozie-6ddf193c-472f-42ee-a74a-6cc6ce66e7f2/generated-scripts/s58_cgpPindel_pindel_71-runner.sh
[2016/09/02 11:59:59] |         Workflow step succeeded: s58_renameSampleFile_214
[2016/09/02 12:00:15] |         Workflow step succeeded: s58_renameSampleFile_213
[2016/09/02 12:38:13] |         Workflow step failed: s58_ASCAT_ascat_62
[2016/09/02 13:37:31] |         Workflow step succeeded: s58_BRASS_cover_101
[2016/09/02 15:22:18] |         Workflow step succeeded: s58_cgpPindel_pindel_71
[2016/09/02 15:22:18] | Setting workflow-run status to failed for: 58
[--plugin, io.seqware.pipeline.plugins.WorkflowWatcher, --, --workflow-run-accession, 58]
Found local reference archive: /var/lib/cwl/stg4d74f530-1cd7-4de3-835f-7de7820a2dab/GRCh37d5_CGP_refBundle.tar.gz
Found local reference archive: /var/lib/cwl/stg6f32c8f1-139e-43cd-b947-427fdd812590/GRCh37d5_battenberg.tar.gz
Workflow run 58 is currently failed

Job Name: s58_ASCAT_ascat_62
Job ID:   817594229
File:     /datastore/oozie-6ddf193c-472f-42ee-a74a-6cc6ce66e7f2/generated-scripts/s58_ASCAT_ascat_62.o817594229
Updated:  Fri Sep 02 12:38:13 UTC 2016
Contents Excerpt:
Calls: ascat.plotRawData -> png
In addition: Warning message:
In png(filename = paste(ASCATobj$samples[i], ".tumour.png", sep = ""),  :
  unable to open connection to X11 display ''
Execution halted
"cd /var/spool/cwl/0/ascat/tmpAscat/ascat; /opt/wtsi-cgp/bin/Rscript /opt/wtsi-cgp/lib/perl5/auto/share/module/Sanger-CGP-Ascat-Implement/ascat/runASCAT.R /opt/wtsi-cgp/lib/perl5/auto/share/module/Sanger-CGP-Asca
t-Implement/ascat /var/spool/cwl/reference_files/ascat/SnpPositions.tsv /var/spool/cwl/reference_files/ascat/SnpGcCorrections.tsv fc969f27-bdc3-cd03-e040-11ac0c48577e fc969f27-bdc3-cd03-e040-11ac0c48577e.count fc
969f27-bdc6-cd03-e040-11ac0c48577e fc969f27-bdc6-cd03-e040-11ac0c48577e.count XX /var/spool/cwl/0/ascat/tmpAscat/ascat/fc969f27-bdc3-cd03-e040-11ac0c48577e.Rdata" unexpectedly returned exit value 1 at (eval 294) 
line 13.
 at /opt/wtsi-cgp/lib/perl5/PCAP/Threaded.pm line 190
Command exited with non-zero status 2
597.47user 7.46system 38:15.68elapsed 26%CPU (0avgtext+0avgdata 6048784maxresident)k
662816inputs+209168outputs (130major+435707minor)pagefaults 0swaps
Calls: ascat.plotRawData -> png
In addition: Warning message:
In png(filename = paste(ASCATobj$samples[i], ".tumour.png", sep = ""),  :
  unable to open connection to X11 display ''
Execution halted
"cd /var/spool/cwl/0/ascat/tmpAscat/ascat; /opt/wtsi-cgp/bin/Rscript /opt/wtsi-cgp/lib/perl5/auto/share/module/Sanger-CGP-Ascat-Implement/ascat/runASCAT.R /opt/wtsi-cgp/lib/perl5/auto/share/module/Sanger-CGP-Asca
t-Implement/ascat /var/spool/cwl/reference_files/ascat/SnpPositions.tsv /var/spool/cwl/reference_files/ascat/SnpGcCorrections.tsv fc969f27-bdc3-cd03-e040-11ac0c48577e fc969f27-bdc3-cd03-e040-11ac0c48577e.count fc
969f27-bdc6-cd03-e040-11ac0c48577e fc969f27-bdc6-cd03-e040-11ac0c48577e.count XX /var/spool/cwl/0/ascat/tmpAscat/ascat/fc969f27-bdc3-cd03-e040-11ac0c48577e.Rdata" unexpectedly returned exit value 1 at (eval 294) 
line 13.
 at /opt/wtsi-cgp/lib/perl5/PCAP/Threaded.pm line 190
Command exited with non-zero status 2
597.47user 7.46system 38:15.68elapsed 26%CPU (0avgtext+0avgdata 6048784maxresident)k
662816inputs+209168outputs (130major+435707minor)pagefaults 0swaps
The method 'do_run' exited abnormally so the Runner will terminate here!
Return value was: 1

Full output: /datastore/oozie-6ddf193c-472f-42ee-a74a-6cc6ce66e7f2/generated-scripts/s58_ASCAT_ascat_62.stdout
denis-yuen commented 8 years ago

Warning, this takes around 24 hours to run to failure and a pretty beefy machine

adamstruck commented 8 years ago

I think I have encountered this error before when I was developing the WDL version... It occurred when the .Rprofile file wasn't being loaded by R because I had changed the home directory.

denis-yuen commented 8 years ago

I wonder if this means the --user=1000 parameter that cwltool adds to the docker run invocation is messing with something. I'll poke around.

adamstruck commented 8 years ago

what was the resolution to the issue?

denis-yuen commented 8 years ago

It turns out cwltool replaces the contents of the working directory when running a docker container. Since that is where the .Rprofile file lives, it gets wiped out resulting in the error above. So we simply create that file anew in the working directory which seems to do the trick. Using a bigger and beefier machine handled the other issue which I suspect was just an out of space error that one of the underlying tools reacted to as a division by zero error.

Otherwise, the rest of the changes are just minor syntax changes for CWL 1.0.