search

ICGC-TCGA-PanCancer / DEWrapperWorkflow

This is a workflow which aggregates the results of the EMBL and DKFZ workflows as one SeqWare workflow
0 stars 1 forks source link

cannot create directory (permission denied) #14

Open manuelgarvan opened 8 years ago

manuelgarvan commented 8 years ago

Hi, I would like to run this workflow using changin the /datastore and /workflow folders to /nvme/datastore and /nvme/workflow on my host.

This is the change I did on the workflow.ini file: common_data_dir=/nvme/datastore/DEWorkflowData

and this is how I call the workflow:

sudo docker run --rm -h master -it -v /var/run/docker.sock:/var/run/docker.sock \
                -v /nvme/datastore:/datastore \
                -v /home/garvan/workflow.ini:/workflow.ini \
                -v /home/garvan/.ssh/cghub_public.key:/home/ubuntu/.ssh/cghub_public.key \
                pancancer/pcawg-dewrapper-workflow:1.0.7 \
                seqware bundle launch --dir /workflow/Workflow_Bundle_DEWrapperWorkflow_1.0.7_SeqWare_1.1.1 --engine whitestar --no-metadata --ini /workflow.ini

This is the error I am getting:

[2016/07/15 00:39:08] | Row #1 , Attempt #1 out of 6 : s62_create_dirs_1
[2016/07/15 00:39:08] |         Submitting 8192M batch with: s62_create_dirs_1
[2016/07/15 00:39:08] |         Running command: bash /datastore/oozie-b9ebe0ac-9d66-48b3-9d00-42f139b07a9c/generated-scripts/s62_create_dirs_1-runner.sh
[2016/07/15 00:39:08] |         Workflow step failed: s62_create_dirs_1
[2016/07/15 00:39:08] | Row #1 , Attempt #2 out of 6 : s62_create_dirs_1
[2016/07/15 00:39:08] |         Submitting 8192M batch with: s62_create_dirs_1
[2016/07/15 00:39:08] |         Running command: bash /datastore/oozie-b9ebe0ac-9d66-48b3-9d00-42f139b07a9c/generated-scripts/s62_create_dirs_1-runner.sh
[2016/07/15 00:39:08] |         Workflow step failed: s62_create_dirs_1
[2016/07/15 00:39:08] | Row #1 , Attempt #3 out of 6 : s62_create_dirs_1
[2016/07/15 00:39:08] |         Submitting 8192M batch with: s62_create_dirs_1
[2016/07/15 00:39:08] |         Running command: bash /datastore/oozie-b9ebe0ac-9d66-48b3-9d00-42f139b07a9c/generated-scripts/s62_create_dirs_1-runner.sh
[2016/07/15 00:39:09] |         Workflow step failed: s62_create_dirs_1
[2016/07/15 00:39:09] | Row #1 , Attempt #4 out of 6 : s62_create_dirs_1
[2016/07/15 00:39:09] |         Submitting 8192M batch with: s62_create_dirs_1
[2016/07/15 00:39:09] |         Running command: bash /datastore/oozie-b9ebe0ac-9d66-48b3-9d00-42f139b07a9c/generated-scripts/s62_create_dirs_1-runner.sh
[2016/07/15 00:39:09] |         Workflow step failed: s62_create_dirs_1
[2016/07/15 00:39:09] | Row #1 , Attempt #5 out of 6 : s62_create_dirs_1
[2016/07/15 00:39:09] |         Submitting 8192M batch with: s62_create_dirs_1
[2016/07/15 00:39:09] |         Running command: bash /datastore/oozie-b9ebe0ac-9d66-48b3-9d00-42f139b07a9c/generated-scripts/s62_create_dirs_1-runner.sh
[2016/07/15 00:39:09] |         Workflow step failed: s62_create_dirs_1
[2016/07/15 00:39:09] | Row #1 , Attempt #6 out of 6 : s62_create_dirs_1
[2016/07/15 00:39:09] |         Submitting 8192M batch with: s62_create_dirs_1
[2016/07/15 00:39:09] |         Running command: bash /datastore/oozie-b9ebe0ac-9d66-48b3-9d00-42f139b07a9c/generated-scripts/s62_create_dirs_1-runner.sh
[2016/07/15 00:39:09] |         Workflow step failed: s62_create_dirs_1
[2016/07/15 00:39:09] | Setting workflow-run status to failed for: 62

...

-----------------------------------------------------------------------
Job Name: s62_create_dirs_1
Job ID:   543148357
File:     /datastore/oozie-b9ebe0ac-9d66-48b3-9d00-42f139b07a9c/generated-scripts/s62_create_dirs_1.o543148357
Updated:  Fri Jul 15 00:39:08 UTC 2016
Contents Excerpt:
mkdir: cannot create directory `/nvme': Permission denied
mkdir: cannot create directory `/nvme': Permission denied
The method 'do_run' exited abnormally so the Runner will terminate here!
Return value was: 1

Full output: /datastore/oozie-b9ebe0ac-9d66-48b3-9d00-42f139b07a9c/generated-scripts/s62_create_dirs_1.stdout
-----------------------------------------------------------------------

-----------------------------------------------------------------------
Job Name: s62_create_dirs_1
Job ID:   543148618
File:     /datastore/oozie-b9ebe0ac-9d66-48b3-9d00-42f139b07a9c/generated-scripts/s62_create_dirs_1.o543148618
Updated:  Fri Jul 15 00:39:08 UTC 2016
Contents Excerpt:
mkdir: cannot create directory `/nvme': Permission denied
mkdir: cannot create directory `/nvme': Permission denied
The method 'do_run' exited abnormally so the Runner will terminate here!
Return value was: 1

Full output: /datastore/oozie-b9ebe0ac-9d66-48b3-9d00-42f139b07a9c/generated-scripts/s62_create_dirs_1.stdout
-----------------------------------------------------------------------

-----------------------------------------------------------------------
Job Name: s62_create_dirs_1
Job ID:   543148871
File:     /datastore/oozie-b9ebe0ac-9d66-48b3-9d00-42f139b07a9c/generated-scripts/s62_create_dirs_1.o543148871
Updated:  Fri Jul 15 00:39:09 UTC 2016
Contents Excerpt:
mkdir: cannot create directory `/nvme': Permission denied
mkdir: cannot create directory `/nvme': Permission denied
The method 'do_run' exited abnormally so the Runner will terminate here!
Return value was: 1

Full output: /datastore/oozie-b9ebe0ac-9d66-48b3-9d00-42f139b07a9c/generated-scripts/s62_create_dirs_1.stdout
-----------------------------------------------------------------------

-----------------------------------------------------------------------
Job Name: s62_create_dirs_1
Job ID:   543149123
File:     /datastore/oozie-b9ebe0ac-9d66-48b3-9d00-42f139b07a9c/generated-scripts/s62_create_dirs_1.o543149123
Updated:  Fri Jul 15 00:39:09 UTC 2016
Contents Excerpt:
mkdir: cannot create directory `/nvme': Permission denied
mkdir: cannot create directory `/nvme': Permission denied
The method 'do_run' exited abnormally so the Runner will terminate here!
Return value was: 1

Full output: /datastore/oozie-b9ebe0ac-9d66-48b3-9d00-42f139b07a9c/generated-scripts/s62_create_dirs_1.stdout
-----------------------------------------------------------------------

-----------------------------------------------------------------------
Job Name: s62_create_dirs_1
Job ID:   543149372
File:     /datastore/oozie-b9ebe0ac-9d66-48b3-9d00-42f139b07a9c/generated-scripts/s62_create_dirs_1.o543149372
Updated:  Fri Jul 15 00:39:09 UTC 2016
Contents Excerpt:
mkdir: cannot create directory `/nvme': Permission denied
mkdir: cannot create directory `/nvme': Permission denied
The method 'do_run' exited abnormally so the Runner will terminate here!
Return value was: 1

Full output: /datastore/oozie-b9ebe0ac-9d66-48b3-9d00-42f139b07a9c/generated-scripts/s62_create_dirs_1.stdout
-----------------------------------------------------------------------

-----------------------------------------------------------------------
Job Name: s62_create_dirs_1
Job ID:   543149624
File:     /datastore/oozie-b9ebe0ac-9d66-48b3-9d00-42f139b07a9c/generated-scripts/s62_create_dirs_1.o543149624
Updated:  Fri Jul 15 00:39:09 UTC 2016
Contents Excerpt:
mkdir: cannot create directory `/nvme': Permission denied
mkdir: cannot create directory `/nvme': Permission denied
The method 'do_run' exited abnormally so the Runner will terminate here!
Return value was: 1

Full output: /datastore/oozie-b9ebe0ac-9d66-48b3-9d00-42f139b07a9c/generated-scripts/s62_create_dirs_1.stdout
-----------------------------------------------------------------------

Also, this is the content of my /nvme/datastore/oozie-b9ebe0ac-9d66-48b3-9d00-42f139b07a9c/generated-scripts/s62_create_dirs_1.sh

#!/usr/bin/env bash
set -o errexit
set -o pipefail

export SEQWARE_SETTINGS=/home/seqware/.seqware/settings
cd /datastore/oozie-b9ebe0ac-9d66-48b3-9d00-42f139b07a9c
mkdir -m 0777 -p shared_workspace
 mkdir -m 0777 -p shared_workspace/settings
 mkdir -m 0777 -p shared_workspace/results
 mkdir -m 0777 -p shared_workspace/working
 mkdir -m 0777 -p shared_workspace/downloads/dkfz
 mkdir -m 0777 -p shared_workspace/downloads/embl
 mkdir -m 0777 -p shared_workspace/inputs
 mkdir -m 0777 -p shared_workspace/testdata
 mkdir -m 0777 -p shared_workspace/uploads
 mkdir -m 0777 -p shared_workspace/data
 mkdir -m 0777 -p /nvme/datastore/DEWorkflowData/dkfz
 mkdir -m 0777 -p /nvme/datastore/DEWorkflowData/embl

thanks

denis-yuen commented 8 years ago

Hi, It looks like you might have the mount reversed or something. -v /nvme/datastore:/datastore \ -v /home/garvan/workflow.ini:/workflow.ini \ means that you're mounting /nvme/datastore on your computer to /datastore inside the container.

However common_data_dir=/nvme/datastore/DEWorkflowData means that you want to use /nvme/datastore inside the container, which doesn't exist which is why you're getting a permission issue.

Out of curiosity, are you with the PCAWG Docker Testing Working Group? Or are you working with the Garvan Institute in Australia?

manuelgarvan commented 8 years ago

Hi Denis,

I am working with the Garvan Institute in Australia, we are collaborating with David Bowtell for the CASCADE project. We are testing different pipelines like GMS and yours which looks like quite promising.

Thank you

Manuel Sopena Ballesteros | Big data Engineer Garvan Institute of Medical Research The Kinghorn Cancer Centre, 370 Victoria Street, Darlinghurst, NSW 2010 T: + 61 (0)2 9355 5760 | F: +61 (0)2 9295 8507 | E: manuel.sb@garvan.org.aumailto:manuel.sb@garvan.org.au

From: Denis Yuen [mailto:notifications@github.com] Sent: Saturday, July 16, 2016 4:58 AM To: ICGC-TCGA-PanCancer/DEWrapperWorkflow Cc: Manuel Sopena Ballesteros; Author Subject: Re: [ICGC-TCGA-PanCancer/DEWrapperWorkflow] cannot create directory (permission denied) (#14)

Hi, It looks like you might have the mount reversed or something. -v /nvme/datastore:/datastore \ -v /home/garvan/workflow.ini:/workflow.ini \ means that you're mounting /nvme/datastore on your computer to /datastore inside the container.

However common_data_dir=/nvme/datastore/DEWorkflowData means that you want to use /nvme/datastore inside the container, which doesn't exist which is why you're getting a permission issue.

Out of curiosity, are you with the PCAWG Docker Testing Working Group? Or are you working with the Garvan Institute in Australia?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/ICGC-TCGA-PanCancer/DEWrapperWorkflow/issues/14#issuecomment-233038234, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ASj6ElsybTXlrvKBSvCjQQRWpWFj0gwwks5qV9gwgaJpZM4JM_BH.

NOTICE Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.