search

common-workflow-library / legacy

Deprecated
https://github.com/common-workflow-library/bio-cwl-tools
Apache License 2.0
100 stars 62 forks source link

permanentFail when running FestivalDemo with DockerRequirement #90

Closed cnjr2 closed 8 years ago

cnjr2 commented 8 years ago

I have had some trouble running workflows with more than one step on docker.

To check whether it was something specific to my workflow or the general setup I tried one of the example workflows, i.e. the FestivalDemo.

It does not complete the run:

Charless-MacBook-Pro:FestivalDemo charles$ cwl-runner filtercount.cwl filtercount-job.json
/Users/charles/anaconda/bin/cwl-runner 1.0.20160515155531
[job grep] /var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmpQX9usS$ docker run -i --volume=/Users/charles/Code/maker_pipeline/cwl/workflows/workflows/FestivalDemo/mock.txt:/var/lib/cwl/job110644079_FestivalDemo/mock.txt:ro --volume=/var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmpQX9usS:/var/spool/cwl:rw --volume=/var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmphe7ye0:/tmp:rw --workdir=/var/spool/cwl --read-only=true --log-driver=none --user=1000 --rm --env=TMPDIR=/tmp --env=HOME=/var/spool/cwl debian:8 grep one /var/lib/cwl/job110644079_FestivalDemo/mock.txt > /var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmpQX9usS/out.txt
[job grep_2] /var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmpy4K4tA$ docker run -i --volume=/Users/charles/Code/maker_pipeline/cwl/workflows/workflows/FestivalDemo/mock2.txt:/var/lib/cwl/job16496370_FestivalDemo/mock2.txt:ro --volume=/var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmpy4K4tA:/var/spool/cwl:rw --volume=/var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmp7ZsUXO:/tmp:rw --workdir=/var/spool/cwl --read-only=true --log-driver=none --user=1000 --rm --env=TMPDIR=/tmp --env=HOME=/var/spool/cwl debian:8 grep one /var/lib/cwl/job16496370_FestivalDemo/mock2.txt > /var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmpy4K4tA/out.txt
[step grep] completion status is success
[job wc] /var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmp6uq3g3$ docker run -i --volume=/var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmpQX9usS/out.txt:/var/lib/cwl/job202387973_tmpQX9usS/out.txt:ro --volume=/var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmpy4K4tA/out.txt:/var/lib/cwl/job202387973_tmpy4K4tA/out.txt:ro --volume=/var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmp6uq3g3:/var/spool/cwl:rw --volume=/var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmpR3nUEb:/tmp:rw --workdir=/var/spool/cwl --read-only=true --log-driver=none --user=1000 --rm --env=TMPDIR=/tmp --env=HOME=/var/spool/cwl debian:8 wc -l /var/lib/cwl/job202387973_tmpQX9usS/out.txt /var/lib/cwl/job202387973_tmpy4K4tA/out.txt > /var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmp6uq3g3/out.txt
wc: /var/lib/cwl/job202387973_tmpQX9usS/out.txt: Is a directory
wc: /var/lib/cwl/job202387973_tmpy4K4tA/out.txt: Is a directory
[job wc] completed permanentFail
[step wc] completion status is permanentFail
[workflow filtercount.cwl] outdir is /Users/charles/Code/maker_pipeline/cwl/workflows/workflows/FestivalDemo
Final process status is permanentFail
Workflow error, try again with --debug for more information:
  Process status is ['permanentFail']
Charless-MacBook-Pro:FestivalDemo charles$

Commenting out the DockerRequirement:

requirements:
  - class: ScatterFeatureRequirement
  # - class: DockerRequirement
  #   dockerPull: "debian:8"

it runs fine:

Charless-MacBook-Pro:FestivalDemo charles$ cwl-runner filtercount.cwl filtercount-job.json
/Users/charles/anaconda/bin/cwl-runner 1.0.20160515155531
[job grep] /var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmpfmMJyz$ grep one /Users/charles/Code/maker_pipeline/cwl/workflows/workflows/FestivalDemo/mock.txt > /var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmpfmMJyz/out.txt
[job grep_2] /var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmpShZ9OP$ grep one /Users/charles/Code/maker_pipeline/cwl/workflows/workflows/FestivalDemo/mock2.txt > /var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmpShZ9OP/out.txt
[step grep] completion status is success
[job wc] /var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmp5lRaDb$ wc -l /var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmpfmMJyz/out.txt /var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmpShZ9OP/out.txt > /var/folders/rk/jx0271ns4fq3ly5b82g4d91r0000gp/T/tmp5lRaDb/out.txt
[step wc] completion status is success
[workflow filtercount.cwl] outdir is /Users/charles/Code/maker_pipeline/cwl/workflows/workflows/FestivalDemo
Final process status is success
{
    "outfile": {
        "path": "/Users/charles/Code/maker_pipeline/cwl/workflows/workflows/FestivalDemo/out.txt", 
        "checksum": "sha1$a19515dda3782e30940944f90bf07a0ae56f3527", 
        "class": "File", 
        "size": 167
    }
}
Charless-MacBook-Pro:FestivalDemo charles$

I presume it has something to do with my docker configuration and file permissions?

I am on a Mac (El Capitan) and, running docker (1.11) and cwl-runner (1.0.20160515155531).

cnjr2 commented 8 years ago

All is well when running with the instructions for boot2docker from here:

cwl-runner --tmp-outdir-prefix=/Users/charles/project --tmpdir-prefix=/Users/charles/project filtercount.cwl filtercount-job.json