Can't 'add' files to a docker container

[gijzelaerr]

currently this doesnt work:

requirements:
  - class: DockerRequirement
    dockerFile: |
       FROM kernsuite/base:2
       RUN docker-apt-install casalite
       ADD casawrap.py /usr/local/bin

I'm trying to a add a file to the container. Probably we don't want to implicitly copy all kind of files to the tempfolder used for constructing the container, and it is probably als not a good idea to use the current work folder or the cwl base folder as a build directory for docker since then all files in those folders are copied to the docker engine. I think it maybe is a good idea to extent the DockerRequirement class with a for examplestage files directive the copies a list of specified files to the Docker build work folder so they can copied to inside the container.

[gijzelaerr]

You can also do a preprocessing run and examine the ADD and COPY statements.

[tetron]

I think we want something like this:

https://github.com/common-workflow-language/common-workflow-language/issues/294#issuecomment-242069193

[tetron]

So yes, it's an oversight that should be added.

[gijzelaerr]

@tetron but it is not possible now right? Get a file in the work build folder with a workaround?

[wdesouza]

I have the same problem because some of my Tools call custom scripts written in R. Here is minimal example.

Custom R script (hello.Rscript):

#!/usr/bin/env Rscript
print("Hello!")

Make script executable:

chmod a+x hello.Rscript

Tool-specific Dockerfile (hello-Dockerfile):

FROM r-base
COPY hello.Rscript /usr/local/bin/

Description of tool (hello.cwl):

cwlVersion: v1.0
class: CommandLineTool
baseCommand: [hello.Rscript]
stdout: output.txt
hints:
  DockerRequirement:
      dockerPull: r-base
      dockerFile: >
        $import: hello-Dockerfile

inputs: []

outputs:
  output:
    type: stdout

All files at the same directory. Run cwl-runner:

 cwl-runner hello.cwl

Error:

/usr/local/bin/cwl-runner 1.0.20170309164828
Resolved 'hello.cwl' to 'file:///home/welliton/hello.cwl'
invalid field `job_order`, expected one of:
[job hello.cwl] /tmp/tmpypRfgM$ docker \
    run \
    -i \
    --volume=/tmp/tmpypRfgM:/var/spool/cwl:rw \
    --volume=/tmp/tmpW_g39Y:/tmp:rw \
    --workdir=/var/spool/cwl \
    --read-only=true \
    --log-driver=none \
    --user=1001 \
    --rm \
    --env=TMPDIR=/tmp \
    --env=HOME=/var/spool/cwl \
    r-base \
    hello.Rscript > /tmp/tmpypRfgM/output.txt
container_linux.go:247: starting container process caused "exec: \"hello.Rscript\": executable file not found in $PATH"
docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \"hello.Rscript\": executable file not found in $PATH".
[job hello.cwl] completed permanentFail
{
    "output": {
        "checksum": "sha1$da39a3ee5e6b4b0d3255bfef95601890afd80709",
        "basename": "output.txt",
        "location": "file:///home/welliton/output.txt",
        "path": "/home/welliton/output.txt",
        "class": "File",
        "size": 0
    }
}
Final process status is permanentFail

A workaround is building Docker image of hello-Dockerfile and using this image at hello.cwl file. However I it is not a good solution.

[suecharo]

I also have problems with this. As a specific use case, files that can not be exported are required in the Docker image, but I have to write in CWL.

[tetron]

Unfortunately this won't be fixed in v1.1.

@suecharo @gijzelaerr As a workaround, you could implement an extension (called DockerExtraFiles in this example):

requirements:
  - class: DockerRequirement
    dockerFile: |
       FROM kernsuite/base:2
       RUN docker-apt-install casalite
       ADD casawrap.py /usr/local/bin
  - class:  gijzelaerr:DockerExtraFiles
    dockerExtraFiles:
       - class: File
         location: casawrap.py

[EricBoix]

Ooops, I just hit that snag. Here is a small archive (docker_build_problem.zip) that provides tiny set of files enabling you to easily experience the issue. Refer to the included Readme.md for reproducing instructions...

[inutano]

Same here. I'd like to help if we could change the spec to make dockerFile able to accept a file object and files to be added in the container as secondaryFiles!