search

hubmapconsortium / airflow-dev

Configuration, workflows, and DAGs for the development instance of Airflow on HIVE infrastructure
MIT License
1 stars 0 forks source link

CWL docker containers can't mount paths with ':' #33

Open mccalluc opened 4 years ago

mccalluc commented 4 years ago

From Joel:

Interesting problem integrating Chuck's CWL workflow onto the salmon workflow: I can't call workflow.cwl in airflow because there are :'s in the paths for the directories in use. Airflow uses timestamps in the jobids, and the way things are set up now those id's get used in the path names. The extra :'s break the syntax of the docker volume specification, as below:


INFO [job workflow.cwl] /tmp/4cf898fg$ docker \
   run \
   -i \
   --volume=/tmp/4cf898fg:/HZfoyx:rw \
   --volume=/tmp/e8pu8i7i:/tmp:rw \
   --volume=/home/airflow/repositories/ingest-pipeline/src/ingest-pipeline/airflow/data/temp/trig_
_2020-02-08T22:34:44.290910+00:00/cwl_out/cluster-marker-genes:/var/lib/cwl/stg706d994b-d3e7-49fd-9
9a2-d236a8783326/cluster-marker-genes:ro \
   --workdir=/HZfoyx \
   --read-only=true \
   --user=1008:1008 \
...
docker: Error response from daemon: invalid volume specification: '/home/airflow/repositories/inges
t-pipeline/src/ingest-pipeline/airflow/data/temp/trig__2020-02-08T22:34:44.290910+00:00/cwl_out/clu
ster-marker-genes:/var/lib/cwl/stg706d994b-d3e7-49fd-99a2-d236a8783326/cluster-marker-genes:ro'.
mccalluc commented 4 years ago

I think --mount could be used instead. https://github.com/moby/moby/issues/8604#issuecomment-332673783 (and further down that thread, if the commas become an issue, the args can be quoted like a CSV.)