search

dnephin / dobi

A build automation tool for Docker applications
https://dnephin.github.io/dobi/
Apache License 2.0
309 stars 36 forks source link

Run jobs in parallel #158

Closed Fuco1 closed 5 years ago

Fuco1 commented 5 years ago

I have a job that is configured by environment variables.

I'm using make to build X into Y and the build job is defined in dobi. However when I run (through make) dobi twice it gives me an error that the container already exists.

I was under the impression that the job always creates a new container but it seems that's not the case. Can we have a feature/flag to always create a new unique container? Or is there some way to do this?

The shell output

> make -j2
INPUT_DATA_FILE=data/input-24.tsv dobi forecast > out/forecast-24.json
INPUT_DATA_FILE=data/input-159.tsv dobi forecast > out/forecast-159.json
[env:set settings] Done
[env:set settings-local] Done
[job: forecast] Rscript forecast-regular.R Start
[env:set settings] Done
[env:set settings-local] Done
[job: forecast] Rscript forecast-regular.R Start
[WARN] [job: forecast] Rscript forecast-regular.R container=forecast-matus-forecast Container does not exist
[ERROR] failed to execute task "forecast:": failed creating container "forecast-matus-forecast": container already exists
Makefile:5: recipe for target 'out/forecast-159.json' failed
dnephin commented 5 years ago

Each job does use a different container, but the docker container name must be unique, and by default the container name uses values that wont change. To run multiple copies of the same job you can use $DOBI_EXEC_ID to set the unique value for the run.