This is not an official Google product.
As of 11 Nov 2017, Dockerflow is no longer actively maintained and will not be enhanced with new features.
For multi-step batch workflows consisting of Docker tasks, we now recommend running in the cloud using:
To run multi-step workflows with dsub, you can create a bash or python script with multiple dsub calls. Execution graphs can be constructed using dsub's job control functionality.
For any Dockerflow functionality that is not satisfied by dsub, please file an issue in the dsub repository.
Dockerflow makes it easy to run a multi-step workflow of Docker tasks using Google Cloud Dataflow for orchestration. Docker steps are run using the Pipelines API.
You can run Dockerflow from a shell on your laptop, and the job will run in Google Cloud Platform using Dataflow's fully managed service and web UI.
Dockerflow workflows can be defined in YAML files, or by writing Java code. Examples of workflows defined in YAML can be found in
Examples of workflows defined in Java can be found in
You can run a batch of workflows at once by providing a CSV file with one row per workflow to define the parameters.
This project was created as a proof-of-concept that Dataflow can be used for monitoring and management of directed acyclic graphs of command-line tools.
Dataflow and Docker complement each other nicely:
While Dockerflow supports a simple YAML workflow definition, a similar approach could be taken to implement a runner for one of the open standards like [Common Workflow Language] (https://github.com/common-workflow-language/common-workflow-language) or Workflow Definition Language.
Install the Google Cloud SDK and run
gcloud init
gcloud auth login
gcloud auth application-default loginRun the following steps on your laptop or local workstation:
git clone this repository.
git clone https://github.com/googlegenomics/dockerflowBuild it with Maven.
cd dockerflow
mvn package -DskipTestsSet up the DOCKERFLOW_HOME environment.
export DOCKERFLOW_HOME="$(pwd)"
export PATH="${PATH}":"${DOCKERFLOW_HOME}/bin"
chmod +x bin/*Run a sample workflow:
dockerflow --project=MY-PROJECT \
--workflow-file=src/test/resources/linear-graph.yaml \
--workspace=gs://MY-BUCKET/MY-PATH \
--input BASE_DIR=gs://MY-BUCKET/MY-PATH/MY-INPUT-FILE.txt
--runner=DirectPipelineRunnerSet MY-PROJECT to your cloud project name, and set MY-BUCKET and MY-PATH
to your cloud bucket and folder. You'll need to have a text file in Cloud Storage
as well, here called MY-INPUT-FILE.txt. You can copy one from
src/test/resources/input-one.txt:
gsutil cp src/test/resources/input-one.txt gs://MY-BUCKET/MY-PATH/input-one.txtThe example will run Dataflow locally with the DirectPipelineRunner, for
orchestration. It will spin up VMs remotely in Google Cloud to run the
individual tasks in Docker. Execution of the local Dataflow runner will block
until the workflow completes. The DirectPipelineRunner is useful for
debugging, because you'll see all of the log messages output to your shell.
To run in your cloud project, and see the pretty Dataflow UI in Google Cloud
Console, you can remove the --runner option to use the default Dataflow runner.
How is Dataflow better than a shell script?
Dataflow provides:
Docker provides:
The Dockerflow command-line expects a static workflow graph definition in YAML, or the Java class name of a Java definition.
If you'd rather define workflows in code, you'll use the Java SDK. See
Everything that can be done with YAML can also be done (and more compactly) in Java code. Java provides greater flexibility too.
The documentation below provides details for defining workflows in YAML. To create a workflow, you define the tasks and execution graph. You can define the tasks and execution graph in a single file, or your graph can reference tasks that are defined in separate YAML files.
A workflow is a recursive format, meaning that a workflow can contain multiple steps, and each of the steps can be a workflow.
Dockerflow has lots of features for creating complex, real-world workflows. The best way to get started with your own workflows is to look at the examples.
The hello, world example shows the most basic workflow in both YAML and Java.
All of the advanced features can be seen in the more complex GATK example. Again, it offers both YAML and Java versions. You'll see pretty much the full range of functionality.
Workflows can be tricky to test and debug. Dataflow has a local runner that makes it easy to fix the obvious bugs before running in your Google Cloud Platform project.
To test locally, set --runner=DirectPipelineRunner. Now Dataflow will run on
your local computer rather than in the cloud. You'll be able to see all of the
log messages.
Two other flags are really useful for testing: --test=true and
--resume=true.
When you set test to true, you'll get a dry run of the pipeline. No calls to
the Pipelines API will be made. Instead, the code will print a log message and
continue. That lets you do a first sanity check before submitting and running on
the cloud. You can catch many errors, mismatched parameters, etc.
When you use the resume flag, Dockerflow will try to resume a failed pipeline
run. For example, suppose you're trying to get your 10-step pipeline to work. It
fails on step 6. You go into your YAML definition file, or edit your Java code.
Now you want to re-run the pipeline. However, it takes 1 hour to run steps 1-5.
That's a long time to wait. With --resume=true, Dockerflow will look to see if
the outputs of each step exist already, and if they do, it will print a log
message and proceed to the next step. That means it takes only seconds to skip
ahead to the failed step and try to rerun it.
Google Cloud Platform has various quotas that affect how many VMs and IP addresses, and how much disk space you can get. Some tips: