Workflow Description Language (WDL)

The Workflow Description Language (WDL) is an open standard for describing data processing workflows with a human-readable and writeable syntax. WDL makes it straightforward to define analysis tasks, connect them together in workflows, and parallelize their execution. The language strives to be accessible and understandable to all manner of users, including programmers, analysts, and operators of a production system. The language enables common patterns, such as scatter-gather and conditional execution, to be expressed simply. WDL is designed for portability, and there are several implementations to choose from that run in a variety of environments, including HPC systems and cloud platforms.

Versioning

WDL versioning follows semantic versioning conventions.

The WDL language has a two-number version (e.g., 1.2). An increase in the minor (second) version number (e.g., 1.1 to 1.2) indicates the addition of, or non-breaking changes to, the language or standard library functions. An increase in the major (first) version number (e.g., 1.0 to 2.0) indicates that breaking changes have been made.

The WDL specification has a three-number version (e.g., 1.2.0). The specification version tracks the language version, but there may also be patch releases (indicated by a change to the patch, or third, version number) that include fixes for typos, additional examples, or non-breaking clarifications of ambiguous language.

Language Specifications

The WDL specification contains all relevant information for users and developers, including those wanting to implement an execution engine. This GitHub project uses the branch for the current version of the specification as its primary branch, so you will always see the current version of the specification so long as you visit this project's root URL. Users are strongly encouraged to use the current version of the specification unless absolutely necessary.

This branch is for version 1.2 of the WDL language specification. All development of new non-breaking features should be done against this branch.

Previous versions of the spec can be found here:

• 1.1
• 1.0

There are a number of draft versions that correspond to initial efforts at creating WDL. While these are functional specifications, they should not be considered feature complete, and they contain many bugs and irregularities.

• draft-3
• draft-2
• draft-1

The next major version of the specification is 2.0. All development of new breaking features should be done against that branch.

Community and Support

The WDL community depends on your involvement to thrive. You are encouraged to ask questions, help other users, and make contributions where you can. Interactions occur primarily on GitHub and Slack. The WDL community also has an official blog where announcements are made.

Asking a Question

• Search in the discussions to see if the question has been asked already; if not, start a new discussion.
• Join our Slack and ask in the #support channel.
• Search the Bioinformatics Stack Exchange for previously answered questions, or ask a new question.
• Search the Google Group for previously answered questions. This group is largely inactive, so you're encoraged to ask new questions in one of the above places instead.

Bugs and New Features

• Search for an existing issue. If your issue has not already been reported, create a new one.
• For feature reqeusts, you are encoraged to first start a discussion at one of the places listed above to get feedback from the community.
• If you'd like to provide a fix/implementation for an issue, please read about contributing before submitting a pull request.

Documentation

• wdl-docs
• learn-wdl
• WDL Resources provided by Terra (a product of the Broad Institute)

Published Workflows

The following are collections of open-source WDL workflows. The WDL task or workflow you need may already be available in one of these repositories, or you may find a similar workflow and customize it to your needs.

• Dockstore
• BioWDL
• Broad Institute WARP
• GATK Workflows
• ENCODE
• Terra (requires registration)

Software and Tools

Execution Engines and Platforms

WDL does not have an official implementation. Third parties are relied upon to provide installable software or hosted platforms that interpret and execute WDL workflows and tasks. Although WDL does not yet have an official compliance program or certification process, implementers are expected to design their tools according to the specification so as to maximize the portability of workflows across implementations. Nonetheless, implementers may provide additional optional features specific. Please see the documentation associated with each tool/platform for information on available execution options and support.

* Also see WDL Runner, a script for launch WDL workflows on GCP using Cromwell

Grammars, Parsers, and Language Support

• The WDL parsers repository provides grammar definitions in various formats and generated parsers for various languages.
• MiniWDL provides python bindings for WDL as well as commandline tools for validation, linting, and generating workflow input templates.
• WOMTool is a standalone Java tool for WDL parsing, validating, linting, and diagramming.
• wdlTools - provides 1) a parser Java/Scala library, based on the ANTLR4 grammars grammars, for WDL draft-2, 1.0, 1.1, and 2.0; and 2) command-line tools for sytanx checking, type-checking, linting, code formatting (including upgrading from older to newer WDL versions), generating documentation, and executing WDL tasks locally.
• sprocket a package manager and linter for WDL files.

IDE Support

Documentation

• wdl-aid generates documentation for the inputs of WDL workflows, based on the parameter_meta information defined in the WDL file.
• wdldoc

Testing

• wdl-tests is a collection of test cases for WDL implementations. A specification is provided for writing new tests that are compatible with automated testing frameworks.
• Pytest-workflow is a implementation-agnostic workflow tester. It can be used with both Cromwell and MiniWDL. Uses pytest as the underlying test framework. Tests can be specified in either YAML format or python code.
• Pytest-wdl is a plugin for the pytest unit testing framework that enables testing of WDL tasks and workflows. Tests can be specified in either YAML format or python code.

Packaging

• miniwdl zip generates a ZIP file including a given WDL source code file and any other WDL files it imports. The ZIP file can be supplied directly to miniwdl run, which can extract it automatically.
• wdl-packager packages a workflow and all of its imports into a zip file. The zip can be used as an imports zip package for cromwell. The utility can add non-WDL files (such as the license) to the zip package and provides options to package the zip in a binary reproducible way.

Contributing

WDL only advances through community contributions. While participating in discussions and submitting issues are great ways to be involved, help is also needed to implement changes to the specification. For more information on how you can contribute, please read the Contributing guide.

RFC Process

Submitted pull requests are subject to the RFC Process. Please review and familiarize yourself with the process if you would like to see changes submitted to the specification.

Governance

The WDL specification is entirely community driven; however, it is overseen by a Governance committee. If you are interested in being involved in WDL governance, please join the Slack and post a message in the #general channel.