Implement a subset of the Common Workflow Language.

[jmchilton]

This should support a subset of draft-3 and v1.0 tools.

What is holding us back from merging the progress so far?

• [ ] no new failing CIs
• [ ] CWL v1.0 required conformance tests must pass?
• [x] Download and generate (most) conformance tests instead of including everything in the Galaxy repo
• [x] Fix upstream 1.2 conformance tests with duplicated labels and possible other issues: https://gist.github.com/nsoranzo/64892b814da7ae752ee1da22bdb2bdf9
• [ ] manage expectations
  • [ ] documentation (badges? how that works) on what is working and what is not
  • [ ] docs on "our" position on native Galaxy tools and workflows vs CWL tools + galaxy hints and CWL workflows
  • [ ] it should be clear to members of the Galaxy ecosystem if they want to write CWL tools or Galaxy tools, and whether they can expect to edit CWL workflows in the UI etc.

CWL Support (Tools):

• Implemented integer, long, float, double, boolean, string, File, Directory, "null", Any, as well as records and arrays thereof. There are two approaches to handling more complex parameters discussed here (#59).
• secondaryFiles that are actual Files are implemented, secondaryFiles containing directories are not yet implemented.
• InlineJavascriptRequirement are support to define output files (see test_cat3 test case).
• EnvVarRequirements are supported (see the test_env_tool1 and test_env_tool2 test cases).
• Expression tools are supported (see parseInt-tool test case).
• Shell tools are also support (see record output test case).
• Default File values are very un-Galaxy and have been hacked into work with Tools - they still don't work with workflows.
• Partial Docker support - this supports the most simple and common pullFrom semantics but not additional ways to fetch containers or additional options such as output directory configuration (https://github.com/common-workflow-language/galaxy/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aopen%20Docker). Additionally, Galaxy mounts the inputs and outputs where it wants instead of CWL required mount points - this needs to be fixed for the conformance tests but may not matter much in practice (I'm not sure).

CWL Support (Workflows):

• Simple connections and tool execution.
• Overriding tool input defaults via literal values and simple expressions.
• MultipleInputFeatureRequirements to glue together multiple file inputs into a File[] or multiple File[] into a single flat File[]. (nested merge is still a TODO).
• Simple scatter semantics for Files and non-Files (e.g. count-lines3).
• Simple subworkflows (e.g. count-lines10).
• Simple valueFrom expressions (e.g. step-valueFrom and step-valueFrom2). This work doesn't yet model non-tool parameters to steps - for complex valueFrom expressions like in step-valueFrom3 do not work yet.

Remaining Work

The work remaining is vast and will be tracked at https://github.com/common-workflow-language/galaxy/issues for the time being.

Implementation Notes:

Tools:

• Non-File CWL outputs are represented as expression.json files. Traditionally Galaxy hasn't supported non-File outputs from tools but CWL Galaxy has work in progress on bringing native Galaxy support for such outputs #27.
• CWL secondary files are just normal datasets with extra files stored in __secondary_files__ directory in the dataset's extra_files_path directory and indexed in a file called secondary_files_index.json in extra_files_path. The upload tools has been augmented to allow attaching arbitrary extra files as a tar file to support getting data into this format initially. CWL requires staging files to include their parent File's basename - but tools describe inputs as just the extension. I'm not sure which way Galaxy should store secondary_files__ in its objectstore - just with the extension or with the basename and extension - both options are implemented and can be swapped by setting the boolean STORE_SECONDARY_FILES_WITH_BASENAME in galaxy.tools.cwl.util.
• CWL Directory types are datasets of a new type "directory" implemented earlier in this branch.
• The tool execution API has been extended to add a inputs_representation parameter that can be set to "cwl" now. The cwl representation for running tools corresonding to the CWL job json format with {class: "File: path: "/path/to/file"} inputs replaced with {"src": "hda", "id": ""}. Code for building these requests for CWL job json is available in the test class.
• Since the CWL <-> Galaxy parameter translation may change over time, for instance if Galaxy develops or refines parameter classes - CWL state and CWL state version is tracked in the database and hopefully for reruns, etc... we could update the Galaxy state from an older version to a new one.
• CWL allows output parameters to be either File or non-File and determined at runtime, so galaxy.json is used to dynamically adjust output extension as needed for non-File parameters.

Workflows:

• This work serializes embedded and referenced tools into the database - this will allow reuse and tracing without require the path to exist forever on the filesystem - this will have problems with default file references in workflows.
• Implements re-mapping CWL workflow connections to Galaxy input connections.
• Fix tool serialization for jobs for path-less tools (such as embedded tools).
• Hack tool state during workflow import for CWL.
• The sort of dynamic shaping of inputs CWL allows has required enhancing Galaxy's map/reduce stuff to allow mapping over dynamic collections that don't yet exist at the time of tool execution and need to be created on the fly. This commit creates them as HDCAs - but likely they should be something else that doesn't appear in the history panel.
• Multi-input scattering but only scatterMethod == "dotproduct" is currently support. Other scatter methods (nested_crossproduct and flatcross_product) are not used by workflows in GA4GH challenge.

Implementation Description:

The reference implementation Python library (mainly developed by Peter Amstutz - https://github.com/common-workflow-language/common-workflow-language/tree/master/reference) is used to load tool files ending with .json or .cwl and proxy objects are created to adapt these tools to Galaxy representations. In particular input and output descriptions are loaded from the tool.

When the tool is submitted, a special specialized tool class is used to build a cwltool compatible job description from the supplied Galaxy inputs and the CWL reference implementation is used to generate a CWL reference implementation Job object. A command-line is generated from this Job object.

As a result of this - Galaxy largely does not need to worry about the details of command-line adapters, expressions, etc....

Galaxy writes a description of the CWL job that it can reload to the job working directory. After the process is complete (on the Galaxy compute server, but outside the Docker container) this representation is reloaded and the dynamic outputs are discovered and moved to fixed locations as expected by Galaxy. CWL allows for much more expressive output locations than Galaxy, for better or worse, and this step uses cwltool to adapt CWL to Galaxy outputs.

Currently all File outputs are sniffed to determined a Galaxy datatype, CWL allows refinement on this and this remains work to be done.

1) CWL should support EDAM declaration of types and Galaxy should provide a mapping to core datasets to skip sniffing is types are found. 2) For finer grain control within Galaxy, extensions to CWL should allow setting actual Galaxy output types on outputs. (Distinction between fastq and fastqsanger in Galaxy is very important for instance.)

Implementation Links:

Hundreds of commits have been rebased into this one and so the details of individual parts of the implementation and how they built on each other are not enitrely clear. To see the original ideas behind individual features - here are some relevant links:

• Implement merge_nested link semantics for workflow steps (a903abd).
• Implement subworkflows in CWL (9933c3c)
• MultipleInputFeatureRequirements:
  • Second attempt: ed8307f
  • First attempt: ae11f56
• Basic, implicit dotproduct scattering of workflows - d1ad64e.
• Simple input StepInputExpressionRequirements - 819a27b
• StepInputExpressionRequirements for multiple inputs - 5e7f622
• Record Types in CWL - e6be28a
• Rework original approach at mapping CWL state to tool state - 669ea55
• Rework approach at mapping CWL state to tool state again to use "FieldTypeToolParameter"s - implements default values, optional parameters, and union types for workflow inputs. d1ca22f
• Initial tracking of "cwl_filename" for CWL jobs (67ffc55).
• Reworked secondary file staging, implement testing and indexing of secondary files - 03d1636.

Testing:

% git clone https://github.com/common-workflow-language/galaxy.git
% cd galaxy
% git checkout cwl-1.0

Start Galaxy.

% GALAXY_RUN_WITH_TEST_TOOLS=1 sh run.sh

Open http://localhost:8080/ and see CWL test tools (along with all Galaxy test tools) in left hand tool panel.

To go a step further and actually run CWL jobs within their designated Docker containers, copy the following minimal Galaxy job configuration file to config/job_conf.xml. (Adjust the docker_sudo parameter based on how you execute Docker).

https://gist.github.com/jmchilton/3997fa471d1b4c556966

Run API tests demonstrating the various CWL demo tools with the following command.

./run_tests.sh -api test/api/test_tools_cwl.py
./run_tests.sh -api test/api/test_workflows_cwl.py
./run_tests.sh -api test/api/test_cwl_conformance_v1_0.py

The first two execute various tool and workflow test cases manually crafted during implementation of this work. The third is an auto-generate test case class that contains Python tests for every CWL conformance test found with the reference specification.

An individual conformance test can be ran using this pattern:

./run_tests.sh -api test/api/test_cwl_conformance_v1_0.py:CwlConformanceTestCase.test_conformance_v1_0_6

Issues and Contact

Report issues at https://github.com/common-workflow-language/galaxy/issues and feel free ping jmchilton on the CWL Gitter channel.

[mr-c]

@jmchilton This is fantastic, thank you for the refresh!

Right now the conformance tests are running at https://ci.commonwl.org/job/galaxy-planemo-conformance/ using planemo's main GitHub branch. What's the best way to update the Jenkins job that so I run the conformance tests using this branch of Galaxy? I see your Testing instructions above, but it would be nice to run CWL conformance tests through the same interface the other implementations are tested with.

With regards to

1. CWL should support EDAM declaration of types and Galaxy should provide a mapping to core datasets to skip sniffing is types are found.

This is already supported in v1.0: any identifier can be used in the format field ( see http://www.commonwl.org/v1.0/CommandLineTool.html#File and http://www.commonwl.org/v1.0/CommandLineTool.html#CommandInputParameter)

Example: (the # BAM comments are optional): format: http://edamontology.org/format_2572 # BAM or format: edam:format_2572 # BAM if elsewhere in the document there is

$namespaces: { edam: "http://edamontology.org/" }
$schemas: [ "http://edamontology.org/EDAM_1.16.owl" ]

I explicitly demo and promote use of EDAM for bioinformatics tools and workflows (though the example workflow repo still needs updating)

2. For finer grain control within Galaxy, extensions to CWL should allow setting actual Galaxy output types on outputs. (Distinction between fastq and fastqsanger in Galaxy is very important for instance.)

Good news: you can do this today without modifying or extending CWL. Any Galaxy output type (a file format, in CWL parlance) that isn't represented in EDAM can be added as an additional format specifier (preferably in a galaxyproject.org namespace): http://www.commonwl.org/v1.0/CommandLineTool.html#CommandInputParameter Then the Galaxy user interface can choose to display only the files that have the the Galaxy specific type(s) when a CWL description specifies both generic and Galaxy specific formats, thus giving the best user experience.

In the case of Galaxy's fastq and fastqsanger this is represented in EDAM: http://edamontology.org/format_1930 http://edamontology.org/format_1932

Hypothetical example if Galaxy's fastqsanger subtype was not represented in EDAM: format: [ http://edamontology.org/format_1930, https://galaxyproject.org/fastqsanger ] or format: [ edam:format_1930, galaxy:fastqsanger ] if elsewhere in the document there is

$namespaces: { edam: "http://edamontology.org/", galaxy: "https://galaxyproject.org/" }
$schemas: [ "http://edamontology.org/EDAM_1.16.owl", "https://galaxyproject.org/formats-release_17.01.owl" ]

Obviously it would be best for bioinformatic CWL descriptions to only use EDAM formats, but this approach means that you won't have to wait for EDAM updates to still have the best user experience in Galaxy (though EDAM releases much faster than they used to).

[mr-c]

@jmchilton can we get a refresh of this PR?

[nsoranzo]

I've rebased the cwl-1.0 branch, fixed some API tests (changes rebased in the original commits) and added 3 commits that I will PR to galaxy. Now tests are back to a decent state if anyone wants to fix more API or CWL conformance tests.

[nsoranzo]

Rebased, fixed a TODO by adding cwltest to pyproject.toml .

[nsoranzo]

Rebased for the upcoming BioHackathon Europe 2021.

[nsoranzo]

@jmchilton Now that in the dev branch we have moved test/unit/tools/ to test/unit/app/tools/, should we also move test/unit/tools/cwl_tools/ to test/unit/app/tools/cwl_tools/, or is there a better place?

[jmchilton]

Now that in the dev branch we have moved test/unit/tools/ to test/unit/app/tools/, should we also move test/unit/tools/cwl_tools/ to test/unit/app/tools/cwl_tools/, or is there a better place?

Sounds like you've picked the right place - unless there are tests in tool_util that depend on these and then we'd require more thought.

[nsoranzo]

I've just ticked off Download and generate (most) conformance tests instead of including everything in the Galaxy repo from the list in the description. We have gone from 1,245 files changed (241,593 added lines) to 107 files (4,756 added lines).

[nsoranzo]

Closing this with the aim of opening this against galaxyproject/dev ASAP.

[mr-c]

New location is https://github.com/galaxyproject/galaxy/pull/12909 ; thank you @nsoranzo !