Experimental CWL task support

[pditommaso]

The goal of this enhancement is to add to Nextflow the ability to execute CWL CommandLineTool definitions in a native manner ie. the CWL descriptor is mapped to NF internal data structure and execute as NF process.

This idea was envisioned by @skptic and @viklund during the September NF hackathon.

A first prototypal implementation is available in the cwl-task branch.

It allows to import a basic CWL CommonLineTool YAML definition file and to execute it a a NF process. For example, given the following CWL definition named samtools.cwl as shown below:

cwlVersion: v1.0
class: CommandLineTool
baseCommand: samtools
arguments: [index]

requirements:
  - class: InitialWorkDirRequirement
    listing:
      - $(inputs.bamfile)

hints:
  - class: ResourceRequirement
    coresMin: 2
    outdirMin: 2048
    ramMin: 4096

inputs:
  bamfile:
    type: File
    inputBinding:
      position: 1
  outbam:
    type: string
    default: test.bai
    inputBinding:
      position: 2

outputs:
  indexout:
    type: File[]
    outputBinding:
      glob: "*.bam"

In the NF script it's possible to import it and execute as shown below:

samtools = cwlTask('samtools.cwl')
out = samtools(bamfile: bamfile_ch, outbam: baifile_ch) 
out.indexout.println()

• The first line imports the CWL descriptions and creates a method handler to invoke it.
• The second line invokes the execution of the tool as a nextflow process. Note as the CWL inputs need to be specified as named parameters providing for each of them a NF channel as for any NF process.
• Finally the invocation returns a map holding the resulting channel from the task execution, in this case a indexout.

[KevinSayers]

The baseCommand should also be able to take a list

cwlVersion: v1.0
class: CommandLineTool
baseCommand: [tar, xf]
inputs:
  tarfile:
    type: File
    inputBinding:
      position: 1
  extractfile:
    type: string
    inputBinding:
      position: 2
outputs:
  example_out:
    type: File
    outputBinding:
      glob: $(inputs.extractfile)

Results in

Command executed:

  [tar, xf] hello.tar Hello.java

I'll try to take a look at this too.

[KevinSayers]

Modifying the previous code. The prefix tag should also be supported.

cwlVersion: v1.0
class: CommandLineTool
baseCommand: tar
inputs:
  tarfile:
    type: File
    inputBinding:
      prefix: xf
      position: 1
  extractfile:
    type: string
    inputBinding:
      position: 2
outputs:
  example_out:
    type: File
    outputBinding:
      glob: $(inputs.extractfile)

[pditommaso]

well, it's veeeery experimental :)

[KevinSayers]

• [ ] CWL inputs that are optional (i.e. int?) need to be handled
• [ ] parse defaults if no input is provided
• [x] Parse baseCommand if a list
• [x] Handle the prefix
• [x] Parse the DockerRequirement
• [ ] Add support for the JS expressions (i.e. $(inputs.foo) )
• [ ] Determine what the process name should be currently Submitted process > name

[stevekm]

Is this feature still being developed?

[pditommaso]

In the hands of contributors.

[KevinSayers]

I personally plan on continuing to work on it once I have more time.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[pditommaso]

In the case this may resurrect, now there's Java "client" for CWL https://github.com/common-workflow-language/cwljava