search

common-workflow-language / cwltool

Common Workflow Language reference implementation
https://cwltool.readthedocs.io/
Apache License 2.0
332 stars 230 forks source link

'$graph' not found when creating of a workflow uses commandline tools defined inside the workflow definition #1325

Open alexiswl opened 4 years ago

alexiswl commented 4 years ago

Expected Behavior

Same packed file to be generated regardless of the build type

Actual Behavior

Tell us what happens instead

If a workflow refers to a commandline tool defined in the run attribute of the step. A non-$graph type of workflow is packed.

If a workflow refers to a commandline tool defined in a separate file and linked to in the run attribute of the step. A $graph type of workflow is packed instead

Workflow Code - one file per tool

Taken from https://github.com/common-workflow-language/common-workflow-language/tree/master/v1.0/examples


cwlVersion: v1.0
class: Workflow
inputs:
  inp: File
  ex: string

outputs:
  classout:
    type: File
    outputSource: compile/classfile

steps:
  untar:
    run: tar-param.cwl
    in:
      tarfile: inp
      extractfile: ex
    out: [example_out]

  compile:
    run: arguments.cwl
    in:
      src: untar/example_out
    out: [classfile]

Workflow Code - Single File

cwlVersion: v1.0
class: Workflow
inputs:
  inp: File
  ex: string

outputs:
  classout:
    type: File
    outputSource: compile/classfile

steps:
  untar:
    run:
      cwlVersion: v1.0
      class: CommandLineTool
      baseCommand: [tar, xf]
      inputs:
        tarfile:
          type: File
          inputBinding:
            position: 1
        extractfile:
          type: string
          inputBinding:
            position: 2
      outputs:
        example_out:
          type: File
          outputBinding:
            glob: $(inputs.extractfile)
    in:
      tarfile: inp
      extractfile: ex
    out: [example_out]

  compile:
    run:
      cwlVersion: v1.0
      class: CommandLineTool
      label: Example trivial wrapper for Java 7 compiler
      hints:
        DockerRequirement:
          dockerPull: java:7-jdk
      baseCommand: javac
      arguments: ["-d", $(runtime.outdir)]
      inputs:
        src:
          type: File
          inputBinding:
            position: 1
      outputs:
        classfile:
          type: File
          outputBinding:
            glob: "*.class"
    in:
      src: untar/example_out
    out: [classfile]

Workflow Code - one file per tool - packed

{
    "$graph": [
        {
            "class": "CommandLineTool",
            "label": "Example trivial wrapper for Java 7 compiler",
            "hints": [
                {
                    "dockerPull": "java:7-jdk",
                    "class": "DockerRequirement"
                }
            ],
            "baseCommand": "javac",
            "arguments": [
                "-d",
                "$(runtime.outdir)"
            ],
            "inputs": [
                {
                    "type": "File",
                    "inputBinding": {
                        "position": 1
                    },
                    "id": "#arguments.cwl/src"
                }
            ],
            "outputs": [
                {
                    "type": "File",
                    "outputBinding": {
                        "glob": "*.class"
                    },
                    "id": "#arguments.cwl/classfile"
                }
            ],
            "id": "#arguments.cwl"
        },
        {
            "class": "Workflow",
            "inputs": [
                {
                    "type": "string",
                    "id": "#main/ex"
                },
                {
                    "type": "File",
                    "id": "#main/inp"
                }
            ],
            "outputs": [
                {
                    "type": "File",
                    "outputSource": "#main/compile/classfile",
                    "id": "#main/classout"
                }
            ],
            "steps": [
                {
                    "run": "#arguments.cwl",
                    "in": [
                        {
                            "source": "#main/untar/example_out",
                            "id": "#main/compile/src"
                        }
                    ],
                    "out": [
                        "#main/compile/classfile"
                    ],
                    "id": "#main/compile"
                },
                {
                    "run": "#tar-param.cwl",
                    "in": [
                        {
                            "source": "#main/ex",
                            "id": "#main/untar/extractfile"
                        },
                        {
                            "source": "#main/inp",
                            "id": "#main/untar/tarfile"
                        }
                    ],
                    "out": [
                        "#main/untar/example_out"
                    ],
                    "id": "#main/untar"
                }
            ],
            "id": "#main"
        },
        {
            "class": "CommandLineTool",
            "baseCommand": [
                "tar",
                "xf"
            ],
            "inputs": [
                {
                    "type": "string",
                    "inputBinding": {
                        "position": 2
                    },
                    "id": "#tar-param.cwl/extractfile"
                },
                {
                    "type": "File",
                    "inputBinding": {
                        "position": 1
                    },
                    "id": "#tar-param.cwl/tarfile"
                }
            ],
            "outputs": [
                {
                    "type": "File",
                    "outputBinding": {
                        "glob": "$(inputs.extractfile)"
                    },
                    "id": "#tar-param.cwl/example_out"
                }
            ],
            "id": "#tar-param.cwl"
        }
    ],
    "cwlVersion": "v1.0"
}

Workflow Code - Single File - Packed

{
    "class": "Workflow",
    "inputs": [
        {
            "type": "string",
            "id": "#main/ex"
        },
        {
            "type": "File",
            "id": "#main/inp"
        }
    ],
    "outputs": [
        {
            "type": "File",
            "outputSource": "#main/compile/classfile",
            "id": "#main/classout"
        }
    ],
    "steps": [
        {
            "run": {
                "cwlVersion": "v1.0",
                "class": "CommandLineTool",
                "label": "Example trivial wrapper for Java 7 compiler",
                "hints": [
                    {
                        "dockerPull": "java:7-jdk",
                        "class": "DockerRequirement"
                    }
                ],
                "baseCommand": "javac",
                "arguments": [
                    "-d",
                    "$(runtime.outdir)"
                ],
                "inputs": [
                    {
                        "type": "File",
                        "inputBinding": {
                            "position": 1
                        },
                        "id": "#main/compile/2486c948-9b99-4be4-b1ed-15ec38c0d842/src"
                    }
                ],
                "outputs": [
                    {
                        "type": "File",
                        "outputBinding": {
                            "glob": "*.class"
                        },
                        "id": "#main/compile/2486c948-9b99-4be4-b1ed-15ec38c0d842/classfile"
                    }
                ],
                "id": "#main/compile/2486c948-9b99-4be4-b1ed-15ec38c0d842"
            },
            "in": [
                {
                    "source": "#main/untar/example_out",
                    "id": "#main/compile/src"
                }
            ],
            "out": [
                "#main/compile/classfile"
            ],
            "id": "#main/compile"
        },
        {
            "run": {
                "cwlVersion": "v1.0",
                "class": "CommandLineTool",
                "baseCommand": [
                    "tar",
                    "xf"
                ],
                "inputs": [
                    {
                        "type": "string",
                        "inputBinding": {
                            "position": 2
                        },
                        "id": "#main/untar/106062ee-ebb7-41c6-b19e-ae5d17248670/extractfile"
                    },
                    {
                        "type": "File",
                        "inputBinding": {
                            "position": 1
                        },
                        "id": "#main/untar/106062ee-ebb7-41c6-b19e-ae5d17248670/tarfile"
                    }
                ],
                "outputs": [
                    {
                        "type": "File",
                        "outputBinding": {
                            "glob": "$(inputs.extractfile)"
                        },
                        "id": "#main/untar/106062ee-ebb7-41c6-b19e-ae5d17248670/example_out"
                    }
                ],
                "id": "#main/untar/106062ee-ebb7-41c6-b19e-ae5d17248670"
            },
            "in": [
                {
                    "source": "#main/ex",
                    "id": "#main/untar/extractfile"
                },
                {
                    "source": "#main/inp",
                    "id": "#main/untar/tarfile"
                }
            ],
            "out": [
                "#main/untar/example_out"
            ],
            "id": "#main/untar"
        }
    ],
    "id": "#main",
    "cwlVersion": "v1.0"
}

Your Environment

mr-c commented 4 years ago

Hello @alexiswl . I see your point.

$graph is currently only added by cwltool when needed and the second input doesn't need it (nor packing) as it is already self-contained.

In CWL v1.2 we will be providing more guidance about packed documents (though they have been implicitly allowed since v1.0). Here is the current text https://github.com/common-workflow-language/cwl-v1.2/commit/63da514cbacd274abc391c7858514b7f68f7a8bd?short_path=f672fea#diff-f672fea773fb5554fdcc9faa6e7b5093 and your feedback would be welcome

If "packing" is just about combining multiple CWL documents together, then the current situation is good enough.

If it is about producing that and a simplified and more consistent serialization, then we should always add $graph when packing.

Maybe the options should be split into two: one that combines into a single document that always uses $graph and is in JSON form (comments are removed, etc) and the second that combines into a single document, if needed, and preserves as much of the original form as possible (YAML-in, YAML-out, keeps comments)

@tetron what do you think of that? Any ideas on names?

@alexiswl would that suffice for you? could you help with the implementation?