search

common-workflow-language / common-workflow-language

Repository for the CWL standards. Use https://cwl.discourse.group/ for support 😊
https://www.commonwl.org
Apache License 2.0
1.45k stars 198 forks source link

investigate Makefile -> CWL conversion #485

Open mr-c opened 7 years ago

mr-c commented 7 years ago

See https://github.com/lindenb/xml-patch-make/blob/master/stylesheets/graph2cwl.xsl (link courtesy @sjackman )

sjackman commented 7 years ago

I used graph2cwl.xsl to convert the Makefile tigmint-make to CWL: https://github.com/bcgsc/tigmint/blob/master/tigmint-make https://github.com/bcgsc/tigmint/blob/master/tigmint-make.cwl The resulting CWL file cannot be parsed by https://view.commonwl.org The error message is Error: The workflow could not be parsed from the given URL I'll try a command line tool to parse this CWL file and hopefully get a more informative error message.

sjackman commented 7 years ago 
❯❯❯ cwl-runner tigmint-make.cwl
/usr/local/bin/cwl-runner 1.0.20170721221557
Resolved 'tigmint-make.cwl' to 'file:///Users/sjackman/work/tigmint/tigmint-make.cwl'
No cwlVersion found, treating this file as draft-2.
Tool definition failed validation:
https://w3id.org/cwl/CommonWorkflowLanguage.yml:841:3: checking object `https://w3id.org/cwl/cwl#Binding`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:844:3:   checking field `fields`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:852:7:     checking object `https://w3id.org/cwl/cwl#secondaryFiles`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:677:7:       object name `https://w3id.org/cwl/cwl#secondaryFiles` previously defined
https://w3id.org/cwl/CommonWorkflowLanguage.yml:900:3: checking object `https://w3id.org/cwl/cwl#InputParameter`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:907:3:   checking field `fields`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:912:7:     checking object `https://w3id.org/cwl/cwl#inputBinding`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:884:7:       object name `https://w3id.org/cwl/cwl#inputBinding` previously defined
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1241:3: checking object `https://w3id.org/cwl/cwl#CommandOutputParameter`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1249:3:   checking field `fields`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1250:7:     checking object `https://w3id.org/cwl/cwl#outputBinding`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1221:7:       object name `https://w3id.org/cwl/cwl#outputBinding` previously defined
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1526:3: checking object `https://w3id.org/cwl/cwl#WorkflowStepInput`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1565:3:   checking field `fields`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1582:7:     checking object `https://w3id.org/cwl/cwl#linkMerge`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1519:7:       object name `https://w3id.org/cwl/cwl#linkMerge` previously defined
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1587:7:     checking object `https://w3id.org/cwl/cwl#default`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:802:7:       object name `https://w3id.org/cwl/cwl#default` previously defined
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1622:3: checking object `https://w3id.org/cwl/cwl#WorkflowStep`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1674:3:   checking field `fields`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1679:7:     checking object `https://w3id.org/cwl/cwl#inputs`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1025:7:       object name `https://w3id.org/cwl/cwl#inputs` previously defined
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1689:7:     checking object `https://w3id.org/cwl/cwl#outputs`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1035:7:       object name `https://w3id.org/cwl/cwl#outputs` previously defined
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1696:7:     checking object `https://w3id.org/cwl/cwl#requirements`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1042:7:       object name `https://w3id.org/cwl/cwl#requirements` previously defined
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1931:3: checking object `https://w3id.org/cwl/cwl#ExpressionEngineRequirement`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1937:3:   checking field `fields`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1942:7:     checking object `https://w3id.org/cwl/cwl#requirements`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1042:7:       object name `https://w3id.org/cwl/cwl#requirements` previously defined
sjackman commented 7 years ago

I fixed the above errors, and now have a new error message.

/usr/local/bin/cwl-runner 1.0.20170721221557
Resolved 'tigmint-make.cwl' to 'file:///Users/sjackman/work/tigmint/tigmint-make.cwl'
Tool definition failed validation:
https://w3id.org/cwl/CommonWorkflowLanguage.yml:841:3: checking object `https://w3id.org/cwl/cwl#Binding`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:844:3:   checking field `fields`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:852:7:     checking object `https://w3id.org/cwl/cwl#secondaryFiles`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:677:7:       object name `https://w3id.org/cwl/cwl#secondaryFiles` previously defined
https://w3id.org/cwl/CommonWorkflowLanguage.yml:900:3: checking object `https://w3id.org/cwl/cwl#InputParameter`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:907:3:   checking field `fields`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:912:7:     checking object `https://w3id.org/cwl/cwl#inputBinding`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:884:7:       object name `https://w3id.org/cwl/cwl#inputBinding` previously defined
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1241:3: checking object `https://w3id.org/cwl/cwl#CommandOutputParameter`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1249:3:   checking field `fields`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1250:7:     checking object `https://w3id.org/cwl/cwl#outputBinding`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1221:7:       object name `https://w3id.org/cwl/cwl#outputBinding` previously defined
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1526:3: checking object `https://w3id.org/cwl/cwl#WorkflowStepInput`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1565:3:   checking field `fields`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1582:7:     checking object `https://w3id.org/cwl/cwl#linkMerge`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1519:7:       object name `https://w3id.org/cwl/cwl#linkMerge` previously defined
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1587:7:     checking object `https://w3id.org/cwl/cwl#default`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:802:7:       object name `https://w3id.org/cwl/cwl#default` previously defined
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1622:3: checking object `https://w3id.org/cwl/cwl#WorkflowStep`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1674:3:   checking field `fields`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1679:7:     checking object `https://w3id.org/cwl/cwl#inputs`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1025:7:       object name `https://w3id.org/cwl/cwl#inputs` previously defined
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1689:7:     checking object `https://w3id.org/cwl/cwl#outputs`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1035:7:       object name `https://w3id.org/cwl/cwl#outputs` previously defined
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1696:7:     checking object `https://w3id.org/cwl/cwl#requirements`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1042:7:       object name `https://w3id.org/cwl/cwl#requirements` previously defined
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1931:3: checking object `https://w3id.org/cwl/cwl#ExpressionEngineRequirement`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1937:3:   checking field `fields`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1942:7:     checking object `https://w3id.org/cwl/cwl#requirements`
https://w3id.org/cwl/CommonWorkflowLanguage.yml:1042:7:       object name `https://w3id.org/cwl/cwl#requirements` previously defined

https://github.com/bcgsc/tigmint/blob/master/tigmint-make.cwl

Any help troubleshooting this error message is greatly appreciated!

manu-chroma commented 7 years ago

@sjackman Try using --debug flag for better error descriptions. eg: cwltool --debug tigmint-make.cwl

mr-c commented 7 years ago

@sjackman we're in the process of completely dropping "draft-2" support in the reference runner, which is the source of those errors. Add cwlVersion: v1.0 and try again?

lindenb commented 7 years ago

I've written that a long time ago and tested it with some simple example. I'm not sure the curent format of CWL is still the one I've tested.

Feel free to submit a PR, but sorry, I won't explore a bug for now :-)

mr-c commented 7 years ago

Hey @lindenb, this is awesome -- and I wish I had seen this earlier, so super cool! Don't worry, we won't harass you. If you want we can migrate the repo so you don't get spammed :-)

lindenb commented 7 years ago

no problem. I should have put the XSLT stylesheets ( https://github.com/lindenb/xml-patch-make/tree/master/stylesheets )

into

https://github.com/lindenb/makefile2graph

rather than in the

https://github.com/lindenb/xml-patch-make

because makefile2graph doesn't need a patch+compilation of a new version of make and it generates a XML graph.

sjackman commented 7 years ago

Does the same graph2cwl.xsl work with the XML output of makefile2graph?

sjackman commented 7 years ago

I've adapted graph2cwl to cwlVersion: v1.0: https://github.com/sjackman/xml-patch-make/blob/graph2cwl-1.0/stylesheets/graph2cwl.xsl

Here's the CWL 1.0 file: https://github.com/bcgsc/tigmint/blob/master/tigmint-make.cwl And the new error message is:

/usr/local/bin/cwl-runner 1.0.20170721221557
Resolved 'tigmint-make.cwl#main' to 'file:///Users/sjackman/work/tigmint/tigmint-make.cwl#main'
I'm sorry, I couldn't load this CWL file, try again with --debug for more information.
The error was: 'CommentedMap' object has no attribute 'decode'
sjackman commented 7 years ago

The output of cwl-runner --print-pre 'tigmint-make.cwl#main' https://gist.github.com/sjackman/b7d5a830e6a9e25f8f444a71b51f1cde

sjackman commented 7 years ago

Please review this commit to see if I did the conversion from draft-2 to v1.0 correction. Note: I most likely did not. https://github.com/sjackman/xml-patch-make/commit/1e47899c9855acda4b5887103e33590fd3420097

mr-c commented 7 years ago

Checking -- yaml output? :-)

mr-c commented 7 years ago

@sjackman Which Makefile are you using?

lindenb commented 7 years ago

Does the same graph2cwl.xsl work with the XML output of makefile2graph?

@sjackman I don't remember if I've used the very same syntax. But the structure of the graph would be the ~same, so that it would be easy to only change the XSLT directives in the xsltstylesheet.

mr-c commented 7 years ago

If someone can give me line by line instructions to run this I can cycle quickly to improve the syntax

sjackman commented 7 years ago

@mr-c https://github.com/bcgsc/tigmint/blob/master/tigmint-make

sjackman commented 7 years ago

@mr-c Line-by-line instructions to reproduce the error:

git clone https://github.com/bcgsc/tigmint
git clone https://github.com/lindenb/xml-patch-make
make -C xml-patch-make
touch tigmint/draft.fa tigmint/reads.fq.gz
xml-patch-make/make-4.1/make-4.1/make -C tigmint -f tigmint-make --xml tigmint/tigmint-make.xml
xsltproc xml-patch-make/stylesheets/graph2cwl.xsl tigmint/tigmint-make.xml >tigmint/tigmint-make.cwl
cwl-runner 'tigmint/tigmint-make.cwl#main'
mr-c commented 7 years ago

@sjackman https://github.com/sjackman/xml-patch-make/pull/1 :-)

sjackman commented 7 years ago

Thanks, Michael! That's awesome. That PR is against my fork of Pierre's upstream repo. Could you please instead send it to @lindenb https://github.com/lindenb/xml-patch-make ?

sjackman commented 7 years ago

I haven't had a chance to test it yet, but you can be sure I shall do as soon as I get a chance.

sjackman commented 7 years ago

@lindenb merged @mr-c's https://github.com/lindenb/xml-patch-make/pull/6

sjackman commented 7 years ago

It works! Thanks, Michael and Pierre. :hugs: https://view.commonwl.org/workflows/github.com/bcgsc/tigmint/tree/master/tigmint-make.cwl

lindenb commented 7 years ago

very cool ! I'm glad to see this old idea resurrected :-) I'll try to move the XSLT stylesheets to makefile2graph later.

sjackman commented 7 years ago

@mr-c I think that I'm close to having this working on my laptop. I'm seeing this error message:

❯❯❯ cwl-runner 'tigmint-make.cwl#main'
…
[step step1] start
Exception on step 'step1'
[step step1] Cannot make job: Invalid job input record:
the `1_target` field is not valid because
  is not a dict
[workflow main] completed permanentFail
{}
Final process status is permanentFail

step1 of tigmint-make.cwl is at https://github.com/bcgsc/tigmint/blob/master/tigmint-make.cwl#L845

    step1:
      in:
        1_target:
          default: __1.ok.flag
        1_dep2:
          source: step2/output
      out: [ output ]
      run: "#tool1"