chanzuckerberg / miniwdl

Workflow Description Language developer tools & local runner
MIT License
172 stars 54 forks source link

Feature request: DOT (or similar) graph generation from WDL #311

Open pshapiro4broad opened 4 years ago

pshapiro4broad commented 4 years ago

I'm not sure if this falls under the umbrella of things miniwdl does, but it would be nice if it could generate a graph file (using DOT or some similar format) from an input WDL. This feature was supported by cromwell's womtool graph command but it has never worked for WDL 1.0. I'm sure miniwdl has all the data needed to generate such a graph, and someone could use it as a library to do this as well.

mlin commented 4 years ago

Thanks for this suggestion; I think it would make a nice example program for the documentation (#228), should be pretty simple yet actually useful in its own right.

mlin commented 4 years ago

building block -- https://github.com/xflr6/graphviz

jdidion commented 4 years ago

I will have some time to take this on in January, if someone else hasn't grabbed it by then. This will be hugely useful since WOMtool graph doesn't yet support WDL 1.0.

mlin commented 4 years ago

I've checked in a crude first cut at this https://github.com/chanzuckerberg/miniwdl/blob/master/examples/wdlviz.py

The results look pretty bad, because I'm unfamiliar with graphviz and generally lacking in the visual design talents. The good news is that it's only about 100 SLOC since it just has to project miniwdl's own model of the workflow dependency graph. So, I'm hoping somebody with more viz talent will be able to run with this and refine it a lot further :sweat_smile:

PRs welcome!!

Examples- install miniwdl & graphviz (using pip3 or conda) and

$ ./wdlviz.py https://raw.githubusercontent.com/mlin/DeepVariant-GLnexus-WDL/master/wdl/DeepVariant_GLnexus.wdl

image

$ ./wdlviz.py https://raw.githubusercontent.com/gatk-workflows/gatk4-germline-snps-indels/master/joint-discovery-gatk4-local.wdl

image

cc @vdauwera @geoffjentry

pshapiro4broad commented 4 years ago

For my usage, here's an example of the level of detail that would be helpful for me. This is the output of "womtool graph" on an older version of the GATK single sample pipeline. The graph shows dependencies, scatter/gather, leaf nodes and root nodes. I think it also shows conditionals. It doesn't show inputs and outputs, which removes a lot of complexity.

PairedSingleSampleWf.autogenerated.dot.txt

outfile

mlin commented 4 years ago

Our new codelab cleans it up a bit, worth a try. Hoping for someone to heroically swoop in and rescue the visual design =)

image

kislyuk commented 4 years ago

imho graphviz is just too dated for this.

Consider emitting mermaid-consumable markdown: https://mermaid-js.github.io/mermaid/

mlin commented 4 years ago

Consolidated from #322

We have a codelab on using the AST to create a crude graphviz visualization, https://miniwdl.readthedocs.io/en/latest/wdlviz.html and a slightly (not much) better version checked in the repo: https://github.com/chanzuckerberg/miniwdl/blob/master/examples/wdlviz.py We probably won't have bandwidth to fully flesh this out in the near future, but would definitely welcome if someone else is able to run with it!