search

nextflow-io / nextflow

A DSL for data-driven computational pipelines
http://nextflow.io
Apache License 2.0
2.61k stars 606 forks source link

the Nextflow DAG should render as a subway plot #5023

Open stevekm opened 1 month ago

stevekm commented 1 month ago

There have been many Issues and support tickets related to new and better ways to render the pipeline DAG especially for use with presentations and repo README's

I think the standard DAG representations are all way too complicated and detailed for normal usage. They grow massive with even a moderately complex pipeline and become unreadable and unintelligible quickly.

I think these diagrams used by nf-core look great and I think these are likely closer to the representation that people often are looking for

sarek_subway

nf-core-rnaseq_metro_map_grey

Not sure what the name for this type of diagram is, a "subway plot"? Can we make this type of diagram an optional output for DAG rendering?

Related? https://github.com/nextflow-io/nextflow/issues/4315

bentsherman commented 1 month ago

These diagrams are made by hand, I think using Adobe tooling, and they require a lot of manual curation. I have a few ideas for how to improve the Mermaid diagram but I don't think we'll be able to achieve these tube diagrams automatically.

You can use the dag.depth config option to reduce the level of detail, and I believe dag.verbose is false by default so it will hide operators. Once we can render the DAG at compile-time, I will be able to include a lot more information that is currently missing like inputs, outputs, and conditional logic.

You might be able to play with the renderer and layout algorithm that Mermaid uses to achieve a better layout, but I haven't looked into it. I believe C4 has similar capabilities.

It could be interesting to encode some information about how to render the DAG, in the pipeline code, by annotating workflows with comments, e.g.:

// @dag-collapse
workflow FOO {
  // ...
}

This would give you more control over which subworkflows to collapse or expand, compared to the dag.depth option which applies a flat level-of-detail across the entire pipeline.