dagster-io / dagster

An orchestration platform for the development, production, and observation of data assets.
https://dagster.io
Apache License 2.0
11.7k stars 1.48k forks source link

Mermaid JS support on Dagit #14968

Open razafinl opened 1 year ago

razafinl commented 1 year ago

What's the use case?

Dagit is used to document all of our software-defined assets.

Markdown is possible as a description or metadata field for each asset.

It seems like it is not possible to add mermaid blocs for these parameters in order to enrich asset description/metadata with simple schemas and diagrams and make explanation of our asset catalog more powerful.

We would like to be able to add simple mermaid bloc which would be viewable as diagram on Dagit.

graph TD;
    A-->B;
    A-->C;
    B-->D;
    C-->D;

Ideas of implementation

Mermaid blocs are interpreted as proper schemas/diagram when used in either

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

tacastillo commented 1 year ago

Thanks for raising this! I'll triage it over to our UI team.

bengotow commented 1 year ago

Wow this is pretty cool! It took me a minute to realize the Mermaid in your github issue is actually live and not a screenshot. Did not know this block type existed / was widely supported! We'll discuss this tomorrow in the front-end sync and see what we can do - I think that we could use https://github.com/temando/remark-mermaid, but we may try to evaluate how much additional JS this requires / if it can be lazily loaded in when needed.

bengotow commented 1 year ago

Hey @razafinl - thanks for filing this. I'm curious what sort of visualization you're looking to embed with Memaid? Is it possible that the flowchart / graphic in the description could be represented natively in the UI in one of the existing asset/op graphs? If you could provide an example or two I think it'd help us understand if the embedded diagrams are something we should prioritize (and potentially advertise as a feature!)

razafinl commented 8 months ago

Hello @bengotow, For my use-cases, we'd like to display a functional schema describing the information being generated/extracted in the associated Software-Defined Asset. In some cases, schemas are better at describing/explaining than raw text.

One example I have is an asset generated from the combination of many data sources (extracted from REST API) to provide a view. A mermaid schema would allow to display which information are used from data sources as well as how the information is merged to create the resulting SDA.