terrastruct / d2

D2 is a modern diagram scripting language that turns text to diagrams.
https://d2lang.com
Mozilla Public License 2.0
16.69k stars 417 forks source link

How to show icons of tools responsible for data processing without disturbing a data flow? #1634

Open ppalka-lingaro opened 1 year ago

ppalka-lingaro commented 1 year ago

I started with this:

Approach A:

Data Flow: https://play.d2lang.com/?script=coEFmpVCEch7XFqZeel6YHdZKSRnFiXnpHJ55qWnFoPUKOjaKRQlliuAvWmloJQJllDigouBFCTnpCbmFWfmpSM4qSkI6aLUtMw8qCyYDZPkQtHoGuLDhawUxAcEAAD__w%3D%3D&layout=elk&theme=0&sketch=1& ELK: image TALA: image

as you can see flow is not on the same level I would like to show that transformation and orchestration is done by same tools, I don't want to repeat that same component in every transformation (circle)

I would expect something line this image is there a way to make a relationship between shape / object and another relationship? that kind of connection should not impact the flow on the same level in my diagram transformation circle will not be

Approach B:

attempt 1) I tried other approach as well, created nasted container and almost worked, but orchestration and transformation are not on the same level,

ETL: "Data Processing" { Data Lake: { raw.class: Data Lake Storage cleansed.class: Data Lake Storage refined.class: Data Lake Storage

    raw -> cleansed : "cleasing"
    cleansed -> refined: "refining"
    }
Tranformation: {
            class: Databricks
            label: "Data Transformation"
            }
Orchestration: {
    class: Data Pipeline
    label: "Data Orchestration"
    }

}

image

and when I add rel between them, Orchestration <-> Tranformation: "orchestrate" This is what happen: image

attempt 2) Finally I added another nasted container and made box transparent and now looks ok,

ETL: "Data Processing" { Data Lake: { raw.class: Data Lake Storage cleansed.class: Data Lake Storage refined.class: Data Lake Storage

    raw -> cleansed : "cleasing"
    cleansed -> refined: "refining"
    }
processing: {
    Tranformation: {
                class: Databricks
                label: "Data Transformation"
                }
    Orchestration: {
        class: Data Pipeline
        label: "Data Orchestration"
        }
        Orchestration --> Tranformation: "orchestrate"
}
processing.style.fill: transparent
processing.style.stroke: transparent

} image

but is there a simpler way? what are you using to achive that? is there any way to adjust position of 2 icons, positions can be used only for root level objects?

ppalka-lingaro commented 1 year ago

when I add connection between data orchestration / ingestion (adf) to raw it's ruin layout

image

I would like to see something more like this, is it possible ?

image

ppalka-lingaro commented 1 year ago

When I separate Ingestion and orchestration and Ingestion is not in ETL container it's ok

image