Can you nest and reference other diagrams?

[jacorbello]

I'd like to create separate diagrams for various applications/departments/etc. Is there a way to create say 3 diagrams:

main.py
- diagrams
  - application1.py
  - appliaction2.py
  - application3.py

and reference them all in a "master" diagram?

with Diagram(path='./diagrams/application1.py', label='Application 1')
with Diagram(path='./diagrams/application2.py', label='Application 2')
with Diagram(path='./diagrams/application3.py', label='Application 3')

This is in an effort to reference application dependencies between applications without having to update the same architecture in multiple diagrams.

[clayms]

The closest thing I can think of would be something similar to using the no-op flag in neato (-n CLI flag).

See "-n[num]" in https://graphviz.gitlab.io/doc/info/command.html This Stack Overflow answer also describes how. Also see the Neato Layout Manual.

This library uses the Digraph class from the Graphviz library. https://github.com/mingrammer/diagrams/blob/c2b10f3924bb06e6f77b6da258b2488a4f7749e1/diagrams/__init__.py#L7 https://github.com/mingrammer/diagrams/blob/c2b10f3924bb06e6f77b6da258b2488a4f7749e1/diagrams/__init__.py#L111

Refering to the Graphviz library Digraph is a "Directed graph source code in the DOT language." https://graphviz.readthedocs.io/en/stable/api.html?highlight=Digraph#digraph https://graphviz.readthedocs.io/en/stable/_modules/graphviz/dot.html#Digraph Or: https://github.com/xflr6/graphviz/blob/master/graphviz/dot.py#L296toL297

The filename parameter of Digraph :

filename – Filename for saving the source (defaults to name + '.gv').

This library generates a graphviz (.gv) file, then renders the output image, and finally removes the graphviz file when exiting the Diagram context.

https://github.com/mingrammer/diagrams/blob/c2b10f3924bb06e6f77b6da258b2488a4f7749e1/diagrams/__init__.py#L150-L151

I tried commenting out the line that removes the graphviz file in hopes that I could somehow use it with the no-op flag, but the graphviz file did not remain.

[clayms]

@jacorbello Pretty sure the answer is "yes."
Have a look at issue https://github.com/mingrammer/diagrams/issues/487#issuecomment-800517538.

That example is furthered below by adding separate external graphs contained in their own Clusters

1. create and save separate graphviz dot files

from diagrams import Diagram, Cluster

from diagrams.k8s.clusterconfig import HPA
from diagrams.k8s.compute import Deployment, Pod, ReplicaSet
from diagrams.k8s.network import Ingress, Service

with Diagram("Exposed Pod with 3 Replicas", show=False) as diag_1:
    net = Ingress("domain.com") >> Service("svc")
    net >> [Pod("pod1"),
            Pod("pod2"),
            Pod("pod3")] << ReplicaSet("rs") << Deployment("dp") << HPA("hpa")

diag_1.dot.save("diag_1.gv")

from diagrams.aws.compute import EC2
from diagrams.aws.database import RDS
from diagrams.aws.network import ELB

with Diagram("Grouped Workers", show=False, direction="TB") as diag_2:
    ELB("lb") >> [EC2("worker1"),
                  EC2("worker2"),
                  EC2("worker3"),
                  EC2("worker4"),
                  EC2("worker5")] >> RDS("events")

diag_2.dot.save("diag_2.gv")

2. load dot files and convert to Digraph

from graphviz import Digraph, Source

diag_import_body1 = Source.from_file(filename="diag_1.gv", format="png").source.split('\n')[1:-2]
diag_import_body2 = Source.from_file(filename="diag_2.gv", format="png").source.split('\n')[1:-2]

diag_import1 = Digraph(body=diag_import_body1)
diag_import2 = Digraph(body=diag_import_body2)

3. Create final Diagram with each Digraph in a separate cluster

with Diagram("", show=False, ) as out_diag:
    with Cluster("1", ) as c1:
        c1.subgraph(diag_import1)
    with Cluster("2", graph_attr={"direction":"TB"}) as c2:
        c2.subgraph(diag_import2)

out_diag

[jobinjosem1]

@clayms Do you know if it is possible to connect the two clusters in the final output? Something in the above example like c1 - Edge(color="red") - c2

[clayms]

@jobinjosem1 I think you would have to add some blank and zero size nodes to the final clusters. Then do something like what is shown here: https://github.com/mingrammer/diagrams/issues/17#issuecomment-723564748

[ristillu]

I'm new to Diagrams. It looks awesome. I am one of the many people who it seems could benefit from a hierarchical breakdown of architecture components into separate files. Does anyone know if it's possible to take something like the solution in here and specifically the part here:

with Diagram("", show=False, ) as out_diag:
    with Cluster("1", ) as c1:
        c1.subgraph(diag_import1)
    with Cluster("2", graph_attr={"direction":"TB"}) as c2:
        c2.subgraph(diag_import2)

out_diag

but instead modifying it a bit like:

with Diagram("", show=False, ) as out_diag:
    with Cluster("1", ) as c1:
        c1.subgraph(diag_import1)
    with Cluster("2", graph_attr={"direction":"TB"}) as c2:
        c2.subgraph(diag_import2)

    SomeNodeType("foo") >> c1 >> c2 >> SomeNodeType("bar")

That is, I would like to build up a diagram with the parent diagram having relationships between the subgraphs that I'm loading in as per @clayms' excellent example. And ideally without delving into the graphviz data structures and iterating through their internals looking for things created by other files that I don't want to open.

The motivating use case is one or multiple people working on a large service oriented architecture where a parent document ties everyone's individual pieces of the overall architecture into a cohesive diagram at the top level.

[zingagent]

For arrows to clusters see #17

Otherwise, I approach the compartmentalisation of the diagrams differently - via python functions.

Here are 3 files to show how I handle it.

First - two independent diagrams

aws.py

from diagrams import Diagram
from diagrams.aws.compute import EC2
from diagrams.aws.database import RDS
from diagrams.aws.network import ELB

def aws_diagram():
    lb = ELB("lb")
    workers = [EC2("worker1"),
               EC2("worker2"),
               EC2("worker3"),
               EC2("worker4"),
               EC2("worker5")]
    events = RDS("events")

    lb >> workers >> events

    return lb, workers, events

def main():
    with Diagram("Grouped Workers", direction="TB", show=True) as aws_diag:
        aws_diagram()

if __name__ == "__main__":
    main()

k8s.py

from diagrams import Diagram
from diagrams.k8s.clusterconfig import HPA
from diagrams.k8s.compute import Deployment, Pod, ReplicaSet
from diagrams.k8s.network import Ingress, Service

def k8s_diagram():
    net = Ingress("domain.com") >> Service("svc")
    pods = [Pod("pod1"),
            Pod("pod2"),
            Pod("pod3")]
    rs = ReplicaSet("rs")
    dp = Deployment("dp")
    hpa = HPA("hpa")
    net >> pods << rs << dp << hpa

    return net, pods, rs, dp, hpa

def main():
    with Diagram("Exposed Pod with 3 Replicas", show=True) as k8s_diag:
        k8s_diagram()

if __name__ == "__main__":
    main()

Then tie everything together

combo.py

from diagrams import Diagram, Cluster
from diagrams.onprem.client import User, Users

from aws import aws_diagram
from k8s import k8s_diagram

with Diagram("Combo Diagram") as combo_diag:

    with Cluster("1", ) as c1:
        lb, workers, events = aws_diagram()

    with Cluster("2", ) as c2:
        net, pods, rs, dp, hpa = k8s_diagram()

    foo = User("foo")
    bar = Users("bar")

    foo >> lb >> bar
    foo >> net >> bar

So here are three diagrams from these files

I also make use of parameters to the functions. So I might pass in a thing to be linked to the component in the diagram. One overall diagram might pass in something high level, so the diagram overall can focus on what is in the called function sub-diagram within a high level overview around it, or another diagram might pass in some object more specific, where the sub-diagram needs to be seen connecting to the details of the wider diagram.

Hope this make sense.

[ristillu]

Amazing, thanks @zingagent