[Feat]: Enhanced Architecture Diagrams

[papazach]

Problem

Combine your real-time monitoring and troubleshooting needs with an overview Diagram depicting your current setup. This could potentially be done at a room level.

Description

So for example if one had a simple setup with an API, a Message Broker and a DB

The user could create this simple illustration - maybe via draw.io plugins, or we could import known formats like mermaid - and attach to each block the corresponding Netdata node.

1. So the user could end up with rooms having an Architecture diagram on top illustrating the setup to which the room nodes participate, so he could click at a component and navigate to a component's single node view.

2. Also the connecting lines of those blocks could potentially color coded and paired with metrics in real-time. So for example the INCOMING HTTP REQ arrow could be paired with a metric (for example the rate of incoming http requests) and its size boldness and color could change dynamically according to this metric value.

3. Sets of components could be illustrated as single blocks in a room linking to other rooms' Setups. So the user could click in them and navigate to another room that has a more detailed view such as the one shown here

Importance

really want

Value proposition

Architecture Diagrams and Metrics at the same place. People will understand even more efficiently whats going on and how things are connected. For example a bug increase in the incoming request rate will be obviously correlated with an increase in DB operations.

Dev / SRE teams could keep their Docs up to date paired with metrics at the same place. They could create meaningful rooms describing parts or their whole setup.

Proposed implementation

No response

[cakrit]

The idea we had discussed in the past was to automatically generate network diagrams, based on knowledge we can get from the inbound and outbound connections of any system (such diagrams can include traffic info that we can get from eBPF). I can't picture how we'd integrate with an arbitrary diagram a user might add. Any idea what such an implementation would be like?

[andrewm4894]

oh - i like the idea of a user being able to define a group of charts or stuff and sort of tag or label them as relating to some specific "thing" and then being able to relate those "things" together.

Typically i think see stuff where you can say this bunch of stuff is related to this "App" and then can map out the different "App"s in the room in some way.

"App" not quite right - need some more general concept - "Stack" is another one i've seen but still not right. "Collection" could be generic enough so a "Collection" can relate to anything you want - idea being to give user as much flexibility as possible without sort of forcing them into concepts like "App" or "Stack" or "Database".

Alternatively we could go opposite and double down on lover level really specific stuff like "Database", "App", "FE", "BE" and just hope we have a rich enough taxonomy to cover it all (hard to get right i think).

But generally love the idea of making it easy for user to layer on some semantic meaning or structure of what it actually in the room or what are the different "things" (in their head) that are buried within all the charts and metrics.

[andrewm4894]

I sort of have a notion of "Views" in my head where a view is a different set of filters and group by's across some subset of the room.

Each "thing" then can just be a saved view - and the user then can define whatever different "views" they want in the room. One thing i like about this then is we can then easily do something like anomaly rate within each view - so helps them just narrow into different subsets of the room that represent different stuff and they may want to monitor and explore differently.

[papazach]

The idea we had discussed in the past was to automatically generate network diagrams, based on knowledge we can get from the inbound and outbound connections of any system (such diagrams can include traffic info that we can get from eBPF). I can't picture how we'd integrate with an arbitrary diagram a user might add. Any idea what such an implementation would be like?

I understand the technical challenge, I was thinking maybe we could use something like react-diagrams (demo here with multiple examples), then the user could draft his own setup for the room.

Those diagrams can also be encoded into .json format so we could persist them in the DB, retrieve and render them as part of the room information.