Dynamic diagrams

[bhozar]

Is there any way to create a dynamic diagram based on the query? I have an ElasticSearch query which returns network traffic with source/destination IP, port and system name/reverse DNS (where available). Would be great to have the complicated web of communications mapped out dynamically based on that data.

Something like this would be ideal: https://static-www.elastic.co/v3/assets/bltefdd0b53724fa2ce/blt15b0be36731cb379/5ca67d343baf7a5933efa063/kibana-graph.jpg

[algenty]

Hi bhozar, CSV support is in roadmap, with an api url and and checked download options in flowchart. you 'll be able to create dynamic diagram. Example of CSV : https://about.draw.io/import-from-csv-to-drawio/

BR, Arnaud

[bhozar]

Great. If you add that feature in is there any chance you can include the option of querying the datsource and returning results as a CSV? Things like Kubernetes spins up and kills nodes all the time. Being able to visualise it would be brilliant. Same thing with network traffic; adding in nodes to a diagram when they are active it more helpful than displaying all nodes all the time.

[bijwaard]

Hi Arnaud and Bhozar,

Next challenge would be to create mapping rules that can somehow be generalized over all these generated nodes. I can imagine you may not want to create/copy rules and assign all the freshly generated nodes individually. A way to generalize rules would of course be helpful anyway;-) E.g. template rules that can be assigned based on a node property.

Kind regards, Dennis

[xkilian]

Yeah, I have to agree creating that the rules are the most tedious part of the process. Anything to make the experience more intuitive, more streamlined, UI aesthetics, templating, anything. ;-) Great idea to link the template to an object property. And making use of Grafana variables to influence how rules or objects are displayed.

[bijwaard]

Additional to an object property, also the hierarchy/grouping could be used t identify the type of object e.g.:

• all objects that have same parent
• all objects that are connected with arrow to another object
• objects that always have similar parent

Moreover, similar objects often have similar metrics based on their type/role. In my case objects have hierarchical metrics based on their grouping, e.g. the ADC within a sensor produces raw signals and therefore has metric [sensorname]:RS.

⁣Sent from TypeApp ​

On Nov 26, 2019, 06:06, at 06:06, xkilian notifications@github.com wrote:

Yeah, I have to agree creating that the rules are the most tedious part of the process. Anything to make the experience more intuitive, more streamlined, UI aesthetics, templating, anything. ;-) Great idea to link the template to an object property. And making use of Grafana variables to influence how rules or objects are displayed.

-- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/algenty/grafana-flowcharting/issues/87#issuecomment-558462151

[algenty]

Lol guys It's very ambitious. I need an army to devlop it :)

[bijwaard]

Hi Arnaud,

Don't think you need an development army, rather a careful look how the rules can be generalized/templated/scripted, and of course taking incremental steps.

As a first step, it would already help a lot when the metric queries and rules could be generated offline and imported into the flowcharting panel (e.g. with a Python script that writes a JSON file). I already parse the draw.io XML files in Python for configuring and linking up our software modules, so generating metric queries and rules should not be too hard.

Looking at the flowcharting panel exported JSON, I'll have to generate targets for metric queries and rulesData for the panel rules, most critical seems to be the objectIDs in the pattern field of shapeData and textData.

e.g. for RMS measurement in Volt from voltage sensor:

targets[
    {
      "data": "{ \"rtd_len\":3 }",
      "hide": false,
      "refId": "P",
      "target": "VT:rms",
      "type": "timeseries"
    },
]

rulesData[
    {
      "pattern": "/VT:rms:L.*/",
      "unit": "volt",
      "type": "number",
      "alias": "VT:rms",
      "aggregation": "current",
      "decimals": 1,
      "colors": [
        "rgba(245, 54, 54, 0.9)",
        "#37872D",
        "#FA6400"
      ],
      "reduce": true,
      "style": "fillColor",
      "colorOn": "a",
      "link": false,
      "linkOn": "a",
      "linkUrl": "",
      "linkParams": false,
      "textOn": "wmd",
      "textReplace": "as",
      "textPattern": "/.*/",
      "dateFormat": "YYYY-MM-DD HH:mm:ss",
      "thresholds": [
        "220",
        "250"
      ],
      "stringWarning": "",
      "stringCritical": "",
      "invert": false,
      "overlayIcon": false,
      "tooltip": true,
      "tooltipLabel": "",
      "tooltipColors": true,
      "tooltipOn": "a",
      "tpDirection": "v",
      "tpGraph": true,
      "tpGraphSize": "100%",
      "tpGraphType": "line",
      "shapeProp": "id",
      "shapeData": [
        {
          "pattern": "320",
          "hidden": false
        },
        {
          "pattern": "343",
          "hidden": false
        }
      ],
      "textProp": "id",
      "textData": [
        {
          "pattern": "343",
          "hidden": false
        }
      ],
      "linkProp": "id",
      "linkData": [],
      "mappingType": 1,
      "valueData": [],
      "rangeData": [],
      "sanitize": false,
      "order": 34
    },
]

Kind regards, Dennis

[xkilian]

Loading the rule files using YML or JSON would be helpful for charts that have lots and lots of rules. Most certainly not a show stopper, but a nice to have when deploying/creaing dashboards.

The queries are not so bad, as you can modify the raw JSON file to add a bunch of metrics for a dashboard and editing the queries is rather quick compared to editing rules.

[CrashLaker]

@bhozar in case it helps i was playing with graphviz editor to drawio it could be any format (csv, svg, graphviz dotfiles) -> drawio -> xml render -> grafana https://github.com/hbmartin/graphviz2drawio

So I took this from the gallery here

and converted it to drawio mxgraph

then to grafana.

also this was done with kubernetes data (heavily inspired on kube ops view):

More about the though process you can check the thread here: https://github.com/algenty/grafana-flowcharting/issues/145#issuecomment-615970428

For it to be truly dynamic would be good to have some X,Y offset "animations" also here With this we can move objects around.

@bijwaard another solution could be to directly bind the metricAlias from the query Tab to the nodeID (but that might be demand too much cpu as discussed here https://github.com/algenty/grafana-flowcharting/issues/145#issuecomment-616153240)

[CrashLaker]

Inspired by DataDog

[algenty]

😮😮😮😮😮

[xkilian]

These nice layouts that @CrashLaker is creating made me think of:

What if flowcharting itself was data driven and could create some graph types automatically based a layout rule.

For example, the Grafana status dots plugin, it simply lays out dots in a row, based on a single series.

Flowcharting could have a layout rule that is data driven to create a list of dots arranged in a pre-determined selectable pattern. It would be based on one or more series to drive the number of dots and each metric would auto-magically be associated with each dot (graphs, links, transforms, etc.). All other rules could reference the auto-generated IDs from the layout rule and work as they do today to provide additional context.

This would permit from Flowcharting directly to create a number of basic layouts that users could use if they do not want to generate their CSV layout files or manually create the flow chart. Think of auto generated layouts such as hex maps, heat map, dots, swirls, all manner of trees, christmas trees, etc..

If you think it is an interesting enough idea, I will create an issue to track it.

[CrashLaker]

Hi @xkilian

I agree with you that even placing multiple objects on the board is challenging not to tell the next process where we need to bind each metric to its corresponding ID in flowcharting.

That indeed requires a ton of manual work.

For example, the Grafana status dots plugin, it simply lays out dots in a row, based on a single series.

I've never used this plugin but as I've explored a bit about draw.io's CSV these past few days I think we could use boomtable's metrics "syntax" to directly make it Draw.io's CSV compatible and also have the metrics to act upon.

but that's not an easy task and would require us an army like @algenty said :)

I encourage you to create and issue where we can document and discuss the ideas but atm I'm just not skilled enough to hop in and make an strong impact to the project.