How about automation / templating

[GGRUser]

In the first place, thank you for this plugin, I can definitely say that this is a great contribution to Grafana.

When having many different parts of network ( lets say 500 just to have a number ) with different numbers of elements ( raging from 20 to 50 ) its a little tedious to add each Node then setting it up etc... It would be nice to have, to configure one node then when adding the others to have a template option of some sorts ( maybe same for links ....).

Tried to add a new dashboard via JSON ( exported an already created dashboard ) and tried to add new nodes but a wild import error occurred after, I suspect that it has something to do with the id's of the nodes. It would be great if when the value is put to "null" then a new ID would be auto-generated. If this could work, and if the right data is available, all the network dashboards could be auto-generated. Any manual draw topology will change relatively fast and the graphs will not follow the changes fast enough :).

NOTE: If it helps you, I can send you any testing feedback on any features of the plugin

Thank you !

[knightss27]

Thanks for using it!

First, I haven't really done any testing at all for this plugin with larger numbers of nodes and links. Realistically it should scale alright, and I have put work into making it more efficient, but you are correct that part of why I haven't tested it is that it would be quite tedious to create that many nodes / links by hand.

I see what you are suggesting with templating (it's part of why there is already an apply all button for node coloring). I think a simple middle ground could be inheriting the last placed node/link's characteristics, so at least when you're making different parts of the network for the first time you can do them all at once with relative ease. If templating was to be part of the program, in what way do you think it should work? i.e. what sort of options and where would you like them to be / what exactly should a template encompass and what should it leave to be set?

Tried to add a new dashboard via JSON ( exported an already created dashboard ) and tried to add new nodes but a wild import error occurred after, I suspect that it has something to do with the id's of the nodes.

Did you try to add nodes programmatically? I hadn't really thought of this but I see how it would be useful for large numbers of nodes/links. I would say I could regenerate node id's like you suggest, but the links keep track of nodes by using their id's, so if any of these nodes were used in links then those links would break. If you can recreate it, I'd be interested in knowing what the error was. Node id's can literally just be any unique string, so if you properly set them all it should just use them?

I'd also be more than happy to let you test versions of the plugin before they get pushed to Grafana, as right now it's just me who goes through some basic tests to ensure things work.

[GGRUser]

First of all, before I continue, again, this tool is awesome ! I don't even have a Facebook account ( probably hater or smth :) ) but this tool made me create an account on GitHub just so I can post something regarding this plugin and try to give you some feedback, I am very enthusiastic about this one. I am thinking that this tool can be optimized from two perspectives: GUI ( For all users ) and JSON ( For the ones that want to go in depth with ...... stuff )

GUI:     From my personal experience, I think users want that the elements they add to be customizable ( colors, forms, icons, size etc..) but when they get to the desired form they only use maximum 3-4 types of "nodes" and "links ". Having this in mind I would put all of these nice features in a collapsed "Weathermap Customization" part of the plugin where you can create how the elements every user desires to use in their Dashboard. ( The template part )

    For the Nodes:

    Add Node Template     -> Name of template     -> Icon     -> Padding     -> Advanced     -> Colors     -> Font size     After defining the 2-5 node templates that you use in creating the map, in the "Network Weathermap" area, for the nodes part you should have:     - Node drop-down with all the nodes     - Drop-down template selection ( so you can select the template for any node you want )     - Label     - Add Node <-> Remove Node     -> The x,y position I think should be completely removed because you can already move the elements on the map and this is transparent for the user.

    For the links:     Add Link Template     -> Name of template     -> Link Stroke Width     -> Spacing     -> Colors     -> Arrow options     -> Link Font Size     -> Link units     -> Enable / Disable features from a-z Sides menu ( For ex if you know that you shall never use with that template "Bw query" or "label offset" )  -> Nice to have     After defining the 2-5 link templates that you use in creating the map, in the "Network Weathermap" area, for the link part you should have:     -> Link drop-down     -> The A Side     -> The Z Side     -> Drop-down template selection     Other Options: -> All Scale Options (including coloring part) -> Tooltip Options -> Grid Options -> "Panel"     These ones should all be be in the "Weathermap Customization" part of the plugin so the working area can be de-bloated.

NOTES:

• These changes would have a great impact on the time needed to create a panel. After making the initial templates I estimate that it will take 70-80% less time for a user to actually create a desired map
• It will add new options in link customization ( every link could have its own template )
• I think that even the JSON code will be a lot lighter because at every node / link you would reference a template not all the settings/options that the node can have

New features in GUI:

"Import map" - for semi-automation purposes

    This would be the most awesome thing for a user. Lets say you have a document with all the info that you need: MapName    NodeA    A-Side        B-Side    NodeB    GGR1            Node1    Q1              Q2           Node2 GGR1            Node2    Q1              Q2           Node3 GGR2            Node4    Q1              Q2           Node5    GGR2            Node5    Q1              Q2           Node6 GGR2            Node4    Q2              Q1           Node6      After setting up the data queries you import this document ( .csv, .xlsx, other format ... ) and poof you have two imported panels: GGR1 -> 3 Nodes with two links GGR2 -> 3 Nodes with three links     Of course, you would have some things to do after... but you already have the map drawn and it will reduce the creation time of the map     Maybe done from a separate part where before the actual import you can select the desired templates for links and nodes

Zoom In/Out in non-editing mode

    I saw that you can move on the map with ctrl + rightclick but i did not find the option to zoom in/out. If you draw a large map whatever settings you put, a user that sees the map will face the issue of not seeing the labels or the hole map in the panel...

Clicking a link that has a "Dashboard link" set

    This should definitely open in a new tab because when you investigate a issue on a network area you want to click on multiple links on the area to view detailed information. If you leave the initial map page then get back etc.. it's tedious at least.

Node link

It would be useful to have the possibility to put a link on the node so it could get you to that elements dashboard.

JSON     -> After writing all that part about the GUI, i have to re-mention this in this JSON part: " I think that even the JSON code will be a lot lighter because at every node / link you would reference a template not all the settings/options that the node can have"     -> After you told me that the id field has no specific allocation algorithm or significance I have managed to successfully export a 2 node dashboard, add another node and then imported it back with 3 elements available so it is working. Having this in mind, it is completely feasible to draw the map from JSON.     -> I maintain my observation with the "null" option in the JSON so you can add 50 identical elements without being forced to edit characters the id field of the node element. After you have the nodes, you can manually enter the links ... ( at this point, without any templating options from GUI, it would be useful )

I do have a question: How does this part of the code work ? I unfortunately did not get the logic behind it.

“{ "anchors": { "0": { "numFilledLinks": 0, "numLinks": 0 }, "1": { "numFilledLinks": 0, "numLinks": 0 }, "2": { "numFilledLinks": 0, "numLinks": 0 }, "3": { "numFilledLinks": 0, "numLinks": 0 }, "4": { "numFilledLinks": 0, "numLinks": 0 } },”

[GGRUser]

I will try to create a script that will auto-generate the JSON code for what I need, in this process, I will make a document identifying all the variables in the code and share it with you. Maybe others will use it as a reference

[knightss27]

First, let me say thank you for so much feedback! It's actually really great to hear some user perspective. I'll try and respond to what I can.

First, GUI:

From my personal experience, I think users want that the elements they add to be customizable ( colors, forms, icons, size etc..) but when they get to the desired form they only use maximum 3-4 types of "nodes" and "links ".

This makes sense to me. I see where you're at with the use of templating and am quite thankful you suggested it as I had never thought about how useful it would be. Your description of what would be in a template is also agreeable.

I think that even the JSON code will be a lot lighter because at every node / link you would reference a template not all the settings/options that the node can have

In fact, I think referencing templates from within the nodes/links (instead of having specific data for all nodes/links) would make templating even more useful, as you could edit a template and change existing nodes/links (though I would keep an option to disconnect nodes/links from templates if you needed to alter a specific one).

Enable / Disable features from a-z Sides menu ( For ex if you know that you shall never use with that template "Bw query" or "label offset" ) -> Nice to have

I understand your thinking here, though I would argue that toggling these options might honestly be more annoying and less useful than just hiding them in a dropdown, though I guess if the toggles are on each template it could work.

For the "importing" feature:

After setting up the data queries you import this document ( .csv, .xlsx, other format ... ) and poof you have two imported panels:

I am wondering how you think the weathermap should go about placing these nodes, or would coordinates also be specified by the user? (as they are not present in your example). I know you can do force-directed graphs, but I am quite averse to the idea due to it's inherent complicated nature, and most people will want to re-arrange their graphs to suit their needs.

I maintain my observation with the "null" option in the JSON so you can add 50 identical elements without being forced to edit characters the id field of the node element. After you have the nodes, you can manually enter the links ... ( at this point, without any templating options from GUI, it would be useful)

I guess it could do this, though if you're already the type of person to generate the JSON yourself, it can't be that hard to add some numbers!

Link anchors...

I do have a question: How does this part of the code work ? I unfortunately did not get the logic behind it.

Since each node has 5 anchor points, we need to know how many links are attached to that point (and this logic saves us attempting to calculate this every render, though it would be only a single run through all links). This data is stored in the "numLinks" key. Second, when we actually render the nodes, there is a lot of calculation to decide how wide/tall/padded a node needs to be (taking into account text size, border width, horizontal and vertical links, link spacing, whether we collapse them vertically). We use the number of links to calculate the sizes. Further, when we are drawing each link, we need to know where to place it. Each link does not know whether it is the 1st, 2nd, 3rd, etc. one attached to any specific anchor, so instead we keep track of how many we've drawn attached to that anchor (and since links are rendered separately, their drawing order doesn't follow a clockwise pattern around the node or anything) and when it's time to draw that link, we check the associated node, see how many links we've drawn at that anchor, and subsequently draw the current link as attached to the left, middle, right, far right, etc.

It's complicated but it works! And saves a bunch of recalculations in the middle of rendering.

Let me know how generating the JSON goes (or if you have any more suggestions/questions), and in the meantime I will write out for myself a more robust plan for template implementation, as I think this is really a great idea.

[GGRUser]

In fact, I think referencing templates from within the nodes/links (instead of having specific data for all nodes/links) would make templating even more useful, as you could edit a template and change existing nodes/links (though I would keep an option to disconnect nodes/links from templates if you needed to alter a specific one).

At any point, if you want a specific edit you can just create a new template for what you want to do and just apply it on the elements (even on one element only). I don't think that you need to complicate by make/keep a disconnect option.

Enable / Disable features from a-z Sides menu ( For ex if you know that you shall never use with that template "Bw query" or "label offset" ) -> Nice to have

I understand your thinking here, though I would argue that toggling these options might honestly be more annoying and less useful than just hiding them in a dropdown, though I guess if the toggles are on each template it could work.

Again, this is just in a "nice" category, maybe just if have nothing else to do :)

For the "importing" feature:

I am wondering how you think the weathermap should go about placing these nodes, or would coordinates also be specified by the user? (as they are not present in your example). I know you can do force-directed graphs, but I am quite averse to the idea due to it's inherent complicated nature, and most people will want to re-arrange their graphs to suit their needs.

At this point, I see the feature concentrating on importing the data and auto-creating the elements. Of course the tool cannot be created in so that the elements would arrange in such a way that every user wants. Maybe I see an option that can let you align "in grid", "in line", "star" or some other that I am not thinking of right now. After rescuing 6 hours of clicking ( for 30 elements) by auto-importing the elements, I think that 5-10 min could be allocated to arrange the map as every user prefers. The coordinates is a good idea but should be optional, if don't have them... well just leave blank and the app will arrange you the nodes in a "grid", "line" etc...

I guess it could do this, though if you're already the type of person to generate the JSON yourself, it can't be that hard to add some numbers!

true, true

Since each node has 5 anchor points, we need to know how many links are attached to that point (and this logic saves us attempting to calculate this every render, though it would be only a single run through all links). This data is stored in the "numLinks" key. Second, when we actually render the nodes, there is a lot of calculation to decide how wide/tall/padded a node needs to be (taking into account text size, border width, horizontal and vertical links, link spacing, whether we collapse them vertically). We use the number of links to calculate the sizes. Further, when we are drawing each link, we need to know where to place it. Each link does not know whether it is the 1st, 2nd, 3rd, etc. one attached to any specific anchor, so instead we keep track of how many we've drawn attached to that anchor (and since links are rendered separately, their drawing order doesn't follow a clockwise pattern around the node or anything) and when it's time to draw that link, we check the associated node, see how many links we've drawn at that anchor, and subsequently draw the current link as attached to the left, middle, right, far right, etc.

It's complicated but it works! And saves a bunch of recalculations in the middle of rendering.

Thank you for the detailed info, i think i got it :). It will help me when I will try to generate the JSON. Will keep you posted on this, hope I will have a functioning method until the end of next week.

[knightss27]

If you'd like to mess around with the newest patch, you can find it here: https://github.com/knightss27/grafana-network-weathermap/releases/tag/v0.3.2

Just wanted to mention it, though of course I've yet to implement the templating (it's next on my list). Let me know if you find any bugs, though I will probably submit it for publishing by Grafana in 24hrs.

[GGRUser]

I had a little time to test, managed tor test v 0.3.3 now and everything seems ok at the moment. If I find anything I will give you a note...

[knightss27]

No worries, Grafana made it the live version and I haven't had any recent bug reports so it seems to have worked out well. I'll let you know when I get to implementing templating so you have the opportunity to play around with that!

[GGRUser]

I managed to generate a small map ( 15 Nodes and the links between them ) only using JSON ( Manual editing), a step forward I would say. Is there a possibility to implement "bend" on the links, this way, all the links would be connected in "Center" and all the "Anchors" part could disappear entirely. A spacing of 25-30 between the middle of the links I think would suffice.

[GGRUser]

Can you please tell me what this field does "numFilledLinks": 0 ? I tested it in every way I could think of and did not found its purpose :).

[dardok]

First of all - thanks for this plugin. Posting to this issue as I have a few comments regarding my efforts at templating the JSON dashboard/panel. Would also like to share that it was of great value in our network demonstrations at the SuperComputing 2022 conference last week in Dallas:

Here are links to the public releases where you can see more as to how it was used in action:

Twitter LinkedIn Facebook

Some 40+ nodes and almost 100 links, using just about every feature of the plugin. Every link gets throughput queries from InfluxDB, bandwidth-as-link-width from PostgreSQL (100G, 200G, and 400G!), and hyperlinks to detailed graphs. Every node gets status from Prometheus, and layout comes from the templating engine (including top/left/bottom/right anchors, more below). It was really impressive eye-candy and at the same time a very useful high-level view of the experiments being conducted between Dallas TX, Berkeley CA, Chicago IL, McLean VA, and Washington DC.

The templating engine was a simple Python/Jinja2 renderer pulling data from an existing PostgreSQL database that was also used to provision and configure many of the nodes and switches in the network (i.e. a Single Source of Truth DB). Four templates: dashboard.json.j2, target.json.j2, node.json.j2, and link.json.j2 were populated by dicts built from queries to the DB, in this case producing an almost 1MB dashboard.json for import into Grafana.

One of the biggest challenges was handling the anchor counters on nodes as well as links. The renderer would use the angle between the connected nodes' coordinates and arrange the top/bottom or left/right anchors to reflect the appropriate relationship. However, it has to essentially go through the entire graph twice in order to compute the numLinks data element in the anchors field. (Per the previous comment, I also found no present use of the numFilledLinks data element). As you explained above, I can tell from experimenting as well as glancing through the code that the numLinks value is crucial for the auto-sizing of the node as well as the relative position of each link.

I suppose as a feature request, I wonder if there's a way that "on load", the plugin could basically do a similar pass through all the sides['A', 'Z']['anchor'] elements and fixup the numLinks values so a templating engine wouldn't have to do it - as well as being a kinda sanity/integrity enforcement on the data. Once it's loaded, it looks like the code only handles relative changes (i.e. +1 or -1) on the counters as the panel is being edited through the Grafana UI. As you noted, recomputing on each render would be relatively expensive (as it stands, with this many nodes and links and queries it's already starting to tax the browser on even a pretty beefy client system), but maybe just once not so bad?

In the same vein, perhaps an "auto anchor point" option on the plugin could do the angle calculation and set top/bottom and left/right automatically on load? And perhaps also at editing time, when the user drags a node around it "fixes" the anchor side? One more might be auto-staggering the link label given the number of links on that side, I used a simple #/N*100 calculation that did pretty well, especially when all links on a side were between just two nodes.

I'd also like to +1 for several suggestions in this comment - specifically "Zoom In/Out in non-editing mode", "Clicking a link that has a "Dashboard link" set opens new tab", and "Node link" would all be great features to have - but I don't think those are really related to templating and probably warrant their own issues to be tracked.

Given that I've done what I've done (and unfortunately can't share directly as the code is technically not approved for public release - though I think it's pretty obvious given my explanation above), I don't really need much by way of the "on load" features at this point - but it might save future templaters some effort and just let them focus on the layout. I do agree that a full auto-layout is likely beyond the scope of the plugin - there are just too many ways to do that and none of them are particularly satisfying - I tried several in the template engine side and ended up still pretty much laying everything out by hand. There's a hundred ways to skin that cat outside of the plugin itself...

Thanks again, and mostly just wanted to share a success story and experience - look forward to an even bigger, faster, stronger network weathermap at SC23 next year (wherever it may be!)...

[knightss27]

Wow! This is really great to read about, and thanks for showing me your usage. That's definitely the most intense usage I've heard of with the plugin, and I'm really glad it managed to hold up at least a little!

I wanted to mention that I have for a long time been worried about performance issues with large numbers of links and nodes. Grafana is already not particularly efficient itself with render calls (I have had to manually cut down on these in the plugin). I'm happy to see that it still manages to work to some degree with such a large map, but I definitely need to rethink some of the more taxing parts of the application. I've actually tried a lot of different methods of memoizing components and function calls previously, but I've never gotten it to work as well as I wished.

re: auto-calculating numLinks and anchor points. I put off this thought a while ago given it felt out of the scope of basic priorities for the plugin, but given the current state and your mention of this I'm actually thinking it might be something to focus on adding. As you say, trying to calculate angles is probably the best way to do this, either on load or with an auto-calculation function. This would certainly make the plugin much easier to use (which goes along well with templating). I will put this on my timeline. I suppose it may even be worth it to write up a small part of the wiki that talks about possibilities with automating the dashboard's construction as well.

Zooming outside of edit mode, new tab dashboard links, and node links will all be released with the next patch (v0.3.4).

Thanks again for using the plugin, and for telling me about it! Please let me know if there is anything else you feel would be worth adding or improving!

[Ryad3]

Hello,

First of all thank you for your works. I'm writing here because i had the same issue, i wanted generate weathermaps because with a lot of nodes, it becames difficult. So I made a script in python which is able to generate a json that i just have to import. So first i just manually made a weathermap just to have a template of the json and understand how it works and manage to add links and nodes automatically. I dont know if you start to work on it but if you are interested feel free to contact me.

Thanks again !