Allow iframes as Statistic Widget (Grafana Image performance)

[herbetom]

It would be really nice if additional to the Pictures/Images for nodeInfos, linkInfos and globalInfos it would be possoble to use iFrames.

Is your feature request related to a problem? Please describe.

Since Grafana 7.0 with the new image renderer plugin (Read more about this plugin here) the time needed to display the image has increased quite a bit.

Describe the solution you'd like

Adding support for iframes as statistic widget. This would allow to offload rendering to the client and also use zooming etc.

Describe alternatives you've considered

Continue to wait for images to load, i don't know :thinking:

[xf-]

I removed the iframe for a reason. The problem is every stats loads in a extra iframe and initializes complete angular. One small devices or mobile. This also increases the ram usage and loading time of the node.

I understand your request opened your instance (3 sec) and compared to ours (2 sec) to load the images.

After analyzing some requests i think the issue is grafana. It stars everything like in a normal browser and nothing really optimized with real big overhead (like the most requests for the meshviewer) and that results in a horrible performance. Your Client stat has 3 querys compared to our 1 query. That results in more loading time in normal browser.

Edit: Compare image renderer vs widget 2 seconds

2.5 seconds (slower with map and multiple instances)

Result Replacing makes the results even more inconsistent in a browser. You can cache images in nginx and set a header - update in background

Example/Test https://stats.ffrn.de/render/dashboard-solo/db/nodes?panelId=2&width=650&height=350&var-Node=a42bb0d17032&theme=light&_t=2105202020030 renders https://stats.ffrn.de/d-solo/000000002/nodes?orgId=1&panelId=2&width=650&height=350&var-Node=a42bb0d17032&theme=light&_t=2105202020030&refresh=1m

In our instance Max Ram usage 45-50 MB for 1 instance that times 4x in our case. mean 200MB RAM on client side in browser. It feels fast in Grafana because the JS is already loaded.

Next issue is iframe would need height settings ...

PS: Your Grafana has no gzip or brotli for js and transfers 9.4 MB mainly JavaScript

[herbetom]

Thanks for the quick answer and analysis! I was thinking iframes could be an easy solution for better performance. But you're right. The resource usage on slower clients is suboptimal with this approach.

You can cache images in nginx and set a header - update in background

I'm not sure if that wouldn't be a big waste of server resources or if that would even be possible. If you had to generate 650 Nodes x 2-4 images x 3 seconds and that every x minutes, ...

Next issue is iframe would need height settings ...

I was about to ask if the picture could be requested in the used size, because the scaling in the browser for Diagramms is kinda "meh" :D But then i better don't do that ;)

PS: Your Grafana has no gzip or brotli for js and transfers 9.4 MB mainly JavaScript

Thats the problem with fast connections ... You don't notice that. While upgrading our Server the gzip option got lost. I added gzip. Thanks a lot! At brotli i have to take a closer look how we could use that.

[xf-]

In my opinion Grafana needs to improve that and create an smaller widget without big instance or some server with running instance only switching the view (that's like browsing in Grafana). Chromium is really fast in rendering. I use it in other projects.

Current ways to improve it to reduce the querys per graph. Server is in most cases faster compared to most smaller clients (even on my laptop in chromium. yes you get an loading animation, but that's maybe possible to build into meshviewer).

I didn't mean to warm up all images. I mean most times a view clients get the most views and instead of showing the live image, you show a images 10 minutes old and say request a new version in background. This would also solve the height issue.

Alternative solution would be a widget js. Loaded once and requests Querys. But current Grafana needs 9MB js/css. That could start 4 widgets easily or show on next node.

Maybe link more the Grafana.

I also tried to run the plugin as service. No improvement and also instead of binary from nodejs to use system chromium - no change in speed.

Grafana Demo has same performance if you share it as image https://play.grafana.org/d/000000012/grafana-play-home?orgId=1

Also take a look at License file or code jQuery + Angular 1.x + React = Hell https://grafana.regensburg.freifunk.net/public/build/vendors~app.41bc4b193210d0f899b3.js.LICENSE.txt

[herbetom]

I thought a bit about the whole issue, but i only had the idea of including a whole Dashboard but that would probably look quite awful. Also i thaught about differentiating. For example larger screens could get the iframe (but then their is still the issue that Grafana would need to be loaded multiple times) and smaller (probably slower) devices the image. But that's also quite a bad idea.

I also (again) experimented with diffrent settings for the grafana-image-renderer. Currently i use the clustered mode:

RENDERING_MODE=clustered
RENDERING_CLUSTERING_MODE=default
RENDERING_CLUSTERING_MAX_CONCURRENCY=5

I also experimented with the reusable mode, but that only got me an average improvement of 150 ms. And since it's experimental and i'm not sure how it handels concurrent requests i reverted (for now) back to clustered mode.

btw the issue where the poor loading time is disccused is this one (while i probably shouln't complain to much about 3s if their are people with 15s :sweat_smile:).

I don't think an loading animation would help much in this case.

I guess i have more or less live with it for the moment and hope that the Grafana image renderer Plugin will get faster (while i don't really think so since it probably would require Grafana to get slimmer, and that seems unlikely :thinking:).

[xf-]

@herbetom in my eyes this is not the issue - the renderer is fast and only . The issue is Grafana loading multiple frameworks and starting them. i tried some stuff as well and could improve the loading time, but not by enough.

Possible hacks

• Add the query(s) as prefetch . On slower server the loading takes > 1.5 or more for the app and afterwards the query's are requests and added (as far i understand). If you use an custom server you can also provide for the solo URL another html (some rewrite rules).
• Maybe it is also possible to crate a smaller JS bundle that only works for the renderer. In theory possible and then replace like in idea before.
• Also som other ideas or tools, but nothing really easy. Maybe take a look in chromium browser into profiler/performance and find some bottlenecks.

I did some tests and it worked (got 100 ms down with few small changes), but it requires a renderer outside of the plugin. If somebody starts to remove some JS and build for solo panel renderer it would work.

[herbetom]

Yeah, the problem seems to come down to the problem (like you said earlier) that you can only decide between using ressources on the client or on the server. Both times you have the problem that Grafana just isn't that lightweight.

One thing i just thought about is to keep Grafana open on the server and just load the new url once i need an image. That reduced the time on my desktop from 1.5 (new load) to 0.8 seconds. Or (since reusing quite certainly just creates new problems) you could start Chromium in advance and once something get's requested you just had to load that specific diagramm.

But this seems like the wrong issue to think about that. I agree with you that embedding iframes wouldn't solve the problem and often just worsen the problem.

Thanks a lot for explaining that to me! :wink: :+1:

[xf-]

OK i will close this and i changed the title to avoid duplicates.

1 thing that maybe would help the experience to add a min height to the images. Longer way to scroll and less jump in height. If you not an regular user 1-2 sec is normally no problem. enough info about the node above. I get most annoyance scroll, waiting and scroll again (not even the header is visible anymore).

[xf-]

@herbetom i added a option for the ratio + alt title while loading

[herbetom]

Thanks a lot, i'll check it out later.

[genofire]

Has somebody a alternative to the grafana image rendering already tested? I have found some software for this alread (untested):

• https://github.com/wojciechka/influxdb-dashboard-renderer
• https://github.com/itskoko/prometheus-renderer

[herbetom]

Thanks for the ratio option, it does help. But i'll found that alt title isn't shown in Chrome+Chromium. In that Browser the alt attribute only seems to be shown if the image failed to load and not like in Firefox also during loading. But then that's the "Premium Feature" for Firefox Users :wink:

For my part, i haven't tested renderes which render directly from the db, but it seems to be worth a look.

[xf-]

@genofire looks not so nice. I think the easiest thing would be to build a Grafana without all the unneeded stuff and maybe some kind of preload for the datasource. That could result in something faster as < 1 sec and inject it with external renderer config. Also you need the graphs config twice. I see no legend in the screenshot.

@herbetom maybe a title would be shown or adding some before element in css with content: attr(alt) and place it somewhere. My main priority was to avoid the jumping of the content. You read the title and it jumps again. You need some time to scroll and less idle waiting time

[herbetom]

That it doesn't jumps anymore is definitely good.

Also i added some cache rules in nginx for the rendered images. They are now stored for 5 Minutes before you get a new version. For that i oriented me at the directives of Freifunk Braunschweig. But for that to work i had to remove the TIME Parameter from the URL. Now it seems to work.

[xf-]

you can also use some features likes request in background (nginx ships the old image and requests the new one in background). I added the timestamp to ensure the browser cache won't show old images. You could remove them from nginx cache key or request to the proxy.

[xf-]

If you get sometimes errors, you need to increase the limits

e.g.

[rendering]
concurrent_limit = 10
concurrent_limit_alerting = 5

[yanosz]

I'ven't used grafana’s image rendering for a while, but nowadays it appears to be problematic. Spec says it needs 16 GB RAM at least (https://grafana.com/docs/grafana/latest/administration/image_rendering/). Just me browsing a small (~ 15 node) mesh today, caused load > 12 on a Hetzner cx21 :-/ ... images take seconds to render. That's a lot of machinery for a simple render job. Yet, I'ven't found a suitable limit, that neither overloads the system nor causes ugly error images.

For me, grafana's server side rendering not an option atm ... sometimes, I miss those rrdtool graph days /o\.

ngx / brotli and Influxdb 2.0 have not hit Debian stable, yet. Hence, I started playing with iframes, having one dashboard per node in kiosk mode (i.e. &kiosk) in Grafana. The Patchset is hakish in won't make it into primetime. However, in the spirit of OpenSource I'd like to provide a source for the desperate (or curious) ones to cherry-pick. https://git.freifunk-koeln.de/ff-koeln/meshviewer/-/commit/7f2dde87dab9151dcea2791ee3670a77e024c823

For the moment, https://map.freifunk-koeln.de shows the result. But it's just a dev & test setup and things could be different in the next days.

[xf-]

@yanosz 15 nodes are something i really don't care. It is much more complicated to get a good/ok performance on a low end device with several hundreds or even about 1000 nodes. Like i described before, the issue is if server (known performance) won't render it, the client needs to do it and also reload every minute (sidebar reloads always). About the 16 GB note - RAM is not the issue, CPU/Harddrive is main issue.

If somebody wants to fix the issue - add a mode in grafana with a light wight template and only the needed stuff (excluding the long critical path).

[yanosz]

Thanks for feedback. Regarding grafana: I fully agree. I'm not used to grafana, hence I don't know the motivation for the render pipeline as of today.

@yanosz 15 nodes are something i really don't care. It is much more complicated to get a good/ok performance on a low end device with several hundreds or even about 1000 nodes.

Maybe, it's also worth pointing out, that I use one kiosk (i.e. one iframe) per node, and not an "every stats loads in a extra iframe"-based approach. In consequence an end-device only renders up to one grafana client instance. Of course, this is a tradeoff.

Like i described before, the issue is if server (known performance) won't render it, the client needs to do it and also reload every minute (sidebar reloads always). About the 16 GB note - RAM is not the issue, CPU/Harddrive is main issue.

Thanks for pointing out that the sidebar reloads overy minute - I've overlooked that. Of course, It doesn't make much sense to reload the iframe, because grafana already refreshes internally. I guess, I'll modify my implementation in the next days. Additionally, there's no kiosk feature for links, yet.

Btw. moving from CX21 to CPX (+1 core, ryzen) decreased the load significantly (12 -> 4.0), but the system is overloaded still. Neverthess, having some cores in stand-by just for rendering charts appears to be unsuitable for a freifunk-typically use-case.

If somebody wants to fix the issue - add a mode in grafana with a light wight template and only the needed stuff (excluding the long critical path).

I agree. Unfortunatly, I don't know much about grafana und this appears to be a relativly big task.

[yanosz]

twmic: Link-Kiosk is implemented. The iframes are not reloaded every 60mins.

https://git.freifunk-koeln.de/ff-koeln/meshviewer/-/commit/438a3ea83cddc33ad5f105cd308ad27f3fcb18e2