performance issues with 500k of influxdb data.

[eostermueller]

Hi,

Our influxdb response has about 500k of response data, see attached influxdb-response_02.json. influxdb-response_02.zip

The majority of the influx response is nulls, but I don't have a clue how to make this work without them. Any suggestions? There are about 130 rows in my status map.

I tried this in both chrome (v74 / windows) and IE (11.1087.16299.0CO ) and the results are about the same. The retrieval of the influx response is sub second -- certainly acceptable.

but

• the whole render takes 30+ seconds
• Once its loaded, tool tips do not show up,
• buttons are slow to click, all of grafana is slow,
• vertical scroll bar does respond, but it takes 3-5 seconds to grab it before its move-able. ..and there were no problems before.

Attached is a saved chrome profile session Profile-20190513T141245_b.zip

--Erik

[eostermueller]

[diafour]

Thank you for bug report. In order to reproduce the problem, please, send versions of grafana and statusmap and also statusmap panel options as exported panel json or as screenshots.

[eostermueller]

Perhaps the most helpful tip is this: The sample influx response I posted above has timestamps from 12:01am to 4am-ish on May 9, 2019. The problem still happens if you set a Grafana timeframe OUTSIDE of those 4 hours.

You'll know you've reproduced this when the UI becomes unresponsive....barely alive. Occasionally, it will respond a little, but if you switch to another app and then back, it will go dead again.

StatusMap version: v0.1.1 Grafana version: 5.4.3 statusMapPanelJson.zip

[eostermueller]

brand spanking new to this code, but all this time in addHeatmap() in rendering.js looks suspicious. What do ya think?

[diafour]

Got some time to investigate this issue. It looks like some kind of overhead from d3. One way to eleminate a bottleneck is to precalculate x, y, width, height, and especially a color before render — this can help to avoid multiple scaling operations. So.. Sorry, no quick fix here :( Refactor of render algorithms is planned in https://github.com/flant/grafana-statusmap/issues/53

[eostermueller]

Thanks for having a look. I can't help but think all of the nulls in my influx response are part of what's choking up d3. The influx response is .5 mb of json data -- and if you zip it up, it's just 8k. That's because 99% of everything in the json looks like this block of nulls below.

Any suggestions (you've had a few good influx sql suggestions for me already!!) on tweaking my influx to eliminate these nulls? Would that help anything?

--Erik

null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,null]

[alextdnreb]

Hey,

I'm struggling a bit with the performance as well. My approach was to divide the received data into multiple panels to benefit from Grafana's lazy loading.

However, when every panel is loaded, my dashboard takes up about 3 gb RAM. My idea was to implement virtual scrolling under the use of Intersection Observers, i.e. the heatmap is only rendered when it is inside of the viewport.

Could this be a feature for this plugin? Maybe some "performance-mode"-option, which deactivates hovering effects and activates virtual scrolling. I could provide an implementation for it!