search

DigitalShoestringSolutions / PowerMonitoring

GNU General Public License v3.0
0 stars 1 forks source link

Feature/show inactive machines in machine dropdowns #36

Closed tobyaharris closed 4 weeks ago

tobyaharris commented 4 months ago

In Grafana individual dashboards, machines can only be examined if they have been producing data recently. Here I've increased the "recently" threshold from one week to two years.

Motivation was a customer had their solution turned off for 10 days, then moved it to another machine and wanted to compare the data. However, the old data was no longer available! Had to dig into grafana settings to get it back. Why was this window set so short?

tobyaharris commented 4 months ago

Greg hesitates because a long window query could slow the pi down. If the variable query is run every time the dash is refreshed (5s), and it has to look at every data point in the bucket, that could get heavy.

tobyaharris commented 4 months ago

Next step: stress test. Find a deployment that has been running for over a year. Duplicate one of its dashboards. Add this machine variable or extend the query range. See if performance affected. If no significant impact, merge this.

tobyaharris commented 4 months ago

Stress test needed => changes requested (ie inclusion of test results)

tobyaharris commented 4 months ago

In attempting to stress test this, I've found a couple of solutions that have been running for 8 months. Even before adding this, they (2GB) struggle to render that much data (even only a single float timeseries, like temperature). Took 10-20s to refresh 6mo of data without the variable query. Sometimes some panels do not get any data with error context deadline exceeded (Client.Timeout exceeded while awaiting headers) due to 30s timeout.

Adding this variable query, set to update on dashboard load, had no noticeable effect on usage. With it refreshing on every time window change however, it took >30s timeout to refresh and slowed everything down.

Need to test some more - if the variable fails to return results in 30s, no data will appear in any panels.

tobyaharris commented 3 months ago

As this needs more careful testing, which I can't conclude immediately, I'd like to avoid having this block v1.0.0. The one week threshold will do for now.

tobyaharris commented 4 weeks ago

I think a compromise might be needed - too long and the query times out, too short and the user has a fustrating experience. 30d is becoming an unofficial standard in many other solutions.