search

grafana / grafana

The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.
https://grafana.com
GNU Affero General Public License v3.0
65.34k stars 12.19k forks source link

Diff() and percent_diff() not working - data not in sorted time order #37216

Closed sarahmarshy closed 3 years ago

sarahmarshy commented 3 years ago

What are you trying to achieve? Use the diff() function for alerting

How are you trying to achieve it?
Eval montonic increasing counter every 1m for 5m. Evaluate points from (1h, now). Get the diff and check if there has been any change. IE, diff() > 0.

What happened? diff() gets negative values when I know it’s not appropriate

What you expected to happen:

I checked the code. In summary: it gets the first and last elements of the data array and diffs them.

It does seem to be doing what I expect with the logic above. IE query(t1) - query(t0) , except t1 and t0 here are not as expected, they are random order. The test data looks like this:

0:Array[15172,1626882120]
1:Array[15192,1626882900]
2:Array[15257,1626884820]
3:Array[15159,1626881460]
4:Array[15170,1626881940]

Clearly Point 3 and 4 occurred before 0 and 1?

How to reproduce it (as minimally and precisely as possible): Create a diff() or percent_diff() alert on an constantly increasing data source and view test data.

Anything else we need to know?:

Environment:

zuchka commented 3 years ago

Hi @sarahmarshy,

Thanks so much for making this issue. I see that you are running Grafana 6.7.4. Do you think you could try reproducing this issue on a newer version? We are on Grafana 8 now, with 8.1 due out in the coming weeks.

I ask because there are numerous old issues about this topic, and many from around the time of Grafana 6.

https://github.com/grafana/grafana/issues/16562 https://github.com/grafana/grafana/pull/21338 https://github.com/grafana/grafana/pull/21337/files

Both alerting and the panel editor have undergone serious transformations since g6. Alerting was dramatically overhauled for g8.

so I'm curious if this behavior is still present...

sarahmarshy commented 3 years ago

@zuchka, thanks so much for responding. I can try to get a newer version running locally and reproduce if that's helpful.

sarahmarshy commented 3 years ago

@zuchka, I looked at a few of the local gdev basic data sources on the v8.0.6 branch, and it does appear to be correct for what I looked at. An example:

0:Array[200,1627506504721]
1:Array[445,1627506554721]
2:Array[100,1627506604721]
3:Array[150,1627506654721]
4:Array[200,1627506704721]
5:Array[220,1627506754721]
6:Array[190,1627506804721]

Time is in increasing order. Result of diff() is -10, which seems correct. Percent diff is -5%.

We are looking into upgrading 8.

Out of curiosity... is 6 not being maintained at all anymore?

zuchka commented 3 years ago

Hi @sarahmarshy

So glad upgrading can get you unblocked here. G8 offers substantial improvements.

And I will have to confirm where we are at with maintaining Grafana 6 / EOL policy 👍