prometheus / alertmanager

Prometheus Alertmanager
https://prometheus.io
Apache License 2.0
6.6k stars 2.15k forks source link

An option to disable the default sorting of Alerts for a group of Alerts #3901

Closed ricardogpsf closed 2 months ago

ricardogpsf commented 3 months ago

Disabling the default sorting of Alerts in a group of Alerts would be helpful.

I saw old discussions in this repo about features like setting what label to use to sort the alerts and the chosen solution to sort alerts by their label values using job and instance as main labels (I can see the sorting code here). But I think having the option to disable the default sorting would be good.

For example, my case is I have a Loki query configured as an alert and I already sort the result by values as I want. Still, alertmanager sorts the result by job, instance, and other labels, if we could disable the sorting the result would be the expected one. That option could be enabled by default and disabling would be a configuration.

What do you think?

Thanks for your time.

grobinson-grafana commented 3 months ago

Hi! đź‘‹ How would you want Alertmanager to sort alerts in such case? Lexicographically by labels?

ricardogpsf commented 3 months ago

By number values, it’s a query of erros counting and the those values are numbers (integers, I think). I already sort the query result by those count values on “Loki side”, so having no sorting in alertmanager side would make sense for that type of cases.

grobinson-grafana commented 3 months ago

Hi! đź‘‹ I'm not sure what you mean by number values? Alerts in Alertmanager are basically a set of labels, annotations, a start time and an end time. That's the full set of data we can sort with, so we could sort newest alerts first as an alternative, for example.

ricardogpsf commented 3 months ago

Sorry, I used the wrong terms. There is a value associated with the alert, no? Not at the alertmanager side, but related to the query we use to fire an alert. Like, in a query that triggers an alert when DB CPU usage is above 80%, we would have a value associated to the alert that could be 95% and we can show it in the alert description. I know that value itself does not go to alertmanager side, but the Alerts in an alert group go in a certain order depending on the query executed, right?! So would be possible to keep the original order that alerts go to alert manage would be helpful without sorting them when they are on alertmanager. Let me know if I'm wrong in the way I understand the alertmanager behavior (probably I'm). Again, Thanks for you time.

grobinson-grafana commented 3 months ago

There is a value associated with the alert, no?

It's not a strict requirement from Alertmanager. But as mentioned, alerts that come from Prometheus need a query that returns some kind of value.

I know that value itself does not go to alertmanager side, but the Alerts in an alert group go in a certain order depending on the query executed, right?!

I'm afraid that's not the case. Alerts in Alertmanger are not ordered, and Prometheus does not send them in a deterministic order. In addition, alerts can come from multiple Prometheus servers at the same time (even in the same groups).

ricardogpsf commented 3 months ago

Yeah, you're right. In my case I was using Loki queries, so I'm able to sort the results.

I'm afraid that's not the case. Alerts in Alertmanger are not ordered, and Prometheus does not send them in a deterministic order. In addition, alerts can come from multiple Prometheus servers at the same time (even in the same groups).

But now I understand the implications of this request and how unviable it is.

I'm gonna close this "issue" request.

I appreciate your time and answers.