search

pyrra-dev / pyrra

Making SLOs with Prometheus manageable, accessible, and easy to use for everyone!
https://demo.pyrra.dev
Apache License 2.0
1.2k stars 108 forks source link

Add support for Prometheus offset modifier #1237

Open fstr opened 1 month ago

fstr commented 1 month ago

Add support for Prometheus offset modifier in indicator metric expressions.

Type: feature request

AWS CloudWatch ELB metrics are reported with a 5-10 minute delay. A Prometheus recording rule is evaluated at the current time, at which you will never have AWS ELB metrics in your Prometheus.

This can be solved by applying an offset 10m to the metric. All PromQL statements that use the recording rules don't need to have the offset applied. When the offset is used in the metric for the recording rule, everything is already recorded with the defined offset.

The offset is not great for fast burning alerts but I prefer to have a slightly delayed but working SLO over no SLO. There can also be other use cases where users benefit from applying a (shorter) offset.

metalmatze commented 4 weeks ago

I would prefer not to add this to the SLO configuration, to be honest. Adding more features than simply selecting the time series by their matchers might open pandora's box.

Do you think that using a recording rule somehow could solve the problem? That way Pyrra could stay rather simple.

fstr commented 3 weeks ago

Thanks for your feedback. I will try by defining a recording rule first and passing the recording rule into Pyrra. I will report back.

Is that also your suggestion for uninitialized counters? We have a couple of counters that are difficult to initialize in the application that emits them, due to the high cardinality. A known solution to that seems to be to join in a default value with or up{job="my-job"} * 0 as described on the Prometheus blog.

metalmatze commented 3 weeks ago

Sounds like a solution and @juliusv is one of the most intelligent people. Please try it out!