search

pyrra-dev / pyrra

Making SLOs with Prometheus manageable, accessible, and easy to use for everyone!
https://demo.pyrra.dev
Apache License 2.0
1.26k stars 113 forks source link

Nested recording rule generation for different SLO time spans? #1248

Open GrgDev opened 3 months ago

GrgDev commented 3 months ago

I have a use case where we have a ton of metrics data. We want to do SLOs up to a 4 week period, but having those queries regularly query 4 weeks worth of raw data is very compute heavy. It would be nice if the generated recording rules for longer time span SLO queries were based on shorter term recording rule time spans. For example, making the generated 4 week SLO recording rule based on the 1 day SLO recording rule instead of the raw data.

Is this possible with the current implementation of Pyrra?

metalmatze commented 3 months ago

This sounds like Thanos downsampling might be worth looking into. Right now, Pyrra's recording rules take into account all of the 4w of data. That's how the recording rule works. I think there was a discussion at some point to split the 4W queries up, but I can't find it right now, and I don't think it went anywhere.