search

pyrra-dev / pyrra

Making SLOs with Prometheus manageable, accessible, and easy to use for everyone!
https://demo.pyrra.dev
Apache License 2.0
1.21k stars 108 forks source link

Support for existing SLOs present in k8s env from record rules. #116

Open Sunn-y-Arora opened 2 years ago

Sunn-y-Arora commented 2 years ago

Tried the application, nice interface

have a question regarding importing/adding support for existing SLOs into the interface.

eg-> , we have a CI git ops setup for defining SLOs which generate prom record rules in k8s cluster (using sloth, is it possible to visualize those already present SLOs in pyrra interface.

metalmatze commented 2 years ago

Hm. I don't think so. I'm also unsure how we could make that work, actually. Can you share the configuration you can in mind so it's a bit more concrete? That would be helpful, thanks!

Sunn-y-Arora commented 2 years ago

I was thinking in terms of while fetching the data from prom. (assuming, operator must be generating record-rules for SLOs from the spec , interface gathering data via them), so is it possible to give support for fetching data generated from other record-rules ( ( eg. say for sloth generated record-rules , slo:objective:ratio{sloth_service="${service}", sloth_slo="${slo}"} will give SLO objective for service's SLO ) assuming for pyraa, similar kind of expression will be building, was wondering can we add support for changing/adding these labels(sloth_service, sloth_slo) while querying the data .

metalmatze commented 2 years ago

I don't think this is something Pyrra will support this way around as described. Pyrra will always go from raw metric to recording rules by generating the entire pipeline, rather than depending on some external source. Currently, the recording rules for ratio SLOs would look like: Input metric: prometheus_http_requests_total{job="prometheus",handler=~"/api.*",code=~"5.."} Generated recording rules:

prometheus_http_requests:increase2w
prometheus_http_requests:burnrate3m
prometheus_http_requests:burnrate15m
prometheus_http_requests:burnrate30m
prometheus_http_requests:burnrate1h
prometheus_http_requests:burnrate3h
prometheus_http_requests:burnrate12h
prometheus_http_requests:burnrate2d

The naming of these recording rules is based on the Prometheus naming and aggregation best practices.

The downside being that it's hard to build a generic Grafana since we don't know that one specific metric ahead of time. That being said, we could think of a compatibility layer, where, if enabled, would generate the additional recording rules that could map to the recording rules that Pyrra creates already. It would mean doubling the amount of metrics Pyrra creates, if I'm not missing some way this could be done.

metalmatze commented 1 year ago

This has actually been implemented and is a feature called generic-rules. It'll ship with v0.5.0 in the next coming days, along with a Grafana dashboard.

Sunn-y-Arora commented 1 year ago

oh, great to know, will check it out