search

DataDog / datadog-operator

Kubernetes Operator for Datadog Resources
Apache License 2.0
287 stars 99 forks source link

SLO burn rate #1022

Open melchiormoulin opened 7 months ago

melchiormoulin commented 7 months ago

Hello, since there a customressource to define SLO https://github.com/DataDog/datadog-operator/pull/807 How can we define burn rate alert based on this SLO CR ? It seems there is some internal discussions ( this comment https://github.com/DataDog/datadog-operator/pull/807#issuecomment-1781049308 )

vlinevych commented 7 months ago

I've found an example of slo alert, but the problem is that it requires slo-hash-id in the query.

For burn rate alert the query would be something like this:

query: "burn_rate(\"slo-hash-id\").over(\"30d\").long_window(\"1h\").short_window(\"5m\") > 14.4"

It would be great to have slo burn rate alert implemented in the operator.

melchiormoulin commented 4 months ago

Hello do you think there is any chance to see this feature please @fanny-jiang @ddjamesfrullo ? :)

jurabek commented 4 months ago

There is possibility to create CRD only for burn rate and pass those values as attributes instead of hard-coded string query: "burn_rate(\"slo-hash-id\").over(\"30d\").long_window(\"1h\").short_window(\"5m\") > 14.4" value, and this gives us referencing parent SLO into this alert more natively, k8s operator does provide such functionality I will try work on that, in my free time.

melchiormoulin commented 1 month ago

Hey do you think it can be implemented ? Maybe @jurabek ? It would be super nice :)