search

newrelic / nr1-slo-r

NR1 SLO-R allows you to define, calculate and report on service-level objective (SLO) attainment.
https://discuss.newrelic.com/t/track-your-service-level-objectives-with-the-slo-r-nerdpack/90046
Apache License 2.0
21 stars 21 forks source link

Alert defined for SLOs (Budget Perspective) #58

Open ricegi opened 4 years ago

ricegi commented 4 years ago

Summary

As SLOs are defined we should think about them in terms of their overall budget. If the attainment objective is 99.98 ... alerting on the rate of budget consumption versus the total amount of time remaining in the time period.

e.g. - Error SLO of 99.5 ... halfway through the measurement period we are at 99.6 attainment - meaning based on a straight line rate calculation for the SLO we are not going to make our time-bound objective.

Desired Behaviour

Alerts defined for SLOs that are sophisticated enough to execute the rate based budget consumption alerting for an SLO

Possible Solution

TBD

Additional context

Use time window and rate of consumption for the alerting context ...

t-morgan commented 4 years ago

Looking around at some examples of others implementing this, it might make more sense to have alerts along the 1h - 3d timescale.

ghost commented 4 years ago

Going to start experimenting with this by adding some combination of remaining error budget %, burn rate, and time to exhaustion at current rate to SLO/R — once I get that nailed down we can tackle alert generation using those components.

ghost commented 4 years ago

Created https://github.com/newrelic/nr1-slo-r/pull/121 to calculate and display budget remaining.

Opened https://github.com/newrelic/nr1-slo-r/issues/123 with a lot more details around what this might look like - do we want to close this card and roll the work into #123?