search

raintank / worldping-api

Worldping Backend Service
Other
25 stars 18 forks source link

after-the-fact alerting runs, in case alerting system was down #13

Open woodsaj opened 8 years ago

woodsaj commented 8 years ago

Issue by Dieterbe Wednesday Jul 22, 2015 at 10:33 GMT Originally opened as https://github.com/raintank/grafana/issues/356

this grew out of #291, basically please read from https://github.com/raintank/grafana/issues/291#issuecomment-122492681 onwards to participate in this ticket. I think we should support backfilling old alerting jobs, but not neccesarily send critical notifications. also this is not high prio.

woodsaj commented 8 years ago

I do not think this should every be implemented. I think the disagreement is based on interpretation of what the alert outcome metrics represent.

To me they represent the outcome of the check. So if a check doesnt run then there rightfully should be gaps in the metrics. When i look at the metrics I think "at this point in time, this is what the alerting system thought the state of the endpoint was".

However if you treat the alert outcome metrics as the state of the endpoint being monitored, then it does make sense to backfill data so that you can accurately calculate percent Uptime.