search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.71k stars 8.13k forks source link

[ResponseOps][Alerting] do not update alerts or run actions, for rules disabled after they start running #192300

Open pmuellr opened 1 week ago

pmuellr commented 1 week ago

We recently had someone ask us what happens when a rule is disabled WHILE it's running.

We do have a check at the beginning of the rule run: https://github.com/elastic/kibana/blob/37dff6429ed77c028ec2c03328ed4b14205f25c3/x-pack/plugins/alerting/server/task_runner/rule_loader.ts#L64-L72

However, I don't see anything beyond that, and most importantly, after the rule runs and is processing alerts and actions.

I did happen to remember that we have some processing to not generate alerts / run actions when a rule times out, added in this PR: https://github.com/elastic/kibana/pull/114289

Feels like we want to do that for rules that we notice are disabled AFTER the rule runs but before alerts/actions are processed. Urg, another i/o!!! :-)

elasticmachine commented 1 week ago

Pinging @elastic/response-ops (Team:ResponseOps)

pmuellr commented 5 days ago

During our triage, Mike noted that this will probably require an additional i/o to read the rule SO, to check if it's disabled. So it might not be worth it.

I think I'd agree - if we can't find an existing i/o where we could make this check, before running actions and updating alerts, it's probably not worth it.