search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.35k stars 7.98k forks source link

[ResponseOps] rules runs recovered actions without ever running active actions #182888

Open pmuellr opened 1 week ago

pmuellr commented 1 week ago

We've seen some cases where rules will run the recovery actions, but we're not seeing any of the active actions being run. Like they've been lost, or perhaps there weren't any active actions set up. But, that doesn't seem to be the case with some of the investigation we've done.

We believe this may be an issue with the "alert after x consecutive matches".

From Ying:

It is set to alert after 5 consecutive matches. If there were not 5 consecutive matches no alert would be created but we still track it as an active alert in the task manager state so if it recovers it would trigger a recovery notification ...

We probably need to update the logic to not consider it recovered if it hasn't met the threshold for an active alert

Ersin was able to repro this.

elasticmachine commented 1 week ago

Pinging @elastic/response-ops (Team:ResponseOps)