Optimize indexing of alerts with concurrent rule execution

[mikecote]

Description

Currently, each rule execution that generates alerts performs its own /_bulk indexing call, leading to frequent and unnecessary index refreshes in Elasticsearch stateless. These forced refreshes (refresh: true) are a workaround to the default 5s refresh interval in stateless. We do this to have faster rule execution times in stateless. However, this approach increases the load on Elasticsearch, particularly during alert bursts, as the refresh throttling mechanism starts activating.

Proposed Solutions

1. Aggregate alerts for bulk indexing

Allow Kibana nodes to aggregate alerts from multiple rule executions into a single bulk indexing request. This would reduce the frequency of forced index refreshes and alleviate the strain on Elasticsearch during high-alert volumes.

2. Manual index refreshing with refresh: wait_for

Introduce manual index refreshes initiated by Kibana every 1s, ensuring alerts are persisted efficiently without forcing refreshes in individual indexing operations. This would align the behaviour more closely with stateful Elasticsearch.

Definition of Done

• [ ] The frequency of forced index refreshes (refresh: true) is reduced during high-alert volumes
• [ ] Alerts are persisted efficiently during high-alert volumes
• [ ] No changes applied when using stateful Elasticsearch
• [ ] Tests

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)