Scaling the alerting throughput ceiling from 3,200 to 32,000+ rules per minute

Problem Statement

Usage of Kibana background tasks and alerting rules is continuously growing and we are approaching our scalability ceiling of 3,200 tasks per minute

Increase the overall alerting rule throughput by 10x before Jan ‘25

Raise the per cluster ceiling from 3,200 rules per minute to 32,000 rules per minute (10x)
Scale horizontally past the current 16 Kibana node limit
Scale vertically by increasing the per node throughput of 200 rules per minute

Making it possible to attain 10x scale at the framework level (32,000 rules per minute)
Attain 10x scale when alerting rules are optimized to use Kibana and Elasticsearch resources efficiently
- Elasticsearch Query rule, Index Threshold rule, etc are already optimized for efficiency
Attain 10x scale when alerting rules complete running within a few seconds
Work with solution teams to optimize their rule types

1. PoC

[x] PoC to attain 10x alerting throughput (32,000 rules per minute) kibana#182394

2. Solve the horizontal scalability limits by allowing more Kibana nodes to run tasks

[x] Allow multiple task claiming strategies (feature flag) kibana#171677
[x] Create a task claiming strategy that doesn't rely on forced index refreshes and performs search+get+update within Kibana kibana#181325
[x] Create a Kibana discovery service to help assign partitions to the Kibana nodes kibana/issues/187696
[x] Assign tasks to partitions kibana#187698
[x] Assign partitions to Kibana nodes kibana#187700

3. Solve the vertical scalability limits by running more tasks per Kibana node

[x] Implement resource based task scheduling to run more tasks on larger Kibana configurations (memory and CPU) kibana#185043
[x] Adjust default capacity when running in ECH kibana#189117
[x] Set Indicator match rules as ExtraLarge cost kibana#189112
[x] Change poll interval default to 500ms kibana#190059

4. Work items remaining before rolling out to Serverless

[x] Cache partitions calculation in Kibana for 10s kibana#189119
[x] Rename task claimers kibana#190542
[x] Add more functional tests for resource based task claiming kibana#189111
[x] Make some new task manager constants configurable kibana#190734
[x] Fix starvation issue when there are multiple limited concurrency tasks kibana#184937
[x] Fix errors during processing task result that are not shown in metrics kibana#184173
[x] Convert logger.warn to thrown errors (or something alike) in mget claims strategy so serverless metrics picks them up kibana#190082
[x] Update some runbooks in preperation for rolling out mget to serverless response-ops-team#226
[x] Fix errors during marking tasks as running that are not shown in metrics kibana#184171
[x] Create ad-hoc rollout overview dashboard
[x] Performance test at small and large scale on serverless

5. Initial rollout to Serverles

6. Work items remaining before 8.16 feature freeze

[x] Increase the rules per minute circuit breaker from 10k to 32k kibana#190057
[x] Set action tasks as Tiny cost kibana#190542
[x] Move tasks directly from idle state to running (skip claiming) kibana#184739
[x] Skip loading dataView and searchSourceClient services unless necessary kibana#184322
[x] Perform partial updates when claiming and releasing tasks kibana#187704
[x] Cache query delay settings kibana#184321
[x] Cache flapping settings and only load when necessary kibana#149884
[x] Allow list on ECH and Docker some new kibana.yml settings kibana#192183
[x] Modify task manager docs to use xpack.task_manager.capacity kibana#192185
[x] Skip loading the rule's schedule a second time kibana#192396
[x] Partial update tasks after they run kibana#192398
[x] Cache and load maintenance windows only when necessary kibana#184324
[x] Partial update rules after they run kibana#192397
[x] Finish jest, functional, integration tests for mget task claimer kibana#184942

7. Optional follow-ups and optimizations after 8.16

[x] Recurring tasks to be rescheduled based on runAt rather than startedAt when the gap between the two is less than 10 seconds kibana#189114
[ ] Remove Kibana validation for event log documents
[ ] Run CPU profile to determine other CPU-intensive operations
[ ] Run heap snapshots to determine other memory-intensive operations
[ ] Fix the bulk API performance so users can manage 32,000 rules kibana#188558
[ ] Move the task claiming away from using scripted sorting (maybe using ES|QL)
[ ] https://github.com/elastic/kibana/issues/188465
[ ] Review rules per minute limits set on serverless and max number of bg nodes (6)
[ ] Autoscaling bg nodes on ECH, currently scaling to 32 Kibana nodes provides 32 UI and 32 BG nodes (could half the needs)
[ ] Ensure we rotate whenever pulling tasks from the limited concurrency queue
[ ] Continue claiming tasks on conflicts kibana#184940

8. Blogpost