[Task Manager] Task Partitioning - Githubissues

elastic / kibana

Your window into the Elastic Stack

https://www.elastic.co/products/kibana

Other

19.77k stars 8.17k forks source link

[Task Manager] Task Partitioning #187698

Closed mikecote closed 2 months ago

mikecote commented 3 months ago

To remove contention when claiming tasks (encountering 429 conflicts), we should partition our tasks into various partitions and make each Kibana node claim tasks on a subset of the partitions.

Depends on https://github.com/elastic/kibana/issues/187696.

Requirements

[ ] Adding a new partition (number) field to the task manager index (mapped field)
[ ] Assigning a partition to a task when scheduling or bulkSchedule by using murmurhash.v3(id) % 256 with the task ID. See PoC.
[ ] Assign a partition to tasks that don't have one after a task runs (or other)

elasticmachine commented 3 months ago

Pinging @elastic/response-ops (Team:ResponseOps)