In RFC https://github.com/opensearch-project/OpenSearch/issues/12918, we proposed to build shard level splitting without the need to stop write traffic on the cluster. This feature is aimed at solving hot shard or large shard problem which arises mostly in search workload where data is never rolled over and supposed to be available at all times in the same index. Large shards in such workloads is caused typically by usage of custom doc ids by users causing uneven shard sizes or even but large shards on nodes. Shards of the index in such cases continue to grow and eventually become too hot/large to be hosted on the same node. This leads to scaling bottlenecks and therefore, this meta issue defines an overall project plan to dial down on high level tasks required to build the solution of these problems.
[ ] Shard routing changes and new allocation decider - Involves cluster state changes for creating new shard routings for child shards, new allocation decider to support the same, new rest and transport actions, request validations, validation based on existing allocation deciders, etc.
[ ] Building online recovery of in-place shard split - New recovery flow which would consist of creating child shards, replay of translog operations, filtered replication of docs, parent primary to child shards handoff, etc.
[ ] Disk, max shards per node/index, awareness validation changes
[ ] Build cancellation of on-going split recovery flow
[ ] Allocation explain, shard level info, shard stats and recovery info changes
[ ] Ensuring split request failure in mixed cluster and other blocking operations - Block actions like change replica count on the source shard, re-triggered online split, blocks, offline resize, etc while online split in progress. Block adding an external retention lease during in-progress split. Split request to fail while any of these operations are in progress.
[ ] Online Recovery of replicas of child primary shards after split - This may be pushed to phase 2 and phase 1 may involve just splitting source primary and failing parent replicas and letting them recover later after child shards are started.
peternied
commented
4 months ago
Please describe the end goal of this project
In RFC https://github.com/opensearch-project/OpenSearch/issues/12918, we proposed to build shard level splitting without the need to stop write traffic on the cluster. This feature is aimed at solving hot shard or large shard problem which arises mostly in search workload where data is never rolled over and supposed to be available at all times in the same index. Large shards in such workloads is caused typically by usage of custom doc ids by users causing uneven shard sizes or even but large shards on nodes. Shards of the index in such cases continue to grow and eventually become too hot/large to be hosted on the same node. This leads to scaling bottlenecks and therefore, this meta issue defines an overall project plan to dial down on high level tasks required to build the solution of these problems.
Supporting References
RFC : In-place Shard Splitting
Issues
Following is the high level break up of tasks of the project. I will keep linking github issues and PRs as and when they are published.
Related component
Indexing