Is your feature request related to a problem? Please describe
Today, if index is set to index.auto_expand_replicas with upper bound to "all", the cluster will automatically expand to same # of replicas as # of data nodes. This is will easily conflicts with shard allocation awareness if there's an uneven distribution of nodes across zones or hosts, which cause some unassigned shards(cluster yellow state).
Describe the solution you'd like
It will be good to ignore the same hots allocation decider and allocation awareness decider if using auto expand to all feature. As both actually aim to achieve resilience goal of avoiding uneven or concentration of shards on single failure host or zone, with all copies distributed to every nodes, it's already achieve the goal and zone/rack awareness cannot improve the resilience but only create conflict.
peternied
commented
3 months ago
Is your feature request related to a problem? Please describe
Today, if index is set to index.auto_expand_replicas with upper bound to "all", the cluster will automatically expand to same # of replicas as # of data nodes. This is will easily conflicts with shard allocation awareness if there's an uneven distribution of nodes across zones or hosts, which cause some unassigned shards(cluster yellow state).
Describe the solution you'd like
It will be good to ignore the same hots allocation decider and allocation awareness decider if using auto expand to all feature. As both actually aim to achieve resilience goal of avoiding uneven or concentration of shards on single failure host or zone, with all copies distributed to every nodes, it's already achieve the goal and zone/rack awareness cannot improve the resilience but only create conflict.
Related component
ShardManagement:Resiliency
Describe alternatives you've considered
No response
Additional context
No response