maplelabs / opensearch-scaling-manager

0 stars 11 forks source link

[BUG] OpenSearch scaling manager is rebalancing the shards before complete installation #512

Closed Akhil-Nair-ML closed 1 year ago

Akhil-Nair-ML commented 1 year ago

Bug Report Checklist

Scale up should be successfully completed then only shard rebalancing will start. As of now the node is getting added and shard rebalancing starts, before the custom role is executed.

Description

OpenSearch scaling manager is starting the OpenSearch while perform a scale out, before the custom role is executed. If the custom role execution fails then the newly added node will be terminated. Repetitive failures can create imbalance in cluster, shards may be lost due to termination of instance and cluster can go to unstable state(red).

Steps to reproduce
  1. Simulate rule to Scale out the cluster
  2. Simulate a failure in execution of custom roles.
  3. The node will get added and shard rebalancing will start before execution of custom roles.
Related issues/PRs

https://github.com/maplelabs/opensearch-scaling-manager/pull/511

Suggest a fix

Move the logic of starting the OpenSearch after the custom role execution is successful.