[BUG] OpenSearch scaling manager is rebalancing the shards before complete installation

[Akhil-Nair-ML]

Bug Report Checklist

• [ ] Have you provided a full/minimal config file to reproduce the issue?
• [x] Have you tested with the latest master to confirm the issue still exists?
• [x] Have you searched for related issues/PRs?
• [x] What's the actual output vs expected output?

Scale up should be successfully completed then only shard rebalancing will start. As of now the node is getting added and shard rebalancing starts, before the custom role is executed.

Description

OpenSearch scaling manager is starting the OpenSearch while perform a scale out, before the custom role is executed. If the custom role execution fails then the newly added node will be terminated. Repetitive failures can create imbalance in cluster, shards may be lost due to termination of instance and cluster can go to unstable state(red).

Steps to reproduce

1. Simulate rule to Scale out the cluster
2. Simulate a failure in execution of custom roles.
3. The node will get added and shard rebalancing will start before execution of custom roles.

Related issues/PRs

https://github.com/maplelabs/opensearch-scaling-manager/pull/511

Suggest a fix

Move the logic of starting the OpenSearch after the custom role execution is successful.