Addition of Optional Parameter - allow_Red_cluster to bypass ISM to run on a Red Opensearch Cluster

[skumarp7]

Issue #, if available:

https://github.com/opensearch-project/index-management/issues/1127

Description of changes:

This PR involves the provision for ISM to run on a red opensearch cluster.

Behaviour before:

• The current implementation of the managedIndexRunner job prohibits any calls to the cluster when the cluster is red. https://github.com/opensearch-project/index-management/blob/a049fdb423efcb02fb08a10f3b984fe392d09bab/src/main/kotlin/org/opensearch/indexmanagement/indexstatemanagement/ManagedIndexRunner.kt#L254

• In case if the cluster goes to red state due to resource crunch, ISM cannot delete any indices due to the above constraint.

• Consider each index is of size 1 GB and the persistent volume of the opensearch cluster is 10GB.

ISM on a Green/Yellow Opensearch Cluster:

• When the cluster volume reaches 95% of the available volume, the cluster goes to red state.

ISM on a Red Opensearch Cluster:

Behaviour Post the Changes:

• The user has the option to set "allow_red_cluster" for each state where all the operations on that state can run on a red cluster if the "allow_red_cluster" is set to true. This parameter is optional.

• If the User sets the parameter to "true", the ISM job will be bypassed to run on the particular state on a red Cluster. Any transitions, Actions under that state will be executed.

• By default, the value of "allow_red_cluster" is false. Hence, the current behaviour of the API is also unchanged if the user doesnt want to bypass ISM to be run on a red cluster.

ISM on a Green/Yellow Opensearch Cluster:

---

ISM on a Red Opensearch Cluster with the optional parameter:

• In case if the cluster goes to red state due to the volume getting consumed more than 95%, the ISM will still execute the delete action because of the param allowRedCluster being set true, which will delete the index which is of size 1 GB. This would help us to get the cluster back to yellow/green state and the ISM will continue to run successfully for the previous stages.

CheckList:

• [ ] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[skumarp7]

Hi ,

Request feedback from the maintainers :)

[skumarp7]

Hi All,

I was trying to write test for the above feature and facing issues to simulate a red cluster. I tried to bring down the data node to make the cluster go read but since the integTest clusters are immutable, that was not possible.

Is there any other approach that i can make use of to simulate the red cluster ?

[Hailong-am]

Hi All,

I was trying to write test for the above feature and facing issues to simulate a red cluster. I tried to bring down the data node to make the cluster go read but since the integTest clusters are immutable, that was not possible.

Is there any other approach that i can make use of to simulate the red cluster ?

you can specify allocation policy for a index, An example

PUT test
{
  "settings": {
    "index.routing.allocation.require.size": "big"
  }
}

it will create index test and allocate shards on node with attribute size is big. As we don't have such node, the primary shards will not able to allocate, that will cause cluster health is red.