[BUG] Cluster block cannnot be removed if stuck in temp state during index block API

[mgodwan]

Describe the bug OpenSearch provides a dedicated API to apply index blocks and return response only when the block is verified on all the shards.

e.g. PUT /testindex/_block/write

The block is applied as follows:

• First, a temporary UUID-based block is added to the index and cluster state is applied.
• Second, shards are checked to have properly applied the UUID-based block.
• Third, the temporary UUID-based block is turned into a full block

If there is some issue during step 2 or step 3, the cluster state will remain with a UUID based block, which cannot be removed by any API. The current API to remove block by updating index settings allows the block to be removed only if it does not have a UUID as it compares the incoming block in API (without UUID) against the applied UUID based block in cluster state.

To Reproduce Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior We should have atomic behavior on this API, or allow a way to rollback the temporary UUID based cluster block so that index operations can resume if needed.

Plugins NA

Screenshots NA

Host/Environment (please complete the following information):

• OS: AL2
• Version OS 1.2

Additional context

[1] https://github.com/opensearch-project/OpenSearch/blob/996d33adb22ac6a523962166f3677f8dc6ac9c1f/server/src/main/java/org/opensearch/cluster/metadata/MetadataIndexStateService.java#L468 [2] https://github.com/opensearch-project/OpenSearch/blob/996d33adb22ac6a523962166f3677f8dc6ac9c1f/server/src/main/java/org/opensearch/cluster/metadata/MetadataIndexStateService.java#L1157 [3] https://github.com/opensearch-project/OpenSearch/blob/996d33adb22ac6a523962166f3677f8dc6ac9c1f/server/src/main/java/org/opensearch/cluster/metadata/MetadataIndexStateService.java#L553

[kotwanikunal]

Next steps:

• Identify if there are additional APIs with an atomic approach.
• Define an approach for the possible solution if there are other APIs found as a part of the first step.