.opendistro-ism-config already exists

[bloodguard]

When creating multiple policy from a fresh AWS OpenSearch cluster, i get this error:

elastic: Error 400 (Bad Request): index [.opendistro-ism-config/4pNn5XhHT3SwvIerntYi-A] already exists [type=resource_already_exists_exception]

From my understanding, the problem comes from the fact that for each policy, the plugins tries to create this index, but only need to create it once. To solve this issue, for now, we need to plan, then apply, then plan and apply again.

It there a workaround for this ? is this a known bug ?

Terraform provider config:

terraform {
  required_version = ">= 0.13.5"

  backend "s3" {}

  required_providers {
    aws     = "~> 3.15.0"
    elasticsearch  = {
      source  = "phillbaker/elasticsearch"
      version = "1.6.3"
    }
  }
}

OpenSearch version: 7.10

[phillbaker]

Hello, can you please include the following:

• redacted version of the terraform resource configuration
• terraform logs by setting TF_LOG=info

[bloodguard]

Logs of the terraform/terragrunt of the problem. Again, this is from starting from a fresh AWS ES cluster

NOTE: All the [REDACTED] represent ressources. Had to remove the names for professional reasons. Kept the ressources name of the policy for clarity.

 # elasticsearch_opendistro_ism_policy.ism_policy["logs-indice-lifecycle"] will be created
  + resource "elasticsearch_opendistro_ism_policy" "ism_policy" {
      + body         = jsonencode(
            {
              + policy = {
                  + default_state = "hot"
                  + description   = "Rotate index when the size of the index is too big."
                  + ism_template  = {
                      + index_patterns = [
                          + "logs*",
                        ]
                      + priority       = 1000
                    }
                  + states        = [
                      + {
                          + actions     = [
                              + {
                                  + rollover = {
                                      + min_index_age = "7d"
                                      + min_size      = "150gb"
                                    }
                                },
                            ]
                          + name        = "hot"
                          + transitions = [
                              + {
                                  + conditions = {
                                      + min_index_age = "14d"
                                    }
                                  + state_name = "warm"
                                },
                            ]
                        },
                      + {
                          + actions     = [
                              + {
                                  + retry          = {
                                      + count = 5
                                      + delay = "1h"
                                    }
                                  + warm_migration = {}
                                },
                            ]
                          + name        = "warm"
                          + transitions = [
                              + {
                                  + conditions = {
                                      + min_index_age = "21d"
                                    }
                                  + state_name = "delete"
                                },
                            ]
                        },
                      + {
                          + actions = [
                              + {
                                  + delete = {}
                                },
                            ]
                          + name    = "delete"
                        },
                    ]
                }
            }
        )
      + id           = (known after apply)
      + policy_id    = "logs-indice-lifecycle"
      + primary_term = (known after apply)
      + seq_no       = (known after apply)
    }
  # elasticsearch_opendistro_ism_policy.ism_policy["logs-rejected-lifecycle"] will be created
  + resource "elasticsearch_opendistro_ism_policy" "ism_policy" {
      + body         = jsonencode(
            {
              + policy = {
                  + default_state = "hot"
                  + description   = "For logs-rejected index. Smaller size and time to live index"
                  + ism_template  = {
                      + index_patterns = [
                          + "logs-rejected*",
                        ]
                      + priority       = 2000
                    }
                  + states        = [
                      + {
                          + actions     = [
                              + {
                                  + rollover = {
                                      + min_index_age = "1d"
                                      + min_size      = "150gb"
                                    }
                                },
                            ]
                          + name        = "hot"
                          + transitions = [
                              + {
                                  + conditions = {
                                      + min_index_age = "3d"
                                    }
                                  + state_name = "delete"
                                },
                            ]
                        },
                      + {
                          + actions = [
                              + {
                                  + delete = {}
                                },
                            ]
                          + name    = "delete"
                        },
                    ]
                }
            }
        )
      + id           = (known after apply)
      + policy_id    = "logs-rejected-lifecycle"
      + primary_term = (known after apply)
      + seq_no       = (known after apply)
    }

Plan: 26 to add, 0 to change, 0 to destroy.
elasticsearch_ingest_pipeline.ingest_pipeline[REDACTED]: Creating...
elasticsearch_ingest_pipeline.ingest_pipeline[REDACTED]: Creating...
elasticsearch_ingest_pipeline.ingest_pipeline[REDACTED]: Creating...
elasticsearch_component_template.component_template[REDACTED]: Creating...
elasticsearch_ingest_pipeline.ingest_pipeline[REDACTED]: Creating...
elasticsearch_opendistro_ism_policy.ism_policy["logs-indice-lifecycle"]: Creating...
elasticsearch_ingest_pipeline.ingest_pipeline[REACTED]: Creating...
elasticsearch_opendistro_ism_policy.ism_policy["logs-rejected-lifecycle"]: Creating...
elasticsearch_component_template.component_template[REACTED]: Creation complete after 1s [REDACTED]
elasticsearch_composable_index_template.index_template[REACTED]: Creating...
elasticsearch_composable_index_template.index_template[READACTED]: Creating...
elasticsearch_composable_index_template.index_template[REDACTED]: Creating...
elasticsearch_composable_index_template.index_template[REDACTED]: Creating...
elasticsearch_opendistro_ism_policy.ism_policy["logs-rejected-lifecycle"]: Creation complete after 1s [id=logs-rejected-lifecycle]
elasticsearch_composable_index_template.index_template[REDACTED]: Creating...
elasticsearch_ingest_pipeline.ingest_pipeline[REDACTED]: Creation complete after 1s [REDACTED]
elasticsearch_composable_index_template.index_template[REDACTED]: Creating...
elasticsearch_composable_index_template.index_template[REDACTED]: Creation complete after 0s [REDACTED]
elasticsearch_composable_index_template.index_template[REDACTED]: Creation complete after 1s [REDACTED]
elasticsearch_composable_index_template.index_template[REDACTED]: Creation complete after 1s [REDACTED]
elasticsearch_composable_index_template.index_template[REDACTED]: Creation complete after 1s [REDACTED]
elasticsearch_composable_index_template.index_template[REDACTED]: Creation complete after 1s [REDACTED]
elasticsearch_composable_index_template.index_template[REDACTED]: Creation complete after 1s [REDACTED]
elasticsearch_ingest_pipeline.ingest_pipeline[REDACTED]: Creation complete after 2s [REDACTED]
elasticsearch_ingest_pipeline.ingest_pipeline[REDACTED]: Creation complete after 2s [REDACTED]
elasticsearch_ingest_pipeline.ingest_pipeline[REDACTED]: Creation complete after 2s [REDACTED]
elasticsearch_ingest_pipeline.ingest_pipeline[REDACTED]: Creation complete after 2s [REDACTED]

Error: error putting policy: /_opendistro/_ism/policies/logs-indice-lifecycle : {
│   "policy": {
│     "description": "Rotate index when the size of the index is too big.",
│     "default_state": "hot",
│     "states": [
│       {
│         "name": "hot",
│         "actions": [{
│           "rollover": {
│             "min_size": "150gb",
│             "min_index_age": "7d"
│           }
│         }],
│         "transitions": [
│         {
│             "state_name": "warm",
│             "conditions": {
│                 "min_index_age": "14d"
│               }
│         }]
│       },
│       {
│         "name": "warm",
│         "actions": [{
│           "warm_migration": {},
│           "retry": {
│             "count": 5,
│             "delay": "1h"
│           }
│         }],
│         "transitions": [{
│           "state_name": "delete",
│           "conditions": {
│             "min_index_age": "21d"
│           }
│         }]
│       },
│       {
│         "name": "delete",
│         "actions": [
│         {
│           "delete": {}
│         }]
│       }
│     ],
│     "ism_template": {
│         "index_patterns": ["logs*"],
│         "priority": 1000
│       }
│   }
│ }
│  : elastic: Error 400 (Bad Request): index [.opendistro-ism-config/hSG52zRiQ4ywalv1fCVkUQ] already exists [type=resource_already_exists_exception]
│ 
│   with elasticsearch_opendistro_ism_policy.ism_policy["logs-indice-lifecycle"],
│   on elastisearch_ism_policy.tf line 2, in resource "elasticsearch_opendistro_ism_policy" "ism_policy":
│    2: resource "elasticsearch_opendistro_ism_policy" "ism_policy" {
│ 
time=2021-10-18T16:18:49Z level=error msg=Module /builds/harbour/sre/services/harbour-logs/infra/elasticsearch-index-template/terragrunt/dev-shared/us-east-1/es-dev/elasticsearch_configuration has finished with an error: 1 error occurred:
    * exit status 1
 prefix=[/builds/harbour/sre/services/harbour-logs/infra/elasticsearch-index-template/terragrunt/dev-shared/us-east-1/es-dev/elasticsearch_configuration] 
time=2021-10-18T16:18:49Z level=error msg=1 error occurred:
    * exit status 1
Cleaning up file based variables
00:04
ERROR: Job failed: exit code 1

Here is the code of the terraform:

resource "elasticsearch_opendistro_ism_policy" "ism_policy" {
    for_each = {for policy in var.ism_policies:  policy.name => policy}
    policy_id = each.value.name
    body = templatefile("${var.ism_policies_path}/${each.value.name}.tpl", {rollover_size=each.value.rollover_size})
}

[theGeorgew]

I encounter this issue when creating multiple (10-30) elasticsearchopendistro* resources in the same terraform apply, it seems that whatever resource is being used to generate the unique identifier, it was being called in parallel, and returning the same value to both resources, thus creating a colision. If I encounter this error, doing a second terraform apply will not have a problem.

[phillbaker]

It seems like we should treat this as a "back off and retry" error (ie a http 409) instead of a permanent failure. There are a few cases where opendistro/opensearch resources use the wrong error code.

[rishabhToshniwal]

Hello Everyone, I am facing this issue as well. I am creating Opensearch 2.7 cluster along with 3 ISM policies. Everytime the 2 ISM policies are getting failed with the below error and one of them getting created. "type":"status_exception","reason":"index [.opendistro-ism-config/8bN1QNU9RHWpcX8WrlB2EA] already exists"},"status":409} Additionally, if I am using null resource to curl and create those ISM policies, I am getting the same issue. Any possible work around instead of rerunning the terraform resource again ??