[BUG] Terraform apply fails when index is created and configured to use ISM rollover policy

[sharathganga]

What is the bug?

I've created an initial index test-logs-000001 which has is_write_index = true property set and this index is managed by opensearch_index_template resource and configured to rollover using ISM policy resource.

But as the rollover occurs, the index test-logs-000002, test-logs-000003...etc, the latest index becomes the write enabled index, and so when I subsequently run terraform plan and apply, it throws the error as below:

Error: elastic: Error 500 (Internal Server Error): alias [test-logs] has more than one write index [test-logs-000001,test-logs-00010] [type=illegal_state_exception]

How can one reproduce the bug?

• Create index, index template and ism rollover policy using the below code and wait for the rollover to occur with then the alias pointed to the latest index and property is_write_index=true

resource "opensearch_ism_policy" "rollover_index_policy" {
  policy_id = "Rollover-and-delete-indexes"
  body = jsonencode(
    {
      "policy" : {
        "description" : "Rollover index and delete it after 7 days",
        "default_state" : "hot",
        "states" : [
          {
            "name" : "hot",
            "actions" : [
              {
                "retry" : {
                  "count" : 3,
                  "backoff" : "exponential",
                  "delay" : "1m"
                },
                "rollover" : {
                  "min_size" : "10gb"
                }
              }
            ],
            "transitions" : [
              {
                "state_name" : "delete",
                "conditions" : {
                  "min_index_age" : "7d"
                }
              }
            ]
          },
          {
            "name" : "delete",
            "actions" : [
              {
                "retry" : {
                  "count" : 3,
                  "backoff" : "exponential",
                  "delay" : "1m"
                },
                "delete" : {}
              }
            ],
            "transitions" : []
          }
        ],
        "ism_template" : [
          {
            "index_patterns" : [
              "test-logs-*"
            ],
            "priority" : "100"
          }
        ]
      }
    }
  )
}

resource "opensearch_index_template" "index_template" {
  name = "test-logs"
  body = jsonencode(
    {
      "index_patterns" : [
        "test-logs-*",
      ],
      "template" : "test-logs-*",
      "settings" : {
        "number_of_shards" : 3,
        "number_of_replicas" : 1,
        "plugins" : {
          "index_state_management" : {
            "rollover_alias" : "test-logs"
          }
        }
      }
    }
  )
}

resource "opensearch_index" "index" {
  name = "test-logs-000001"
  aliases = jsonencode(
    {
      "test-logs" = {
        "is_write_index" = true
      }
    }
  )

  depends_on = [opensearch_index_template.index_template]
}

What is your host/environment?

$ uname -a
Linux PBL244 5.15.90.1-microsoft-standard-WSL2 #1 SMP Fri Jan 27 02:56:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

$ terraform version
Terraform v1.5.3
on linux_amd64
+ provider registry.terraform.io/opensearch-project/opensearch v1.0.0

[prudhvigodithi]

Hey @sharathganga I assume you are using the latest version of the provider https://registry.terraform.io/providers/opensearch-project/opensearch/2.0.0 Adding @rblcoder @afrodidact @premkirank Thank you

[sharathganga]

@prudhvigodithi I'm using the provider version 1.0.0.

[rblcoder]

@sharathganga @prudhvigodithi once the rollover happens, the index created using terraform will no longer have is_write_index true, using plan and apply again will throw the more than one write index error. ignore_changes can be used to ignore the changes. Or update the terraform code with the current write-enabled index.

[prudhvigodithi]

Hey @rblcoder can you please take a look at the provider code to make this change? Please let me know I can assign this bug to you. Thank you

[sharathganga]

@sharathganga @prudhvigodithi once the rollover happens, the index created using terraform will no longer have is_write_index true, using plan and apply again will throw the more than one write index error. ignore_changes can be used to ignore the changes. Or update the terraform code with the current write-enabled index.

@rblcoder, I've already tried using lifecycle policy with ignore_changes=all on the index resource but that didn't help either. This resource is using for_each argument so it wouldn't be realistic to get the current write enabled index as the indexes are at various states of rolled over index.

[rblcoder]

The following steps work for me -
Post creating the resources through terraform, and a rollover, terraform apply did not throw errors. This is using OpenSearch provider version = "2.0.0" and OpenSearch version 2.11.0

[sharathganga]

@rblcoder , I'm using AWS Opensearch 1.3 and provider version 1.0.0. When I tried using the provider version 2.0.0, it gave me a different error while creating opensearch_index_template resource.

╷
│ Error: elastic: Error 400 (Bad Request): [1:35] [index_template] unknown field [settings] [type=x_content_parse_exception]
│
│   with opensearch_index_template.index_template,
│   on main.tf line 60, in resource "opensearch_index_template" "index_template":
│   60: resource "opensearch_index_template" "index_template" {
│
╵

[rblcoder]

Can you please try with index specified as follows?

resource "opensearch_index" "index" {
  name = "test-logs-000001"
  number_of_shards = 3
  number_of_replicas = 1

  aliases = jsonencode(
    {
      "test-logs" = {
        "is_write_index" = true
      }
    }
  )

[sharathganga]

@rblcoder number_of_shards and number_of_replicas is being managed by the opensearch_index_template resource with rollover alias configured.

[malaquf]

We've just migrated from phillbaker's project to this project and started facing the same issue. At phillbaker's library, this fix has been implemented for following the write index from ISM/ILM. Maybe we need the same approach in this project?

[rblcoder]

@malaquf We have specified

Computed: true 

for number_of_replicas https://github.com/opensearch-project/terraform-provider-opensearch/pull/123/files The PR above is merged. This fixed the error thrown when index configuration has no number_of_replicas specified.

[malaquf]

Interesting, in our case, we do have number_of_replicas specified, and we still face the issue. We are on v2.2.1.

[rblcoder]

@malaquf I created a policy using Terraform code for the following example https://opensearch.org/docs/latest/im-plugin/ism/policies/#sample-policy-with-ism-template-for-auto-rollover Next I waited for the rollover to happen. Applying the Terraform code again did not give errors.

Output of GET /log after rollover

{
  "log-000001": {
    "aliases": {
      "log": {
        "is_write_index": false
      }
    },
    "mappings": {
      "properties": {
        "contractor": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "name": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    },
    "settings": {
      "index": {
        "replication": {
          "type": "DOCUMENT"
        },
        "number_of_shards": "1",
        "plugins": {
          "index_state_management": {
            "rollover_alias": "log"
          }
        },
        "provided_name": "log-000001",
        "creation_date": "1713583424647",
        "number_of_replicas": "1",
        "uuid": "Jgntl4tARBahGdefUh08NQ",
        "version": {
          "created": "136337827"
        }
      }
    }
  },
  "log-000002": {
    "aliases": {
      "log": {
        "is_write_index": true
      }
    },
    "mappings": {},
    "settings": {
      "index": {
        "replication": {
          "type": "DOCUMENT"
        },
        "number_of_shards": "1",
        "plugins": {
          "index_state_management": {
            "rollover_alias": "log"
          }
        },
        "provided_name": "log-000002",
        "creation_date": "1713584187139",
        "number_of_replicas": "1",
        "uuid": "8BB_8wnfQgaRJeb3xzncFw",
        "version": {
          "created": "136337827"
        }
      }
    }
  }
}

[ctav4]

We migrated from phillbaker's project to this project and after this we started to see this issue as soon as the first index was deleted due to rollover policy. This behavior didn't occur when using phillbaker's project.

Is someone still facing this issue?

The error is: │ Error: elastic: Error 500 (Internal Server Error): alias [proj-sandbox_index] has more than one write index [proj-sandbox_index-000001,proj-sandbox_index-000006] [type=illegal_state_exception]

Our Terraform code is:

resource "opensearch_index" "first-index" {
  name               = "proj-sandbox_index-000001"
  number_of_replicas = 3
  force_destroy      = var.index_force_destroy
  rollover_alias = "proj-sandbox_index"

  aliases = <<EOF
  {
    "proj-sandbox_index": {
      "is_write_index": true,
      "index.number_of_replicas": 3
    }
  }
  EOF
  lifecycle {
    ignore_changes = [
      number_of_replicas,
      aliases,
      routing_partition_size
    ]
  }
}