[BUG] opensearch_cluster_settings attempts to null-out existing settings.

What is the bug?

Resource opensearch_cluster_settings attempts to apply null to unset properties, despite them being defaulted and valid in-cluster.

How can one reproduce the bug?

Create resource block opensearch_cluster_settings
Don't set properties
Terraform plan/apply
Terraform will attempt to null-out the properties

What is the expected behavior?

Existing and valid settings should be pulled into Terraform state from the cluster and not overridden unless the property is set in the resource block.

What is your host/environment?

uname -a && terraform -version
Darwin bne-nb-ariel 22.5.0 Darwin Kernel Version 22.5.0: Thu Jun  8 22:21:34 PDT 2023; root:xnu-8796.121.3~7/RELEASE_ARM64_T8112 arm64 arm Darwin
Terraform v1.5.3
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v4.62.0
+ provider registry.terraform.io/hashicorp/external v2.3.1
+ provider registry.terraform.io/hashicorp/local v2.4.0
+ provider registry.terraform.io/hashicorp/null v3.2.1
+ provider registry.terraform.io/hashicorp/random v3.5.1
+ provider registry.terraform.io/opensearch-project/opensearch v1.0.0
+ provider registry.terraform.io/phillbaker/elasticsearch v2.0.4

Do you have any screenshots?

Do you have any additional context?

Opensearch - by design for SaaS - prevents managing certain cluster settings. Ideally, settings that aren't available for a given OpenSearch version shouldn't be configurable in the resource block. But implementing that might be a lot of boilerplate.

https://github.com/phillbaker/terraform-provider-elasticsearch/issues/288#issuecomment-1248473782

[Untriage] Hey @arichtman-srt I see your point, so you suggest to retain the existing default settings and add/update with the settings as part of the opensearch_cluster_settings? @phillbaker can you please add your thoughts?

I think there are 2 pieces.

Trying to null out or otherwise unset cluster settings. I think if they properties are not set on the opensearch_cluster_settings we should not try to apply them. From my read of the HCL documentation, a null value is equivalent to the property not existing, so this would be consistent with the language.
OpenSearch restricted properties per-version. As documented here, some properties are not available for user modification, depending on OpenSearch version. Ideally the resource should fail validation if blocked properties are set. But I recognize this would be annoying to implement and maintain.

Hey @arichtman-srt, having declaring the empty global settings throws the following error.

resource "opensearch_cluster_settings" "global" {
}

Error

│ Error: elastic: Error 400 (Bad Request): Validation Failed: 1: no settings to update; [type=action_request_validation_exception]
│ 
│   with opensearch_cluster_settings.global,
│   on main.tf line 22, in resource "opensearch_cluster_settings" "global":
│   22: resource "opensearch_cluster_settings" "global" {

Adding as follows, works as expected

resource "opensearch_cluster_settings" "global" {
  action_auto_create_index    = "my-index-000001,index10,-index1*,+ind*"
}

Notice I have removed the setting cluster_max_shards_per_node = 10 and added action_auto_create_index = "my-index-000001,index10,-index1*,+ind*", works as expected and does not remove all the existing settings.

  # opensearch_cluster_settings.global will be updated in-place
  ~ resource "opensearch_cluster_settings" "global" {
      + action_auto_create_index    = "my-index-000001,index10,-index1*,+ind*"
      - cluster_max_shards_per_node = 10 -> null
        id                          = "settings"
    }

Can you please try with the latest release? Thank you

Adding @phillbaker @bbarani @peterzhuamazon

I experience the same thing on provider 2.2.0 against an AWS hosted opensearch 2.11 cluster. When I added the cluster settings resource I got the following plan:

Terraform will perform the following actions:

  # opensearch_cluster_settings.global will be created
+   resource "opensearch_cluster_settings" "global" {
+       action_auto_create_index = "false"
+       id                       = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

After applying that and planning again, now I get:

Terraform will perform the following actions:

  # opensearch_cluster_settings.global will be updated in-place
!   resource "opensearch_cluster_settings" "global" {
-       cluster_routing_allocation_cluster_concurrent_rebalance      = 2 -> null
-       cluster_routing_allocation_disk_watermark_high               = "22.0gb" -> null
-       cluster_routing_allocation_disk_watermark_low                = "25.0gb" -> null
-       cluster_routing_allocation_node_concurrent_recoveries        = 2 -> null
-       cluster_routing_allocation_node_initial_primaries_recoveries = 2 -> null
        id                                                           = "settings"
-       indices_recovery_max_bytes_per_sec                           = "94mb" -> null
        # (1 unchanged attribute hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Same here (we're using JSON TF config):

"opensearch_cluster_settings": {
    "index": {
        "action_auto_create_index": false,
    }
}

Initial terraform apply works:

Terraform will perform the following actions:

  # opensearch_cluster_settings.index will be created
  + resource "opensearch_cluster_settings" "index" {
      + action_auto_create_index = "false"
      + id                       = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

opensearch_cluster_settings.index: Creating...
opensearch_cluster_settings.index: Creation complete after 2s [id=settings]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Reapplying with no config changes should result in an empty plan, but doesn't. Applying the plan succeeds:

Terraform will perform the following actions:

  # opensearch_cluster_settings.index will be updated in-place
  ~ resource "opensearch_cluster_settings" "index" {
      - cluster_routing_allocation_disk_watermark_high = "22.0gb" -> null
      - cluster_routing_allocation_disk_watermark_low  = "25.0gb" -> null
        id                                             = "settings"
        # (1 unchanged attribute hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

opensearch_cluster_settings.index: Modifying... [id=settings]
opensearch_cluster_settings.index: Modifications complete after 1s [id=settings]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Since always having a non-empty plan is problematic for us, let's try a workaround:

"opensearch_cluster_settings": {
    "index": {
        "action_auto_create_index": false,
    },
    "lifecycle": {
        "ignore_changes": [
            "cluster_routing_allocation_disk_watermark_low",
            "cluster_routing_allocation_disk_watermark_high"
        ]
    }
}

terraform apply shows an empty plan.

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

So far, that would be an acceptable workaround. However, changing the config with the workaround in place:

"opensearch_cluster_settings": {
    "index": {
        "action_auto_create_index": true,
    },
    "lifecycle": {
        "ignore_changes": [
            "cluster_routing_allocation_disk_watermark_low",
            "cluster_routing_allocation_disk_watermark_high"
        ]
    }
}

… and running terraform apply fails

Terraform will perform the following actions:

  # opensearch_cluster_settings.index will be updated in-place
  ~ resource "opensearch_cluster_settings" "index" {
      ~ action_auto_create_index                       = "false" -> "true"
        id                                             = "settings"
        # (2 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

opensearch_cluster_settings.index: Modifying... [id=settings]
╷
│ Error: elastic: Error 401 (Unauthorized)
│ 
│   with opensearch_cluster_settings.index,
│   on elasticsearch.tf.json line 156, in resource[5].opensearch_cluster_settings[0].index:
│  156:                     }
│ 
╵

Adding another workaround (removing the state):

$ terraform state rm opensearch_cluster_settings.index
Removed opensearch_cluster_settings.index
Successfully removed 1 resource instance(s).

… and running terraform apply succeeds

Terraform will perform the following actions:

  # opensearch_cluster_settings.index will be created
  + resource "opensearch_cluster_settings" "index" {
      + action_auto_create_index = "true"
      + id                       = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Warnings:

- Resource targeting is in effect

To see the full warning notes, run Terraform without -compact-warnings.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

opensearch_cluster_settings.index: Creating...
opensearch_cluster_settings.index: Creation complete after 4s [id=settings]

Warnings:

- Applied changes may be incomplete

To see the full warning notes, run Terraform without -compact-warnings.

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

It's obviously problematic that one would have to manually remove the state before applying config changes. So it would be good to address this bug.

This is an AWS Opensearch domain, ES version 7.10, latest version of this provider (2.2.1).

opensearch-project / terraform-provider-opensearch