Cannot update-in-place a running instance when adding observability field

ptran32 commented 3 weeks ago

Readiness Checklist

[ X ] I am running the latest version
[ X ] I checked the documentation and found no answer
[ X ] I checked to make sure that this issue has not already been filed
[ X ] I am reporting the issue to the correct repository (for multi-repository projects)

Expected Behavior

I expect Terraform to update-in-place my running Elastic Cloud instance and enable observability

Current Behavior

When doing a Terraform plan, it shows me that the instance will be updated-in-place:

terraform plan                                                                                                            ✔ 
ec_deployment.example_minimal: Refreshing state... [id=feXXXX623fXXXXfacc83178651]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # ec_deployment.example_minimal will be updated in-place
  ~ resource "ec_deployment" "example_minimal" {
        id                     = "XXXXXXX"
        name                   = "patrice-test-monitoring"
      + observability          = {
          + deployment_id = "XXXXXXX"
          + logs          = true
          + metrics       = true
          + ref_id        = (known after apply)
        }
        # (12 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

But when running Terraform apply, it is failing with an error: clusters.cluster_settings_change_prohibited: The requested cluster metadata changes are not permitted

ec_deployment.example_minimal: Modifying... [id=XXXXX]
│ Error: failed updating deployment
│ 
│   with ec_deployment.example_minimal,
│   on main.tf line 1, in resource "ec_deployment" "example_minimal":
│    1: resource "ec_deployment" "example_minimal" {
│ 
│ api error: 1 error occurred:
│       * clusters.cluster_settings_change_prohibited: The requested cluster metadata changes are not permitted.

The code I have used

resource "ec_deployment" "example_minimal" {

  name = "patrice-test-monitoring"

  region                 = "gcp-europe-west1"

  version                = "8.13.4"
  deployment_template_id = "gcp-general-purpose"

  observability = {
    deployment_id = "XXXX"
  }

  elasticsearch = {
    hot = {
      autoscaling = {}
    }
  }

  kibana = {}

  enterprise_search = {}

  integrations_server = {}
}

Steps to Reproduce

Deploy the Elastic Cloud instance using the terraform code above by removing the observability's section
Add the observability section and use a deployment ID if you have a dedicated monitoring instance or use "self" if you don't have one
run terraform plan
run terraform apply

Context

This is blocking me for sending metrics to a dedicated monitoring instance as recommended in this guide

Possible Solution

Make the update-in-place possible via Terraform.

Your Environment

Version used: provider[registry.terraform.io/elastic/ec] 0.10.0 / terraform version 1.7.5
Running against Elastic Cloud SaaS or Elastic Cloud Enterprise and version: Elastic Cloud Enterprise / version 8.13.4
Environment name and version (e.g. Go 1.9): Mac OS
Server type and version: N/A
Operating System and version: N/A
Link to your project: N/A

tobio commented 3 weeks ago

@ptran32 this error is coming from a bug in the underlying Cloud API. We expect to have a fix for it our by the end of August.

This error is only encountered when using an API Key from a user who is not the admin of an organisation. You could workaround the issue by using an org admin's API key in the meantime if that's possible.

ptran32 commented 3 weeks ago

Thank you for your response, I'll wait until the fix release as the workaround won't be possible due to security reasons.

chrispenny commented 3 weeks ago

Also experienced this error when trying to update ec_deployment.elasticsearch.config.user_settings. Noting also that I'm using an API key that is an organisation admin, so I don't believe the workaround works.

   # module.deployment_01.ec_deployment.default will be updated in-place
  ~ resource "ec_deployment" "default" {
      ~ elasticsearch          = {
          ~ config            = {
              ~ user_settings_yaml = <<-EOT
                    action.auto_create_index: false
                    xpack.security.audit.enabled: true
                  - xpack.security.audit.logfile.events.ignore_filters:
                  -     no_index_filter:
                  -         indices: ["", ".*"]
                EOT
                # (1 unchanged attribute hidden)
            }
            # (12 unchanged attributes hidden)
        }
        # (11 unchanged attributes hidden)
    }

argomark commented 3 weeks ago

I am getting the same error The requested cluster metadata changes are not permitted. I am also using an Org admin API key.

chrispenny commented 4 days ago

Hi @ptran32,

Any update on a resolution for this issue, or a functioning workaround?

tobio commented 3 days ago

@chrispenny I understand the fix for this has been released. Are you still currently seeing these errors?

ptran32 commented 3 days ago

Hi all,

The workaround we did prior to the merge of this feature was to manually enable observability on the instance (in Elastic Cloud UI)

I can also confirm that adding observability on a new running instance, does work without issue.

I am closing this issue.

elastic / terraform-provider-ec