computed_fields does not work when split() is used for multi-part YAML

rootwyrm commented 1 year ago

Terraform Version, Provider Version and Kubernetes Version

Terraform version: 1.4.4
Kubernetes provider version: 2.19.0
Kubernetes version: 1.21

Affected Resource(s)

kubernetes_manifest

Terraform Configuration Files

Please note this is a partial example to demonstrate reproduction as the full config is too large and contains sensitive information. To reproduce, obtain the official Elastic Search operator from ElasticSearch here.

... kubernetes cluster configuration goes here ...

data "local_file" "eck_operator_crd" {
  filename = "${path.module}/crds.yaml"
}
data "local_file" "eck_operator" {
  filename = "${path.module}/operator.yaml"
}

resource "kubernetes_manifest" "eck_operator_crds" {
  depends_on = [
    data.local_file.eck_operator_crds
  ]
  for_each = {
    for value in [
      for yaml in split(
        "\n---\n",
        "\n${replace(data.local_file.eck_operator_crd.content, "/(?m)^---[[:blank:]]*(#.*)?$/", "---")}\n"
      ) :
      yamldecode(yaml)
      if trimspace(replace(yaml, "/(?m)(^[[:blank:]]*(#.*)?$)+/", "")) != ""
    ] : "${value["kind"]}--${value["metadata"]["name"]}" => value
  }
  manifest = each.value

  computed_fields = [
    "metadata.labels",
    "metadata.annotations",
    "metadata.creationTimestamp",
    "webhooks.clientConfig.caBundle"
  ]
}

resource "kubernetes_manifest" "eck_operator" {
  ## Note: dependency isn't strictly necessary, but reduces risk of transient failure during deploys
  depends_on = [
    data.local_file.eck_operator,
    kubernetes_manifest.eck_operator_crds
  ]
  for_each = {
    for value in [
      for yaml in split(
        "\n---\n",
        "\n${replace(data.local_file.eck_operator.content, "/(?m)^---[[:blank:]]*(#.*)?$/", "---")}\n"
      ) :
      yamldecode(yaml)
      if trimspace(replace(yaml, "/(?m)(^[[:blank:]]*(#.*)?$)+/", "")) != ""
    ] : "${value["kind"]}--${value["metadata"]["name"]}" => value
  }
  manifest = each.value

  computed_fields = [
    "metadata.labels",
    "metadata.annotations",
    "metadata.creationTimestamp",
    "webhooks.clientConfig.caBundle"
  ]
}

Debug Output

Comprehensive debug is unavailable due to sensitive data and size.

Example partial output:

$ terraform --version
Terraform v1.4.4
on linux_amd64

  # module.elasticsearch[0].kubernetes_manifest.eck_operator["Namespace--elastic-system"] will be updated in-place
  ~ resource "kubernetes_manifest" "eck_operator" {
      + computed_fields = [
          + "metadata.labels",
          + "metadata.annotations",
          + "metadata.creationTimestamp",
          + "webhooks.clientConfig.caBundle",
        ]
      ~ object          = {
          ~ metadata   = {
              ~ labels                     = {
                  - "kubernetes.io/metadata.name" = "elastic-system"
                  + name                          = "elastic-system"
                }
                name                       = "elastic-system"
                # (14 unchanged attributes hidden)
            }
            # (3 unchanged attributes hidden)
        }
        # (1 unchanged attribute hidden)
        # (1 unchanged block hidden)
    }
  # module.elasticsearch[0].kubernetes_manifest.eck_operator["ValidatingWebhookConfiguration--elastic-webhook.k8s.elastic.co"] will be updated in-place
  ~ resource "kubernetes_manifest" "eck_operator" {
      + computed_fields = [
          + "metadata.labels",
          + "metadata.annotations",
          + "metadata.creationTimestamp",
          + "webhooks.clientConfig.caBundle",
        ]
      ~ object          = {
          ~ webhooks   = [
              ~ {
                  ~ clientConfig            = {
                      ~ caBundle = SENSITIVE DATA REMOVED -> "Cg=="
...
  # module.elasticsearch[0].kubernetes_manifest.eck_operator_crds["CustomResourceDefinition--agents.agent.k8s.elastic.co"] will be updated in-place
  ~ resource "kubernetes_manifest" "eck_operator_crds" {
      ~ object   = {
          ~ metadata   = {
              - creationTimestamp          = null
                name                       = "agents.agent.k8s.elastic.co"
                # (14 unchanged attributes hidden)
            }
            # (3 unchanged attributes hidden)
        }
        # (1 unchanged attribute hidden)
        # (1 unchanged block hidden)
    }

Demonstrating the problem on Terraform 1.3.9 to show that creationTimestamp was also not honored. Note that this behaves identically with object.metadata.creationTimestamp and metadata.creationTimestamp; only one is shown for brevity.

  # module.elasticsearch[0].kubernetes_manifest.eck_operator_crds["CustomResourceDefinition--stackconfigpolicies.stackconfigpolicy.k8s.elastic.co"] will be updated in-place
  ~ resource "kubernetes_manifest" "eck_operator_crds" {
      + computed_fields = [
          + "object.metadata.labels",
          + "object.metadata.annotations",
          + "object.metadata.creationTimestamp",
          + "object.webhooks.clientConfig.caBundle",
        ]
      ~ object          = {
          ~ metadata   = {
              ~ creationTimestamp          = null -> (known after apply)
                name                       = "stackconfigpolicies.stackconfigpolicy.k8s.elastic.co"
                # (14 unchanged elements hidden)
            }
            # (3 unchanged elements hidden)
        }
        # (1 unchanged attribute hidden)
        # (1 unchanged block hidden)
    }

Steps to Reproduce

Add your cluster configuration and obtain the relevant YAMLs from ElasticSearch (do not edit the YAMLs in any way)
terraform plan; terraform apply
Confirm that the operator has been deployed (API should return all values, beware of sensitive values within)
terraform plan will show on every subsequent run that metadata.labels and webhooks.clientConfig.caBundle are to be updated in place even though both are listed as a computed_field.

Expected Behavior

metadata.labels and webhooks.clientConfig.caBundle should be treated as a computed field due to explicit listing.

Actual Behavior

Terraform will attempt to apply changes to computed_fields on every run.

Important Factoids

This is a minimum reproducible example; there are likely other cases. However, this is the easiest reproduction method using publicly available, well-documented multi-part YAML.
The split() is because this is specifically a multi-part YAML.
Prior to 1.4.3, metadata.creationTimestamp was also not honored in computed_fields
Using object.* does not change behavior; Terraform will always attempt to update both metadata.labels and webhook.clientConfig.caBundle.

References

GH-32892 - the 1.4.3+ change is what fixed creationTimestamp, it was not being treated as a computed value previously

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

sheneska commented 1 year ago

Hi @rootwyrm, we are going to reproduce this issue to understand what exactly the issue may be.

github-actions[bot] commented 7 months ago

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!

alexsomesan commented 7 months ago

@rootwyrm This isn't reproducible anymore with the latest version of Terraform and the provider. Can you please confirm on your side if this is still an issue using these current versions?

➜  issue-K2068 terraform plan               
data.local_file.eck_operator: Reading...
data.local_file.eck_operator_crd: Reading...
data.local_file.eck_operator: Read complete after 0s [id=616b3904c95831f554893a66833ba5aea68c4493]
data.local_file.eck_operator_crd: Read complete after 0s [id=d481659af3f4be74f07b01bd8b0feed7ffba6a47]
kubernetes_manifest.eck_operator_crds["CustomResourceDefinition--stackconfigpolicies.stackconfigpolicy.k8s.elastic.co"]: Refreshing state...
kubernetes_manifest.eck_operator_crds["CustomResourceDefinition--elasticmapsservers.maps.k8s.elastic.co"]: Refreshing state...
kubernetes_manifest.eck_operator_crds["CustomResourceDefinition--beats.beat.k8s.elastic.co"]: Refreshing state...
kubernetes_manifest.eck_operator_crds["CustomResourceDefinition--elasticsearchautoscalers.autoscaling.k8s.elastic.co"]: Refreshing state...
kubernetes_manifest.eck_operator_crds["CustomResourceDefinition--enterprisesearches.enterprisesearch.k8s.elastic.co"]: Refreshing state...
kubernetes_manifest.eck_operator_crds["CustomResourceDefinition--logstashes.logstash.k8s.elastic.co"]: Refreshing state...
kubernetes_manifest.eck_operator_crds["CustomResourceDefinition--kibanas.kibana.k8s.elastic.co"]: Refreshing state...
kubernetes_manifest.eck_operator_crds["CustomResourceDefinition--apmservers.apm.k8s.elastic.co"]: Refreshing state...
kubernetes_manifest.eck_operator_crds["CustomResourceDefinition--elasticsearches.elasticsearch.k8s.elastic.co"]: Refreshing state...
kubernetes_manifest.eck_operator_crds["CustomResourceDefinition--agents.agent.k8s.elastic.co"]: Refreshing state...
kubernetes_manifest.eck_operator["ConfigMap--elastic-operator"]: Refreshing state...
kubernetes_manifest.eck_operator["ClusterRoleBinding--elastic-operator"]: Refreshing state...
kubernetes_manifest.eck_operator["ServiceAccount--elastic-operator"]: Refreshing state...
kubernetes_manifest.eck_operator["ClusterRole--elastic-operator-edit"]: Refreshing state...
kubernetes_manifest.eck_operator["Secret--elastic-webhook-server-cert"]: Refreshing state...
kubernetes_manifest.eck_operator["Namespace--elastic-system"]: Refreshing state...
kubernetes_manifest.eck_operator["Service--elastic-webhook-server"]: Refreshing state...
kubernetes_manifest.eck_operator["ClusterRole--elastic-operator-view"]: Refreshing state...
kubernetes_manifest.eck_operator["ClusterRole--elastic-operator"]: Refreshing state...
kubernetes_manifest.eck_operator["StatefulSet--elastic-operator"]: Refreshing state...
kubernetes_manifest.eck_operator["ValidatingWebhookConfiguration--elastic-webhook.k8s.elastic.co"]: Refreshing state...

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.
➜  issue-K2068 terraform version 
Terraform v1.8.0
on darwin_arm64
+ provider registry.terraform.io/hashicorp/kubernetes v2.29.0
+ provider registry.terraform.io/hashicorp/local v2.5.1

alexsomesan commented 7 months ago

May I also add that these versions also include a better alternative to the split() function trick, specifically dedicated functions to decode multi-doc YAML manifests natively.

Have a look at the example here: https://www.hashicorp.com/blog/terraform-1-8-adds-provider-functions-for-aws-google-cloud-and-kubernetes#terraform-kubernetes-provider

hashicorp / terraform-provider-kubernetes