Annotations added by gke or kubernetes threated as changes

[i2dcarrasco]

Hello,

Yesterday I had to upgrade the kubernetes-alpha provider because it had a bug with the annotation "ingress.gcp.kubernetes.io/pre-shared-cert" which avoided to apply any change in terraform because a RPC error. After upgrade I had to adjust some things and even destroy the ingress because it started to fail on update. After recreate the ingress and check that there are no more changes, I've noticed that aplying again later, some changes has started to appear, and after investigate it a bit I've noticed that those changes are in "object" and are related with some annotations added by Google Kubernetes Engine or Kubernetes itself.

Looks like in the old version those changes were ignored because I had never got those changes before, and now I have to add all those changes to ignore list. Is not a big problem, but it worries me because I don't know if in the future more annotations will be added and then detected as changes, and also I don't want to ignore all the annotations changes because I'm using it and I don't want to lock it changes.

Terraform Version, Provider Version and Kubernetes Version

Terraform version: v0.14.10
Kubernetes provider version: v0.3.2
Kubernetes version: 1.19.9-gke.100

Affected Resource(s)

• Ingress

Terraform Configuration Files

resource "kubernetes_manifest" "as_elastic_ingress" {
  provider = kubernetes-alpha.as

  manifest = {
    apiVersion = "networking.k8s.io/v1beta1"
    kind       = "Ingress"
    metadata = {
      name      = local.k8s_as_elastic.service_ingress.as_elastic.name
      namespace = "tradeinn-as-pro"
      annotations = {
        "ingress.gcp.kubernetes.io/pre-shared-cert"   = "lb-cookie,lb-tradeinn"
        "kubernetes.io/ingress.global-static-ip-name" = google_compute_global_address.tradeinn_web_as.name
        "networking.gke.io/v1beta1.FrontendConfig"    = local.k8s_as_elastic.service_ingress.as_elastic.name
        "ingress.kubernetes.io/limit-rps"             = "10000"
      }
    }
    spec = {
      backend = {
        serviceName = "${local.k8s_as_elastic.service_ingress.as_elastic.name}-test"
        servicePort = 8080
      }
    }
  }
}

Debug Output

 # kubernetes_manifest.as_elastic_ingress will be updated in-place
  ~ resource "kubernetes_manifest" "as_elastic_ingress" {
      ~ object   = {
          ~ metadata   = {
              ~ annotations                = {
                  - "ingress.kubernetes.io/backends"              = jsonencode(
                        {
                          - k8s1-97b1834d-tradeinn-as-p-tradeinn-web-as-elasti-808-fa47271b = "HEALTHY"
                        }
                    ) -> null
                  - "ingress.kubernetes.io/forwarding-rule"       = "k8s2-fr-7u7ibz51-tradeinn-as-pro-tradeinn-web-as-elast-f13jl3v7" -> null
                  - "ingress.kubernetes.io/https-forwarding-rule" = "k8s2-fs-7u7ibz51-tradeinn-as-pro-tradeinn-web-as-elast-f13jl3v7" -> null
                  - "ingress.kubernetes.io/https-target-proxy"    = "k8s2-ts-7u7ibz51-tradeinn-as-pro-tradeinn-web-as-elast-f13jl3v7" -> null
                  - "ingress.kubernetes.io/redirect-url-map"      = "k8s2-rm-7u7ibz51-tradeinn-as-pro-tradeinn-web-as-elast-f13jl3v7" -> null
                  - "ingress.kubernetes.io/ssl-cert"              = "lb-cookie,lb-tradeinn" -> null
                  - "ingress.kubernetes.io/target-proxy"          = "k8s2-tp-7u7ibz51-tradeinn-as-pro-tradeinn-web-as-elast-f13jl3v7" -> null
                  - "ingress.kubernetes.io/url-map"               = "k8s2-um-7u7ibz51-tradeinn-as-pro-tradeinn-web-as-elast-f13jl3v7" -> null
                    # (4 unchanged elements hidden)
                }
                # (15 unchanged elements hidden)
            }
            # (3 unchanged elements hidden)
        }
        # (1 unchanged attribute hidden)
    }

Steps to Reproduce

1. terraform apply with the above configuration
2. Wait some time until Google or K8s add some new annotations when the ingress is ready

Expected Behavior

Ignore the annotations added by Kubernetes because are managed by the cluster and may change.

Actual Behavior

Changes are detected and then it tries to remove it all the time.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[i2dcarrasco]

Also I've noticed that I'm not able to ignore that change...

I've tried:

  lifecycle {
    ignore_changes = [
      object.metadata.annotations,
      object.metadata.0.annotations,
      object.0.metadata.0.annotations,
      object.0.metadata.annotations,
      manifest.metadata,
      manifest.0.metadata
    ]
  }

And the changes appears all the time.

Best regards.

[EronWright]

One detail to look at is the managedFields annotation; is there a field manager associated with the added annotation?

The alpha provider uses server-side apply, and should be considering only the fields that are under terraform's management. I observe that the plugin is also considering fields that don't have a field manager. Am wondering whether full support is there yet.

[dpkirchner]

Something I've noticed is that, at least for the resources I'm using, the annotation differences actually cause a replace rather than an update in place. The resource I'm using is custom, Application from argo-cd, and ends up with a managedFields entry for the annotation (created by an argo notification). I'm not exactly sure how CRDs work but I assume that indicates the annotation was created by something outside of k8s.

I post this just to add another data point, not necessarily to ask for a solution (given #208 will probably fix it anyway).

  # module.app["xyz"].kubernetes_manifest.apps[0] must be replaced
-/+ resource "kubernetes_manifest" "apps" {
      ~ object   = {
          ~ metadata   = {
              - annotations = {
                  - notified.notifications.argoproj.io = jsonencode(
                        {
                          - on-deployed:[0].N-abcdef= 1621525675
                        }
                    )
                } -> null
                # (2 unchanged elements hidden)
            }
            # (3 unchanged elements hidden)
        } # forces replacement
        # (1 unchanged attribute hidden)
    }

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    notified.notifications.argoproj.io: '{"on-deployed:[0].N-abcdef":1621525675}'
  creationTimestamp: "2021-05-05T18:52:26Z"
  generation: 7236
  managedFields:
  - apiVersion: argoproj.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:notified.notifications.argoproj.io: {}
    manager: argocd-notifications-backend
    operation: Update
    time: "2021-05-20T15:47:55Z"

Note for folks that might have the same issue: downgrading to 0.2.1 resolves this issue, but you miss out on some custom resource related improvements (that don't happen to affect me, but just FYI).

[EcaterinaGr]

I also face this issue but on "finalizers" which are added. The beta release seems not fixing that neither