Multiple metrics on kubernetes_horizontal_pod_autoscaler_v2beta2 creates a permadiff

[sandal-maker]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: v1.2.1
Kubernetes provider version: v2.18.1
Kubernetes version: GKE 1.24.9-gke.3200

Affected Resource(s)

• kubernetes_horizontal_pod_autoscaler_v2beta2

Terraform Configuration Files

resource "kubernetes_horizontal_pod_autoscaler_v2beta2" "app" {
  metadata {
    name = "app"
  }

  spec {
    min_replicas = 5
    max_replicas = 10

    scale_target_ref {
      api_version = "apps/v1"
      kind        = "Deployment"
      name        = kubernetes_deployment.app.metadata[0].name
    }

    metric {
      type = "Resource"
      resource {
        name = "cpu"
        target {
          type                = "Utilization"
          average_utilization = "70"
        }
      }
    }

    metric {
      type = "External"
      external {
        metric {
          name = "pubsub.googleapis.com|subscription|num_undelivered_messages"
          selector {
            match_labels = {
              "resource.labels.subscription_id" = var.application_sub.name
            }
          }
        }
        target {
          type          = "AverageValue"
          average_value = "2"
        }
      }
    }
  }
}

Debug Output

Terraform will perform the following actions:

  # module.app.module.app.kubernetes_horizontal_pod_autoscaler_v2beta2.app will be updated in-place
  ~ resource "kubernetes_horizontal_pod_autoscaler_v2beta2" "app" {
        id = "default/app"

      ~ spec {
            # (3 unchanged attributes hidden)

          ~ metric {
              ~ type = "External" -> "Resource"

              - external {
                  - metric {
                      - name = "pubsub.googleapis.com|subscription|num_undelivered_messages" -> null

                      - selector {
                          - match_labels = {
                              - "resource.labels.subscription_id" = "application-submitted"
                            } -> null
                        }
                    }

                  - target {
                      - average_utilization = 0 -> null
                      - average_value       = "2" -> null
                      - type                = "AverageValue" -> null
                    }
                }

              + resource {
                  + name = "cpu"

                  + target {
                      + average_utilization = 70
                      + type                = "Utilization"
                    }
                }
            }
          ~ metric {
              ~ type = "Resource" -> "External"

              + external {
                  + metric {
                      + name = "pubsub.googleapis.com|subscription|num_undelivered_messages"

                      + selector {
                          + match_labels = {
                              + "resource.labels.subscription_id" = "application-submitted"
                            }
                        }
                    }

                  + target {
                      + average_value = "2"
                      + type          = "AverageValue"
                    }
                }

              - resource {
                  - name = "cpu" -> null

                  - target {
                      - average_utilization = 70 -> null
                      - type                = "Utilization" -> null
                    }
                }
            }

            # (2 unchanged blocks hidden)
        }
        # (1 unchanged block hidden)
    }

Steps to Reproduce

1. Add kubernetes_horizontal_pod_autoscaler_v2beta2 resource with more than 1 metric sub stanza
2. terraform init
3. terraform plan
4. Apply successful
5. terraform plan (again, no config changes)
6. => Terraform plans to change the order of the two metrics and recreate them

Expected Behavior

The expectation was that there'd be no planned changes to be performed on the kubernetes_horizontal_pod_autoscaler_v2beta2 resource.

Actual Behavior

After the initial terraform apply all subsequent terraform runs permanently plans a difference attempting re-order the metrics.

In my case I have two metrics

1. type resource using CPU utilization
2. type external using pubsub messaging metric

The initial Terraform run applies them successfully. But all subsequent Terraform runs tries to change the order these two metrics are created.

It would appear the plan rearranges the metrics array every time.

Important Factoids

The cluster in on GCP and using GKE. The GKE version is mentioned above.

References

I could not find any at the moment. Apologies if I have missed an existing ticket related to this. Please link the existing one here and we can close this. Thanks.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[sandal-maker]

In addition to the above, I've tested the kubernetes_horizontal_pod_autoscaler_v2 and found that the issue above can be found in kubernetes_horizontal_pod_autoscaler_v2 resource as well.

[arybolovlev]

Hi @sandal-maker,

Thank you for reporting this issue. This is a duplicate of https://github.com/hashicorp/terraform-provider-kubernetes/issues/1188.

For now the workaround is to change the order in your Terraform code -- make it the same as in the API reply from Kubernetes.

Thank you.