terraform-google-modules / terraform-google-kubernetes-engine

Configures opinionated GKE clusters
https://registry.terraform.io/modules/terraform-google-modules/kubernetes-engine/google
Apache License 2.0
1.14k stars 1.17k forks source link

GKE Hub Membership in fleet-membership Module is Flappy #1434

Closed Jberlinsky closed 2 years ago

Jberlinsky commented 2 years ago

TL;DR

When combining cluster creation with fleet membership, use of a data resource to read the cluster's id causes the google_gke_hub_membership.primary resource to be re-created.

Expected behavior

Since the cluster's id does not change, the google_gke_hub_membership.primary resource should not be re-created.

Observed behavior

  # module.hub[0].google_gke_hub_membership.primary[0] must be replaced
-/+ resource "google_gke_hub_membership" "primary" {
      ~ id            = "projects/*****/locations/global/memberships/*****" -> (known after apply)
      - labels        = {} -> null
      ~ name          = "projects/*****/locations/global/memberships/*****" -> (known after apply)
        # (2 unchanged attributes hidden)

      ~ authority {
          ~ issuer = "https://container.googleapis.com/v1/projects//*****/locations/northamerica-northeast1/clusters/*****" -> (known after apply) # forces replacement
        }

      ~ endpoint {
          ~ gke_cluster {
              ~ resource_link = "//container.googleapis.com/projects/*****/locations/northamerica-northeast1/clusters/*****" -> (known after apply) # forces replacement
            }
        }
    }``

Terraform Configuration

module "gke" {
  source  = "terraform-google-modules/kubernetes-engine/google//modules/beta-private-cluster"
  version = "~> 23.0.0"

  depends_on = [
    google_service_account.gke_cluster,
    google_project_iam_member.gke_cluster,
    google_kms_key_ring_iam_binding.gke_system-encrypter_decrypter,
    google_kms_crypto_key.application_layer_encryption,
  ]

  project_id             = var.project_id
  network_project_id     = var.vpc_host_project
  name                   = "${var.cluster_name_prefix}-${var.environment_code}"
  region                 = var.region
  regional               = var.regional
  zones                  = var.regional ? data.google_compute_zones.this_zone.names : ["${data.google_compute_zones.this_zone.region}-${var.zone_id}"]
  network                = var.shared_vpc_name
  subnetwork             = local.subnetwork
  ip_range_pods          = local.ip_range_pods
  ip_range_services      = local.ip_range_services
  master_ipv4_cidr_block = local.master_ipv4_cidr_block

  kubernetes_version = var.cluster_master_version

  # Cluster Functionality
  identity_namespace = "enabled"

  enable_binary_authorization = true
  http_load_balancing         = true
  horizontal_pod_autoscaling  = true

  enable_l4_ilb_subsetting = var.enable_l4_ilb_subsetting
  database_encryption = [
    {
      state    = "ENCRYPTED"
      key_name = google_kms_crypto_key.application_layer_encryption.id
    }
  ]
  authenticator_security_group = var.authenticator_security_group

  # Cluster Networking
  enable_private_endpoint = !var.override_create_public_cluster_while_waiting_for_gitlab
  enable_private_nodes    = true

  datapath_provider         = var.enable_dataplane_v2 ? "ADVANCED_DATAPATH" : "DATAPATH_PROVIDER_UNSPECIFIED"
  default_max_pods_per_node = var.default_max_pods_per_node
  configure_ip_masq         = local.can_access_gke_to_deploy_configuration
  non_masquerade_cidrs      = var.non_masquerade_cidrs

  master_authorized_networks = local.master_authorized_networks

  # Labeling
  cluster_resource_labels = var.enable_asm ? { "mesh_id" = "proj-${data.google_project.this.number}" } : {}

  # Cluster Identity & Compute
  create_service_account = false
  service_account        = google_service_account.gke_cluster.email

  enable_shielded_nodes = var.enable_shielded_nodes

  node_pools = [
    {
      name               = "${var.cluster_name_prefix}-${var.environment_code}-node-pool"
      version            = var.cluster_nodepool_version
      node_locations     = var.regional ? join(",", data.google_compute_zones.this_zone.names) : "${data.google_compute_zones.this_zone.region}-${var.zone_id}"
      machine_type       = var.gke_machine_type
      min_count          = var.cluster_node_pool_min_count
      initial_node_count = var.cluster_node_pool_initial_node_count
      max_count          = var.cluster_node_pool_max_count
      disk_size_gb       = var.cluster_node_pool_disk_size_gb
      max_surge          = var.cluster_node_pool_max_surge
      image_type         = var.image_type
      auto_repair        = var.enable_autorepair
      auto_upgrade       = var.enable_autoupgrade
    }
  ]
}

module "hub" {
  count = var.enable_asm ? 1 : 0

  depends_on = [
    google_project_iam_member.hub_default_service_account,
    module.gke,
    google_gke_hub_feature.servicemesh,
  ]

  source  = "terraform-google-modules/kubernetes-engine/google//modules/fleet-membership"
  version = "~> 23.0.0"

  project_id      = var.project_id
  location        = module.gke.location
  cluster_name    = module.gke.name
  membership_name = "${var.project_id}-${module.gke.name}"
}

Terraform Version

$ terraform version
Terraform v1.3.1
on darwin_arm64
+ provider registry.terraform.io/hashicorp/google v4.40.0
+ provider registry.terraform.io/hashicorp/google-beta v4.40.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.14.0
+ provider registry.terraform.io/hashicorp/local v2.2.3
+ provider registry.terraform.io/hashicorp/random v3.4.3

$

Additional information

No response

Jberlinsky commented 2 years ago

From chatting with @bharathkkb in https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/pull/1435, it appears that the depends_on of module.gke is causing a deferred read, which forces this issue. The implicit dependency caused by referencing module.gke.cluster_name should be sufficient to solve the problem without a module-level code fix.

Closing.