terraform-google-modules / terraform-google-kubernetes-engine

Configures opinionated GKE clusters
https://registry.terraform.io/modules/terraform-google-modules/kubernetes-engine/google
Apache License 2.0
1.15k stars 1.17k forks source link

`private_endpoint_subnetwork` not working #2119

Closed mruoss closed 1 month ago

mruoss commented 1 month ago

TL;DR

In #2009 support for private_endpoint_subnetwork was added. The problem is that master_ipv4_cidr_block has a default value which leads to both being set which again leads to the following error:

Error: googleapi: Error 400: When masterIpv4Cidr is set, privateEndpointSubnetwork must be unset.

Expected behavior

If private_endpoint_subnetwork is set, master_ipv4_cidr_block should not be set on the underlying resource.

Observed behavior

No response

Terraform Configuration

module "infra_gke_cluster" {
  source                              = "terraform-google-modules/kubernetes-engine/google//modules/beta-private-cluster-update-variant"
  version                             = "33.0.4"
  project_id                          = data.google_project.this.project_id
  name                                = "some-cluster"
  regional                            = false
  region                              = local.region
  zones                               = data.google_compute_zones.all_zones_in_region.names
  network                             = module.vpc.network_name
  subnetwork                          = local.subnets.gke.subnet_name
  ip_range_pods                       = "ip-range-pods"
  ip_range_services                   = "ip-range-svc"
  node_metadata                       = "GKE_METADATA"
  cluster_dns_provider                = "CLOUD_DNS"
  cluster_dns_domain                  = "infra.europe-west4"
  cluster_dns_scope                   = "VPC_SCOPE"
  maintenance_start_time              = "02:00"
  dns_cache                           = true
  remove_default_node_pool            = true
  gcs_fuse_csi_driver                 = true
  workload_config_audit_mode          = "BASIC"
  security_posture_mode               = "BASIC"
  security_posture_vulnerability_mode = "VULNERABILITY_BASIC"
  enable_gcfs                         = false # Image Streaming
  enable_private_nodes                = true
  enable_private_endpoint             = true
  create_service_account              = false
  master_ipv4_cidr_block              = null
  service_account                     = "****"
  private_endpoint_subnetwork         = local.subnets.gke-master.subnet_name
  enable_l4_ilb_subsetting            = true
  authenticator_security_group        = "***"
  gateway_api_channel                 = "CHANNEL_STANDARD"

  master_authorized_networks = [***]

  kubernetes_version = "1.30"

  node_pools_oauth_scopes = {
    "all" = [
      "https://www.googleapis.com/auth/cloud-platform",
      "https://www.googleapis.com/auth/devstorage.read_only"
    ]
  }

  cluster_autoscaling = {
    enabled             = true
    autoscaling_profile = "OPTIMIZE_UTILIZATION"
    max_cpu_cores       = 10000
    min_cpu_cores       = 0
    max_memory_gb       = 32000
    min_memory_gb       = 0
    disk_size           = 100
    disk_type           = "pd-balanced"
    auto_upgrade        = true
    auto_repair         = true
    gpu_resources       = []
  }

  timeouts = {
    create = "45m"
    update = "45m"
    delete = "45m"
  }

  cluster_resource_labels = {
    managed-by-terraform = true
  }
}

Terraform Version

Terraform v1.9.5

Additional information

No response

mruoss commented 1 month ago

This can be worked around by setting master_ipv4_cidr_block to null explicitely