Beta-private-cluster Unreachable Agent - please check if GKE Connect Agent is deployed correctly.

Unreachable Agent - please check if GKE Connect Agent is deployed correctly

Deployed beta-private-cluster complains about - Unreachable Agent - please check if GKE Connect Agent is deployed correctly. When deployed to fleet I do see the above error .

Some information about this -

The management and dev clusters are on shared vpc on GKE
I preferably like not to use the shared vpc but invidiual vpc and then use the connect gateway to connect these two clusters on ArgoCD .
But whatever I do the cluster just does not register as the gke agent is missing

Expected behavior

The cluster should register on fleet and the gke agent should be present.

Observed behavior

Terraform Configuration

module "gke_dev" {

  source = "terraform-google-modules/kubernetes-engine/google//modules/beta-private-cluster"

  version            = "~> 31.0"
  kubernetes_version = "1.28.9"

  name = "${var.name}-${var.environment}-gke-cluster"

  deletion_protection = false

  gateway_api_channel = "CHANNEL_STANDARD"

  # Required variables
  project_id = var.project_id
  region     = var.region

  fleet_project_grant_service_agent = true
  fleet_project                     = var.fleet_project

  authenticator_security_group = "gke-security-groups@company.com"

  network_project_id = "${var.name}-management"
  network           = "company-management"
  subnetwork        = "dev-gke-cluster-subnet-1"
  ip_range_pods     = "dev-gke-cluster-pod-subnet-1"
  ip_range_services = "dev-gke-cluster-service-subnet-1"

  service_account_name = "company-dev-gke"

  enable_l4_ilb_subsetting = true

  # Other variables
  http_load_balancing        = true
  network_policy             = true
  horizontal_pod_autoscaling = true
  gce_pd_csi_driver          = true

  filestore_csi_driver = true

  stack_type = "IPV4"

  enable_private_endpoint  = true
  enable_private_nodes     = true
  remove_default_node_pool = true

  enable_cost_allocation          = true
  enable_shielded_nodes           = true
  enable_vertical_pod_autoscaling = true

  grant_registry_access = true
  registry_project_ids  = [var.project_id]

  master_global_access_enabled = true
  enable_intranode_visibility  = true
  gke_backup_agent_config      = true

  master_ipv4_cidr_block = var.master_ipv4_cidr_block

  master_authorized_networks = var.master_authorized_networks

  # cluster_resource_labels = local.default_labels
  node_metadata = "GKE_METADATA_SERVER"

  node_pools = [
    {
      name         = "default-pool"
      machine_type = "e2-standard-2"

      min_count          = 1
      max_count          = 100
      initial_node_count = 1

      local_ssd_count = 0

      spot         = true
      disk_size_gb = 100
      disk_type    = "pd-standard"
      image_type   = "UBUNTU_CONTAINERD" # "COS_CONTAINERD"
      enable_gcfs  = false
      enable_gvnic = false
      auto_repair  = true
      auto_upgrade = true
      # service_account    = "project-service-account@<PROJECT ID>.iam.gserviceaccount.com"
      preemptible = false

    }
  ]

  node_pools_oauth_scopes = {
    all = [
      "https://www.googleapis.com/auth/cloud-platform"

    ]
  }

  node_pools_labels = {
    all = {}

    default-pool = var.gke_node_pool_default_labels
    # longhorn     = merge(local.default_labels, local.longhorn_labels)
  }

  node_pools_tags = {
    all = []

    default-node-pool = [
      "default-node-pool",
    ]
  }
}

Terraform Version

1.8.3

Additional information

I am not able to register ArgoCD dev cluster to management due to this issue - The cluster is not able to reach out to the other cluster . I am following this documentation - https://cloud.google.com/blog/products/containers-kubernetes/connect-gateway-with-argocd

terraform-google-modules / terraform-google-kubernetes-engine

Beta-private-cluster Unreachable Agent - please check if GKE Connect Agent is deployed correctly. #1965