Discover web_backend_service name created previously from k8s api

[mcortinas]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

Description

I'm trying to deliver a service in GKE using helm with Google IAP protection enabled in a specific web_backend_service using terraform. My main issue is the names of resources created in GCP for resource web_backend_service there isn't a name matching from terraform code after create the HTTP Load Balancer with helm.

Let me explain better...

1. I deliver the service with helm using a code like this...

   resource "helm_release" "jenkins" {
   ...
   }

2. After that, I'm trying to allow permission with Terraform but the main issue is the name of the web_backend_service is not well known in my terraform code.... Let me share in my case, GCP create 2 HTTPS loadbalancers with the same Backend Names:

   • k8s-be-30667--289e6a19e503de26
   • k8s1-289e6a19-jenkins-onefront-jenkins-onefront-8080-588e64ecl Let me show you also the names of the two HTTPS loadbalancers (both of them with the same Backend Names showed previously)
   • k8s2-um-x0rsnpra-jenkins-onefro-jenkins-onefront-secon-05u5hswf
   • k8s2-um-x0rsnpra-jenkins-onefront-jenkins-onefront-bytgshhl

3.Let me shoy you the BackendConfig yaml created in k8s/gke with helm

apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  annotations:
    meta.helm.sh/release-name: jenkins-onefront
    meta.helm.sh/release-namespace: jenkins-onefront
  labels:
    app.kubernetes.io/component: jenkins-master
    app.kubernetes.io/instance: jenkins-onefront
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: jenkins
    helm.sh/chart: jenkins-2.12.1-2
  name: jenkins
  namespace: jenkins-onefront
  selfLink: /apis/cloud.google.com/v1/namespaces/jenkins-onefront/backendconfigs/jenkins
  uid: 4a0f56dd-8577-4bd3-b3b9-50bef09127f9
spec:
  iap:
    enabled: true
    oauthclientCredentials:
      secretName: iap

My problem is there are no relationship betweeen these resource names created on GCP and the web_backend_service parameter name in terraform resource iap_web_backend_service_iam - web_backend_service k8s-be-30667--289e6a19e503de26 k8s1-289e6a19-jenkins-onefront-jenkins-onefront-8080-588e64ecl

Affected Resource(s)

https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/iap_web_backend_service_iam

resource "google_iap_web_backend_service_iam_binding" "binding" {
  project = google_compute_backend_service.default.project
  web_backend_service = google_compute_backend_service.default.name 
  role = "roles/iap.httpsResourceAccessor"
  members = [
    "user:jane@example.com",
  ]
}

b/330932997

[rileykarson]

This doesn't require feature work in the provider, but would be nice to have documented somehow. These values are hard to access w/ Terraform as they are often not quite the right format to interpolate in Terraform, and stored as annotations (which are hard to use w/ TF)

[rdeknijf]

They are indeed hard to access. You can get the correct service id from the annotation on the k8s service. (In my case the service was created by Helm so I had to use a data source)

web_backend_service = jsondecode(data.kubernetes_service.<your_service>.metadata[0].annotations["cloud.google.com/neg-status"])["network_endpoint_groups"][<your_port>]

[ldnicolasmay]

@rdeknijf 's solution worked like a charm. Thank you!

For anyone like me who's a little new to this, here's a bit more of a complete example using the google_iap_web_backend_service_iam_member resource:

resource "kubernetes_namespace_v1" "my_ns" {
  metadata {
    name = "my_namespace"
  }
}

resource "helm_release" "my_helm_release" {
  chart            = "my_chart"
  name             = "my_thing"
  repository       = "https://blah.blah/helm"
  namespace        = kubernetes_namespace_v1.my_ns.metadata[0].name
  create_namespace = false

  depends_on = [kubernetes_namespace_v1.my_ns]
}

data "kubernetes_service" "my_webserver_service" {
  metadata {
    name      = "my-webserver"
    namespace = kubernetes_namespace_v1.my_ns.metadata[0].name
  }

  depends_on = [helm_release.my_helm_release]
}

resource "google_iap_web_backend_service_iam_member" "my_webserver_iap_members" {
  for_each = toset(
    formatlist("group:%s", [
      "group1@example.com",
      "group2@example.com",
      "group3@example.com",
    ])
  )
  project = "your-project-id"
  role    = "roles/iap.httpsResourceAccessor"
  member  = each.value
  web_backend_service = jsondecode(
    data.kubernetes_service.my_webserver_service.metadata[0].annotations["cloud.google.com/neg-status"]
  )["network_endpoint_groups"][80]

  depends_on = [data.kubernetes_service.my_webserver_service]
}

[bryan0515]

This request can be closed given that there is a work around and no action item on the provider side.

[mroach]

@ldnicolasmay Does this also work for you when the Service doesn't exist yet and needs to be created? I'm trying to create a brand new service that includes a couple annotations that I configure (e.g. cloud.google.com/backend-config) and the problem is that if I try to look for cloud.google.com/neg-status, I get an Invalid index error.

I tried creating a lifecycle postcondition to wait for the attribute to be set by k8s, but this results in the same error since Terraform think it already knows the ultimate value of this map.

My last attempt was to manually define a kubernetes_manifest so i could use the wait feature. Unfortunately this creates the exact same problem for me when I try to reference it. When generating a plan, Terraform thinks it already knows what's in the annotations map and throws Invalid index. I can't get Terraform to just wait until the object is created. I tried adding this to computed_fields and it still won't wait.

Maybe I'm missing something, but this workaround doesn't work for creating new services.

My last attempts:

resource "kubernetes_manifest" "management_service" {
  # lifecycle {
  #   postcondition {
  #     condition = contains(keys(self.metadata[0].annotations), "cloud.google.com/neg-status")
  #     error_message = "The related Google Cloud Backend Service has not been created"
  #   }
  # }
  computed_fields = ["metadata.annotations[\"cloud.google.com/neg-status\"]"]

  manifest = {
    apiVersion = "v1"
    kind = "Service"
    metadata = {
      name = "rabbitmq-management"
      namespace = var.stage
      labels = { app = "rabbitmq" }
      annotations = {
        "cloud.google.com/neg" = jsonencode({ "ingress" = true })
        "cloud.google.com/backend-config" = jsonencode({
          "default" = kubernetes_manifest.backend_config_management.manifest.metadata.name
        })
      }
    }
    spec = {
      type = "ClusterIP"
      selector = { app = "rabbitmq" }
      ports = [
        {
          port = 80
          target_port = local.ports.management_ui
          name = "management"
        }
      ]
    }
  }

  wait {
    fields = {
      "annotations[\"cloud.google.com/neg-status\"]" = ".+"
    }
  }
}

data "google_compute_backend_service" "management" {
  depends_on = [kubernetes_manifest.management_service]

  name = jsondecode(
    kubernetes_manifest.management_service.manifest.metadata.annotations["cloud.google.com/neg-status"]
  )["network_endpoint_groups"]["80"]
}

resource "google_iap_web_backend_service_iam_member" "internal_member" {
  for_each = toset(var.management_users)

  web_backend_service = data.google_compute_backend_service.management.name
  role                = "roles/iap.httpsResourceAccessor"
  member              = each.value
}

terraform --version
Terraform v1.8.3
on linux_amd64
+ provider registry.terraform.io/hashicorp/external v2.3.2
+ provider registry.terraform.io/hashicorp/google v5.15.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.25.2
+ provider registry.terraform.io/hashicorp/null v3.2.2
+ provider registry.terraform.io/hashicorp/random v3.6.0