ASM: Can't enable metrics export to prometheus

[gaurav517]

I am following the guide at https://cloud.google.com/service-mesh/docs/third-party-integrations#same-project and I am using following overlay file as custom_overlays to enable metrics export to prometheus:

kind: IstioOperator
spec:
  values:
    telemetry:
      enabled: true
      v2:
        enabled: true
        prometheus:
          enabled: true
        stackdriver:
          enabled: true
          inboundAccessLogging: "FULL"
          outboundAccessLogging: "ERRORS_ONLY"
          topology: true
  components:
    ingressGateways:
      - name: istio-ingressgateway
        enabled: true
        k8s:
          service:
            type: ClusterIP
            clusterIP: None
            ports:
              - name: status-port
                port: 15021
                protocol: TCP
                targetPort: 15021
              - name: http2
                port: 80
                protocol: TCP
                targetPort: 8080
              - name: https
                port: 443
                protocol: TCP
                targetPort: 8443
              - name: tcp-istiod
                port: 15012
                protocol: TCP
                targetPort: 15012
              - name: tls
                port: 15443
                protocol: TCP
                targetPort: 15443

I can confirm that service part of the overlay is working but don't see telemetry part working. There are no exporter containers/services. And when I use kiali I see:

Cannot load the graph: Post "http://prometheus.istio-system:9090/api/v1/query": dial tcp: lookup prometheus.istio-system on 10.148.0.10:53: no such host

Below is my terraform code:

module "asm" {
  source = "terraform-google-modules/kubernetes-engine/google//modules/asm"
  version = "16.0.0"
  project_id = "compact-haiku-317306 "
  cluster_name = "primary"
  location = "australia-southeast1-a"
  cluster_endpoint = "my-cluster-endpoint"
  enable_all = true
  managed_control_plane = false
  outdir = "/tmp"
  custom_overlays = [ "./resources/asm-service-overlay.yaml" ]
}

I can see following in terraform apply and everything looks good in apply:

module.asm.module.asm_install.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): install_asm_1.9 --verbose --project_id compact-haiku-317306 --cluster_name primary --cluster_location australia-southeast1-a --mode install   --custom_overlay /Users/kgaurav2/Documents/personal/iac/./resources/asm-service-overlay.yaml --output_dir /tmp --enable_all
...
...
module.asm.module.asm_install.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): install_asm_1.9: Running: './istio-1.9.6-asm.2/bin/istioctl install -f asm/istio/istio-operator.yaml -f /Users/kgaurav2/Documents/personal/iac/./resources/asm-service-overlay.yaml --set revision=asm-196-2 -c asm_kubeconfig --skip-confirmation'
module.asm.module.asm_install.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): install_asm_1.9: -------------

module.asm.module.asm_install.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): - Processing resources for Istio core.

module.asm.module.asm_install.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): ✔ Istio core installed
module.asm.module.asm_install.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): - Processing resources for Istiod.
module.asm.module.asm_install.module.gcloud_kubectl.null_resource.run_command[0]: Still creating... [1m20s elapsed]

module.asm.module.asm_install.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): - Processing resources for Istiod. Waiting for Deployment/istio-system/istiod-asm-196-2

module.asm.module.asm_install.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): ✔ Istiod installed
module.asm.module.asm_install.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): - Processing resources for Ingress gateways.

module.asm.module.asm_install.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): - Processing resources for Ingress gateways. Waiting for Deployment/istio-system/istio-ingressgateway
module.asm.module.asm_install.module.gcloud_kubectl.null_resource.run_command[0]: Still creating... [1m30s elapsed]

module.asm.module.asm_install.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): ✔ Ingress gateways installed
module.asm.module.asm_install.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): - Pruning removed resources
module.asm.module.asm_install.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): ✔ Installation complete
module.asm.module.asm_install.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): install_asm_1.9: ...done!
module.asm.module.asm_install.module.gcloud_kubectl.null_resource.run_command[0] (local-exec): install_asm_1.9: Running: './istio-1.9.6-asm.2/bin/istioctl profile dump -f asm/istio/istio-operator.yaml -f /Users/kgaurav2/Documents/personal/iac/./resources/asm-service-overlay.yaml'

[bharathkkb]

Since the apply is working, this could be a potential issue with the install_asm script (which we use internally). Could you open an issue in https://github.com/GoogleCloudPlatform/anthos-service-mesh-packages/issues with the apply logs.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 7 days