Scraping metrics from multiple replicas in Kong for Kubernetes

[ikethecoder]

I have Kong deployed to Kubernetes using Kong's Helm Chart with the replicas set to 3. And I have the prometheus plugin setup with global scope.

My understanding is that the Prometheus plugin will gather metrics on each of the replicas, so Prometheus will need to scrape from each replica. The problem I am facing is that because Kong does not get deployed as a StatefulSet, there is no service route to each individual replica - so prometheus is not scraping the full set of metrics from all 3 replicas.

What's the recommended approach for getting the full set of metrics? Use a Kong sidecarContainer that forwards /metrics to a central PushGateway? Deploy Kong as a StatefulSet? Or am I missing something obvious or another option?

[hbagdi]

You don't need a statefulset for such a case. I suggest a read on Prometheus' k8s service discovery documentation to understand how it works. As long as you have a single k8s Service that points to these pods, you should be able to scrape all the pods of kong.

[ikethecoder]

Great thanks very much Harry! I was able to use the kubernetes_sd_configs as suggested:

    - job_name: 'kubernetes-kong'
      metrics_path: /metrics
      kubernetes_sd_configs:
      - role: pod
        namespaces:
          names:
          - ${var.namespace}
        selectors:
          - role: "pod"
            label: "app.kubernetes.io/name=kong"
            field: "status.phase=Running"
      relabel_configs:
      - source_labels: [ __meta_kubernetes_pod_container_port_number ]
        regex: '(8001)'
        action: keep
      - source_labels: [ __meta_kubernetes_pod_name ]
        action: replace
        target_label: instance