jakubdyszkiewicz commented 4 years ago

Problem

Our current implementation of Prometheus integration on K8S relies on adding prometheus.io/scrape, prometheus.io/path and prometheus.io/port annotations. The problem is that those annotations are placed on Pod and there is support only for 1 value which means that scraping metrics from both a dataplane and an application is not possible.

The second problem is that those annotations are placed by injector therefore after turning the metrics on, you have to redeploy for pods.

Explored Solutions

Prometheus Operator

Prometheus Operator exposes some CRD which enables the user to configure additional scraping targets. I tried to use PodMonitor

kind: PodMonitor
metadata:
  name: kuma-pod-monitor
  namespace: default
spec:
  selector:
    matchLabels:
      kuma.io/sidecar-injected: true
  podMetricsEndpoints:
    - targetPort: 5670
      path: /metrics

but it seems to work only with labels, not with annotations. Not sure if adding labels by injector is a good idea. It also requires to coordinate configuration of port/path in mesh and here. Even so, this solution would only work for Kubernetes Operator, we need more broad approach.

Kubernetes SD

Prometheus ships with Kubernetes SD, which let's you define from which pods you could scrape the metrics. Assuming that injector would add following annotations prometheus.metrics.kuma.io/port, prometheus.metrics.kuma.io/path, prometheus.metrics.kuma.io/scrape. We can apply following config

    - job_name: kuma-dataplanes
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_metrics_kuma_io_scrape
      - action: replace
        target_label: __metrics_path__
        regex: (.+)
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_metrics_kuma_io_path
      - action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_metrics_kuma_io_port]

Pros:

It works with any Prometheus deployment
It doesn't require any external binary

Cons:

There is no information about mesh since Pod does not know about it
You have to redeploy Pod to apply it (it relies on extra annotations that injector would add)

Note: you can't see all labels from the pod, but those can be mapped using action: labelmap

Kuma Prometheus SD

To integrate Prometheus on Universal environments we introduced kuma-prometheus-sd binary which generates file that is later consumed by prometheus' file_sd plugin. This can be also used on K8S. You have to deploy this binary next to prometheus server.

To do this you have to modify Prometheus server deployment adding new container

      containers:
      - name: kuma-prometheus-sd
        image: kong-docker-kuma-docker.bintray.io/kuma-prometheus-sd:0.3.2
        imagePullPolicy: Always
        args:
          - run
          - --log-level=debug
          - --name=test-01
          - --cp-address=http://kuma-control-plane.kuma-system:5681
          - --output-file=/var/run/kuma.io/prometheus-sd/kuma.file_sd.json
        volumeMounts:
          - mountPath: "/var/run/kuma.io/prometheus-sd"
            name: kuma-prometheus-sd-volume

and volume that is used for both containers

      volumes:
      - name: kuma-prometheus-sd-volume
        emptyDir: {}

and mount a new volume into existing container

        volumeMounts:
        - mountPath: "/var/run/kuma.io/prometheus-sd"
          name: kuma-prometheus-sd-volume

Then you need to change the Prometheus config

- job_name: 'kuma-dataplanes'
  scrape_interval: "5s"
  file_sd_configs:
  - refresh_interval: 1s
    files:
    - /var/run/kuma.io/prometheus-sd/kuma.file_sd.json

Pros:

It's dynamic. You don't have to redeploy the pods. You can change the port or path on the fly.
Information about the mesh
Consistent behaviour with Universal deployment

Cons:

Requires an extra binary
Requires modification of the prometheus deployment

Summary

Using Kuma Prometheus SD seems to be the best option from the end user perspective (enabling/disabling metrics, querying by mesh). The question is - is there a way to improve UX of configuring Prometheus this way or is it good enough with the proper instruction?