Add servicemonitor config alongside plain prometheus config

[surajssd]

In the doc "Bring your own Prometheus" it is worth adding ServiceMonitor configuration alongside plain Prometheus config for those who use Promethues Operator to manage their monitoring stack.

[dnaranjor]

Nice feature this one! I'm facing precisely this in my cluster, and I'm tackling with this now

In this document: https://linkerd.io/2.10/tasks/exporting-metrics/ you could find some useful information about how to deploy a ServiceMonitor object and achieve this goal

[dnaranjor]

Editing the following ServiceMonitor object, need to better understand how Federation works, since in our setup we have only one Prometheus instance running to observe all cluster services:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    k8s-app: linkerd-prometheus
    release: monitoring
  name: linkerd-federate
  namespace: {{.Namespace}}
spec:
  endpoints:
  - interval: 30s
    scrapeTimeout: 30s
    params:
      match[]:
      - '{job="linkerd-proxy"}'
      - '{job="linkerd-controller"}'
    path: /federate
    port: admin-http
    honorLabels: true
    relabelings:
    - action: keep
      regex: '^prometheus$'
      sourceLabels:
      - '__meta_kubernetes_pod_container_name'
  jobLabel: app
  namespaceSelector:
    matchNames:
    - {{.Namespace}}
  selector:
    matchLabels:
      component: prometheus

[alpeb]

What's described under https://linkerd.io/2.10/tasks/exporting-metrics/ is simply for exporting Linkerd's Prometheus data into a separate, probably longer-lived, Prometheus instance. So the separate Prometheus instance only needs to scrape one endpoint, i.e. the /federate endpoint of Linkerd's Prometheus instance, thus the simplicity of the ServiceMonitor exemplified.

The present issue on the other hand, is about not using Linkerd's Prometheus instance, but using a separate one as described in https://linkerd.io/2.10/tasks/external-prometheus/, configured to scrape all the pods that have been injected with Linkerd, but using a ServiceMonitor instead of directly in a ConfigMap as shown in that doc. You can see that scrape config is not trivial. I'm not sure if that's possible to do through a ServiceMonitor at all. A PodMonitor might be a better shot.

So to summarize this requires translating the scrape config found in the original Linkerd Prom config in terms of a PodMonitor spec. Massive Internet Points awaiting if someone from the community achieves that! :wink:

[cpretzer]

Thanks for the detailed analysis @alpeb

I agree that this would be a great doc/tutorial

[0dragosh]

+1 for this, it would be very helpful

[Becram]

+1 that would be great

[alpeb]

Turns out the Lokomotive project, which embeds linkerd, already provides a ServiceMonitor: https://github.com/kinvolk/lokomotive/blob/master/assets/charts/components/linkerd2/templates/service-monitors.yaml We'd be glad to include something like that as an optional resource in the linkerd2 repo itself :wink:

[FredyR4zox]

Hello,

I have done the PodMonitors that linkerd's prometheus uses. I have tested and it works. The only PodMonitor that doesn't get any service discovery is linkerd-service-mirror. But even with linkerd's prometheus it does not get any, so it does not matter.

Here it goes:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  labels:
    app: linkerd
    release: prometheus-stack
  name: linkerd-controller
  namespace: monitoring
spec:
  namespaceSelector:
    matchNames:
      - linkerd-viz
      - linkerd
  selector:
    matchLabels: {}
  podMetricsEndpoints:
    - relabelings:
      - sourceLabels:
        - __meta_kubernetes_pod_container_port_name
        action: keep
        regex: admin-http
      - sourceLabels:
        - __meta_kubernetes_pod_container_name
        action: replace
        targetLabel: component
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  labels:
    app: linkerd
    release: prometheus-stack
  name: linkerd-service-mirror
  namespace: monitoring
spec:
  namespaceSelector:
    any: true
  selector:
    matchLabels: {}
  podMetricsEndpoints:
    - relabelings:
      - sourceLabels:
        - __meta_kubernetes_pod_label_linkerd_io_control_plane_component
        - __meta_kubernetes_pod_container_port_name
        action: keep
        regex: linkerd-service-mirror;admin-http$
      - sourceLabels:
        - __meta_kubernetes_pod_container_name
        action: replace
        targetLabel: component
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  labels:
    app: linkerd
    release: prometheus-stack
  name: linkerd-proxy
  namespace: monitoring
spec:
  namespaceSelector:
    any: true
  selector:
    matchLabels: {}
  podMetricsEndpoints:
    - relabelings:
      - sourceLabels:
        - __meta_kubernetes_pod_container_name
        - __meta_kubernetes_pod_container_port_name
        - __meta_kubernetes_pod_label_linkerd_io_control_plane_ns
        action: keep
        regex: ^linkerd-proxy;linkerd-admin;linkerd$
      - sourceLabels: [__meta_kubernetes_namespace]
        action: replace
        targetLabel: namespace
      - sourceLabels: [__meta_kubernetes_pod_name]
        action: replace
        targetLabel: pod
      - sourceLabels: [__meta_kubernetes_pod_label_linkerd_io_proxy_job]
        action: replace
        targetLabel: k8s_job
      - action: labeldrop
        regex: __meta_kubernetes_pod_label_linkerd_io_proxy_job
      - action: labelmap
        regex: __meta_kubernetes_pod_label_linkerd_io_proxy_(.+)
      - action: labeldrop
        regex: __meta_kubernetes_pod_label_linkerd_io_proxy_(.+)
      - action: labelmap
        regex: __meta_kubernetes_pod_label_linkerd_io_(.+)
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
        replacement: __tmp_pod_label_$1
      - action: labelmap
        regex: __tmp_pod_label_linkerd_io_(.+)
        replacement:  __tmp_pod_label_$1
      - action: labeldrop
        regex: __tmp_pod_label_linkerd_io_(.+)
      - action: labelmap
        regex: __tmp_pod_label_(.+)

[junnplus]

any progress?