Prometheus ServiceMonitors

[lukpre]

Hello everyone

I was wondering if there are any plans or if it's currently possible to get a Prometheus ServiceMonitor created upon deploying a postgres cluster/database through the postgres-operator ? Is there already an option for that or how could this be done ?

Besides using your operator we have a centralized monitoring infrastructure using the prometheus-operator on k8s and it would be great if metrics of deployed postgres clusters/databases would be made available via the servicemonitor object.

I've checked your pgMonitor Repository but this seems to deploy a complete monitoring stack and would include manual work on the prometheus side (if only using the exporters). Coulnd't find any other issues or questions regarding this that's why i was wondering.

Thank you very much for an answer Best regards

[jkatz]

I think the short answer is that there are no plans -- the idea with the monitoring installer is that it can give you a turnkey monitoring collection/visualization solution out-of-the-box, but the main components that the Postgres Operator itself provides is the exporter to be able to scrape the stats.

Now, if there is a concrete proposal where we could include a "one-size-fits-all" ServiceMonitor for Prometheus, I'd be willing to entertain including it as part of the installer, but my concern is that it could be too specific to each installation and I'm not sure if it's universally available?

1968 includes an ask for help with documentation on how one can connect their existing Prometheus/Grafana installations into the Postgres Operator. Perhaps there can be documentation around how one can add a ServiceMonitor into their installation.

The primary focus of the Postgres Operator is automation around PostgreSQL and its components to keep it up, running, healthy, usable etc. So I'd be willing to entertain some patches that help further that goal without getting to specialized in the independent components, but I don't see adding direct support for a ServiceMonitor for Prometheus to be on the roadmap.

[bilbof]

@lukpre these should work - I've verified they work for the latest release, I just took the values from crunchy data's examples repo and translated them.

---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: crunchy-postgres-exporter
  namespace: monitoring
spec:
  jobLabel: crunchy-postgres-exporter
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      postgres-operator.crunchydata.com/crunchy-postgres-exporter: "true"
  podMetricsEndpoints:
  - interval: 30s
    port: exporter
    scrapeTimeout: 10s
    relabelings:
    # Keep exporter port and drop all others
    # https://prometheus.io/docs/prometheus/latest/configuration/configuration/#pod
    - sourceLabels: [__meta_kubernetes_pod_container_port_number]
      action: keep
      regex: "9187"
    # Set label for namespace
    - sourceLabels: [__meta_kubernetes_namespace]
      targetLabel: kubernetes_namespace
    # Set label for pod name
    - sourceLabels: [__meta_kubernetes_pod_name]
      targetLabel: pod
    # Convert namespace and cluster name to pg_cluster=namespace:cluster
    - sourceLabels: [__meta_kubernetes_namespace,__meta_kubernetes_pod_label_postgres_operator_crunchydata_com_cluster]
      targetLabel: pg_cluster
      separator: ":"
      replacement: '$1$2'
    # Convert kubernetes pod ip to ip
    - sourceLabels: [__meta_kubernetes_pod_ip]
      targetLabel: ip
    # Convert postgres-operator.crunchydata.com/instance to deployment
    - sourceLabels: [__meta_kubernetes_pod_label_postgres_operator_crunchydata_com_instance]
      targetLabel: deployment
    # Convert postgres-operator.crunchydata.com/role to role
    - sourceLabels: [__meta_kubernetes_pod_label_postgres_operator_crunchydata_com_role]
      targetLabel: role
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: crunchy-postgres-exporter-v4
  namespace: monitoring
spec:
  jobLabel: crunchy-postgres-exporter-v4
  namespaceSelector:
    any: true
  selector:
    matchLabels:
      crunchy-postgres-exporter: "true"
  podMetricsEndpoints:
  - interval: 30s
    port: exporter
    scrapeTimeout: 10s
    relabelings:
    # Keep exporter port and drop all others
    - sourceLabels: [__meta_kubernetes_pod_container_port_number]
      action: keep
      regex: "9187"
    # Set label for namespace
    - sourceLabels: [__meta_kubernetes_namespace]
      targetLabel: kubernetes_namespace
    # Set label for pod name
    - sourceLabels: [__meta_kubernetes_pod_name]
      targetLabel: pod
    # Convert namespace and cluster name to pg_cluster=namespace:cluster
    - sourceLabels: [__meta_kubernetes_namespace,__meta_kubernetes_pod_label_pg_cluster]
      targetLabel: pg_cluster
      separator: ":"
      replacement: '$1$2'
    # Convert kubernetes pod ip to ip
    - sourceLabels: [__meta_kubernetes_pod_ip]
      targetLabel: ip
    # Set deployment_name as deployment label
    - sourceLabels: [__meta_kubernetes_pod_label_deployment_name]
      targetLabel: deployment
    # Set label for role
    - sourceLabels: [__meta_kubernetes_pod_label_role]
      targetLabel: role