Prometheus exporter example

[dobesv]

I was able to hook this up to prometheus by adding a sidecar container to the deployment:

      - name: prometheus-exporter
        image: quay.io/prometheus/haproxy-exporter:v0.9.0
        args:
          - --haproxy.scrape-uri=http://127.0.0.1:1936/haproxy?stats;csv
        ports:
          - name: prom-metrics
            containerPort: 9101

And using CoreOS prometheus operator, a service and servicemonitor:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: haproxy-prom-metrics
  name: haproxy-prom-metrics
  namespace: haproxy
spec:
  ports:
  - name: prom-metrics
    port: 9101
    targetPort: prom-metrics
  selector:
    app: haproxy
  sessionAffinity: None
  type: ClusterIP

---

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: haproxy
  namespace: prometheus
  labels:
    app: haproxy-prom-metrics
spec:
  jobLabel: app
  selector:
    matchLabels:
      app: haproxy-prom-metrics
  namespaceSelector:
    matchNames:
    - haproxy
  endpoints:
  - port: prom-metrics
    interval: 30s
    path: /metrics

Maybe this information will help somebody.

[jcmoraisjr]

Great, just labeling this as an enhancement in order to properly update the examples. Tks!

[remoe]

Use this prometheus deployment as a base installation:

https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus/manifests

Here is a working sample:

Add this to haproxy-ingress manifest:

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  labels:
    run: haproxy-ingress
  name: haproxy-ingress
  namespace: ingress-controller
spec:
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      run: haproxy-ingress
  template:
    metadata:
      labels:
        run: haproxy-ingress
    spec:
      hostNetwork: true
...
      serviceAccountName: ingress-controller
      containers:
...
      - name: prometheus-exporter
        image: quay.io/prometheus/haproxy-exporter:v0.9.0
        args:
          - --haproxy.scrape-uri=http://127.0.0.1:1936/haproxy?stats;csv
        ports:
          - name: prom-metrics
            containerPort: 9101              

apiVersion: v1
kind: Service
metadata:
  labels:
    app: haproxy-prom-metrics
  name: haproxy-prom-metrics
  namespace: ingress-controller
spec:
  ports:
  - name: prom-metrics
    port: 9101
    targetPort: prom-metrics
  selector:
    run: haproxy-ingress
  sessionAffinity: None
  type: ClusterIP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: haproxy
  namespace: monitoring
  labels:
    app: haproxy-prom-metrics
    k8s-app: haproxy-metrics
spec:
  jobLabel: app
  selector:
    matchLabels:
      app: haproxy-prom-metrics
  namespaceSelector:
    matchNames:
    - ingress-controller
  endpoints:
  - port: prom-metrics
    interval: 30s
    path: /metrics

It's important to know that Prometheus manifest match the Service Monitors:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
...
  serviceMonitorSelector:
    matchExpressions:
    - key: k8s-app
      operator: Exists

After this, one can add/import the following dashboard:

https://grafana.com/dashboards/367

[AlexProfi]

Use this prometheus deployment as a base installation:

https://github.com/coreos/prometheus-operator/tree/master/contrib/kube-prometheus/manifests

I use this example and get error

error: unable to recognize "haproxy/monitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1" How to fix this?

[AlexProfi]

Also need this steps. Maybe not all manifests are needed but I install it all for quick test

git clone https://github.com/coreos/prometheus-operator.git

cd prometheus-operator/contrib/kube-prometheus 

kubectl apply -f manifests/

[onelapahead]

Sorry to dig up an old thread, but I'm wondering if anyone is able to quickly articulate why using the above exporter would be more favorable than using the built-in Prometheus metrics from /metrics on port 10254?

I'm going to do my own testing but if anyone from this thread has already done a comparison that'd be very helpful.

[dobesv]

Does HAProxy have native prometheus metrics support now?

[onelapahead]

@dobesv I do not believe HAProxy itself has native Prometheus support, but it seems this particular Ingress controller does, here are the docs where its mentioned: https://haproxy-ingress.github.io/docs/configuration/command-line/#stats

And some GH issues that allude to it:

• https://github.com/jcmoraisjr/haproxy-ingress/issues/271 (think this one could be closed @jcmoraisjr)
• https://github.com/jcmoraisjr/haproxy-ingress/issues/569
• https://github.com/jcmoraisjr/haproxy-ingress/issues/897

So I'm just wondering if folks have found the controller's built-in metrics sufficient, or if they prefer the exporter's metrics instead.

[dobesv]

I think the controller is outputting its own metrics, rather than all the HAProxy metrics, e.g. the ones in this file:

https://github.com/jcmoraisjr/haproxy-ingress/blob/42034000bc7e4516696fc7f107b6a2d9b82588a3/pkg/controller/metrics.go

Have you tried fetching the metrics and seeing if it has all the HAProxy metrics in there?

[onelapahead]

Ah I see the distinction, thank you @dobesv! I did hit the metrics endpoint but it was on a fresh deployment of the Ingress controller so there wasn't much to look through.

That makes sense that one might want both: the metrics for the controller itself, and then the exporter to get metrics for the HAProxy process.

Will do some testing and post my updated example that has a ServiceMonitor which will scrape both the controller's metrics and the exporter's.

[jcmoraisjr]

Just my 2cents: 10254 has the controller metrics and also some haproxy metrics from the controller's perspective, e.g. the proxy latency and a more accurate idle_pct compatible measure, see some words about them in the faq. HAProxy has also it's own prometheus compatible /metrics since 2.0 and controller's doc talks a bit about the differences and links the upstream doc as well. Hth.