Glusterfs prometheus exporter

[prabhanjanps]

We have installed glusterfs as a two node cluster.This is being consumed by k8s via heketi. Prometheus runs as a pod in k8s. We have installed this Prometheus exporter for Gluster Metrics on the gluster peers and are able to see the metrics at http://xx.xx.xx.xx:8080/metrics.

How to add this metrics in the prometheus as target?

Some blogs guide us to modify the config.yaml of prometheus , but we're not able to find it. Some also pointed towards configmap of prometheus.

Any thoughts? is that all thats reruired?

[aravindavk]

As per my understanding we need to set some annotations to exporter pod so that Prometheus operator detects that annotations and automatically adds to its configuration.

@sidharthanup please share the details if you have any easy way to set this up in k8s

[sidharthanup]

@prabhanjanps Have you deployed Prometheus using the Prometheus operator? If yes, then you should define a k8s service to pick up the metrics from the exporter i.e the glusterfs pods using the label selector. Here's an example of a service that selects the gluster pod( Note: This is used in the GCS project which uses glusterd2 and not glusterd ) using the specified labels under spec.selector, picks up metrics from port 8080 and exposes the metrics at port 8080: https://github.com/gluster/gcs/blob/master/deploy/templates/gcs-manifests/gcs-gd2-services.yml.j2#L30 . Now the next step would be to define a ServiceMonitor CRD. This should pick up the service you just now defined. Then define the Prometheus CRD that picks up the Service Monitor. And finally define a NodePort Service to expose the final set of structured metrics exposed by the prometheus server outside the cluster. Example: https://github.com/gluster/gcs/blob/master/deploy/templates/gcs-manifests/gcs-prometheus-bundle.yml.j2 . Also refer: https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md , for information on how to deploy Prometheus Operators and the CRDs. Basically service(that exposes metrics from the exporter) --> ServiceMonitor CR --> Prometheus CR . Now how the Prometheus Operator works is it detects this configuration( that the Prometheus CR generates ) and deploys Prometheus servers that uses this config and manages state of the servers during later config changes.

Although we haven't tried it, if you are using standalone Prometheus server ( without the operator ), I would suggest using a config map. And the config file should be prometheus.yml and not config.yaml. Also make sure that in the Prometheus deployment yaml, you have specified the volumeMount correctly and it should be /etc/prometheus/.Here's an example prometheus deployment :

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus-server
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus-server
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:v2.1.0
          args:
            - "--config.file=/etc/prometheus/prometheus.yml"
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: prometheus-config
              mountPath: /etc/prometheus/
      volumes:
        - name: prometheus-config
          configMap:
            name: prometheus-server-conf

And then deploy the config map. Use https://prometheus.io/docs/prometheus/latest/configuration/configuration/#kubernetes_sd_config in the scrape config.

The recommended way to deploy prometheus would be using the Prometheus Operator. You just have to deploy the CRDs and the operator generates the config dynamically rather than you implicitly specifying it via a config map. Again, https://github.com/coreos/prometheus-operator/blob/master/Documentation/user-guides/getting-started.md is an excellent guide on how to deploy everything.

[airstring]

We have installed glusterfs as a two node cluster.This is being consumed by k8s via heketi. Prometheus runs as a pod in k8s. We have installed this Prometheus exporter for Gluster Metrics on the gluster peers and are able to see the metrics at http://xx.xx.xx.xx:8080/metrics.

How to add this metrics in the prometheus as target?

Some blogs guide us to modify the config.yaml of prometheus , but we're not able to find it. Some also pointed towards configmap of prometheus.

Any thoughts? is that all thats reruired?

can you give me an up-to-date binary file? thanks!

[aravindavk]

can you give me an up-to-date binary file? thanks!

I started working on a different approach(https://github.com/kadalu/gluster-metrics-exporter) to provide the Prometheus exporter for GlusterFS. It is in a very early stage of development, but we from https://github.com/kadalu actively working on Monitoring and management solutions for GlusterFS. Let us know if this new exporter is useful to you.