Feat/Add support to blackbox-exporter to prometheus-exporter-operator

[slopezz]

Closes https://github.com/3scale/platform/issues/456

This PR adds support for blackbox exporter to prometheus-exporter-operator.

• Added possibility to deploy a blackbox_exporter using PrometheusExporter type probe (to monitor http endpoints, tcp endpoints... it depends on what you configure on the configmap)

• Added Grafana Dashboard to view stats of targets monitored by a specific blackbox-exporter

• Added full examples of what is needed to monitor different N targets:

  • A single PrometheusExporter type probe object
  • A single Configmap with the blackbox modules configuration (like http_2xx, tcp_connect...), that will be used by deployed PrometheusExporter type probe
  • N ServiceMonitor objects that needs to point to the single deployed probe PrometheusExporter type probe, which requires a specific relabeling of source_label __param_target into target_label target

• Adds documentation about the full provided examples of new probe exporter type

• Add examples of possible prometheus rules (alerts) using probe metrics (like status, or SSL Certificate expiration).

Taking into account that prometheus-operator had called Probe to the new custom resource that uses blackbox_exporter, it has been decided to use the same naming convention PrometheusExporter type probe (details on https://github.com/prometheus-operator/prometheus-operator/pull/2832).

[slopezz]

@slopezz wouldn't it be possible to add a field in the Probe spec to list the endpoints you want checked and have that automatically create the required ServiceMonitors? Seems a bit weird to me having to manually manage those.

@roivaz The purpose of the operator is deploying easily different prometheus exporters using simple PrometheusExporter resource (so you don't need to know that behind the scenes, there is a service, deployment with specific images, args, commands, secret/configmap mounts... and then a ServiceMonitor and its GrafanaDashboard).

For this specfic case of blackbox, apart from deploying the exporter, in order to be able to use it because by its own does nothing, you need to create extra resources for every endpoint to monitor:

• The idea would be to use the new Probe resource (where you need to specify the probe url of the deployed blackbox_exporter):

  apiVersion: monitoring.coreos.com/v1
  kind: Probe
  metadata:
  name: prometheus-exporter-probe-http-kubernetes-io
  spec:
  prober:
  url: prometheus-exporter-probe-staging-blackbox:9115
  module: http_2xx
  targets:
  staticConfig:
    static:
      - https://kubernetes.io

• But taking into account this is a pretty new resource, so it is very possible it won't be available on your cluster, you can achieve the same result using the more standard ServiceMonitor resource. Instead of using using blackbox prober.url field, you need to use the selector.matchLabels pointing to the blackbox exporter service (so the same content on both cases).

  apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
  name: prometheus-exporter-probe-http-kubernetes-io
  spec:
  endpoints:
  - interval: 30s
  path: /probe
  port: metrics
  relabelings:
    - sourceLabels: [__param_target]
      targetLabel: target
  params:
    module:
    - http_2xx
    target:
    - https://kubernetes.io
  selector:
  matchLabels:
    app: prometheus-exporter-probe-staging-blackbox 

• In addition, the relabelling of the target label is intended for the dashboard only (you need a way to filter which probe is being used). Normally, when a metric comes from the exporter itself (like mysql, redis...), all source labels remains the same on prometheus after being scraped by the ServiceMonitor (and you can use these metrics directly on grafana), however on that case, where you have an object (Probe/ServiceMonitor) pointing to the real deployed exporter (prober.url/selector.matchLabels), this target labels disappears, and is converted to __param_target, so in order to keep a label that identifies the probe, the relabelling is needed.

It would be nice to integrate an array of target/module into the blackbox PromerheusExporter resources (so you forget to create individual Probes/ServiceMonitors, but would be difficult to manage):

• Operator would require to create a new object with a unique name (important), and add the module/target values
  • name field should be required on the CR? maybe it could be used the hash of the target? it needs to be complian with kubernetes object naming (no dots, no special chars...)
• The problem comes when you delete targets from the array of targets, from ansible would be difficult to remove them, because if it disappear, on the next reconcile will ignore if it existed before and will be needed to be removed manually (don't know if I am explaining myself

So to summarize, I think by the moment, I would go with the standard prometheus probes monitoring, where you deploy an independent blackbox-exporter (with a PromenetheusExporter resource) and then create N resources for every target (Probes or ServiceMonitors) pointing to that deployed exporter, and then, the relabelling of the target label is intended for the dashboard only.