Add prometheus exporter

[lukas-vlcek]

FYI, we have recently released new version of Prometheus exporter plugin (for OpenSearch 1.3.0.) here https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/tag/1.3.0.0

Happy to discuss any missing features in /issues section.

[segalziv]

Thanks for the note @lukas-vlcek . We still haven't setup the monitoring through prom/grafana. If you would like to join the project and contribute the monitoring integration that would be great.

[albertollamaso]

@segalziv what would be the metrics you would like to export. I can try to raise a PR adding prometheus exporter to the controler

[segalziv]

@albertollamaso thanks. We plan to integrate one of the OpenSearch open source plugins for Prometheus exporter.

Either

https://github.com/aiven/prometheus-exporter-plugin-for-opensearch which is updated, or https://github.com/aparo/opensearch-prometheus-exporter

The idea is that once adding config to the cluster CRD, it would install the plugin, and export through service. That's a high level draft, open to suggestions.

[rursprung]

@segalziv: the one from aparo is AFAIK discontinued, the maintained one is the one from aiven.

once an application exposes the prometheus metrics on an unsecured HTTP endpoint you can "just" add the following k8s annotations (to the template of the StatefulSet in this case, not the StatefulSet itself, of course):

      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/scheme: 'http'
        prometheus.io/port: '9200'
        prometheus.io/path: '/_prometheus/metrics'

and of course add a NetworkPolicy which grants prometheus access to it.

the main issue will be that the security plugin interferes with the prometheus plugin, see https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/issues/60 (and the older issues linked there at the beginning).

the discussion in this ticket so far has centered around OpenSearch. what about OpenSearch Dashboards (i'm not aware of a prometheus exporter plugin for it yet, there was this discussion on the one for kibana a while ago: https://github.com/pjhampton/kibana-prometheus-exporter/issues/235)? and what about the opensearch-k8s-operator itself (you'd have to collect metrics and expose them on an HTTP endpoint and annotate your pods accordingly)?

note that it doesn't hurt to have this enabled even if somebody wouldn't have prometheus set up - then there are just a couple of additional endpoints running, exposing metrics and that's it.

[idanl21]

Hey @rursprung, quick question - is the exporter is able to work with security plugin enabled? if it is, please add to the CRD under 'general' an Boolean 'exporter' filed, if it set to true, add the annotations and expose the metrics. about opensearch-dashboard && opensearch-k8s-operator itself, lets focus on the opensearch exporters for now, when we will have the working logic of it we will add support for all the other exporters . Thanks, let me know of you have any question

[idanl21]

Implemented on that PR, closing the issue https://github.com/Opster/opensearch-k8s-operator/pull/415