Prometheus remote writer

[p4k8]

Is your feature request related to a problem? Please describe.

Problem is storing perfdata in a Prometheus-compatible storage (Victoriametrics, Cortex, Thanos, Mimir etc) for it to be compatible with other monitoring tools while being cost-efficient.

Describe the solution you'd like

Native Prometheus remote protocol writer, similar to InfluxDB and OpenTSDB perfdata writers.

Describe alternatives you've considered

As of now I employ workaround consisting of writing with InfluxDB writer to the vmagent tool of Victoriametrics suite. It works fine most of the time, but data model is different and it might name and store things in somewhat awkward way due to conversion.

Additional context

https://prometheus.io/docs/concepts/remote_write_spec/ https://github.com/google/snappy/tree/main https://github.com/prometheus/compliance/tree/main/remote_write_sender

[martialblog]

Just some thoughts on this: Instead of a Writer for Prometheus alone it might be even more beneficial to use OpenTelemetry https://opentelemetry.io/

This would allow for sending data to even more metric backends, including Prometheus.

Prometheus will move also move to OTel in the longterm: https://prometheus.io/blog/2024/03/14/commitment-to-opentelemetry/

Edit: Elastic and Influx are on their way to fully support OTel as well. Another argument for an OTel Writer.

Together with tools like the OpenTelemetry Collector or Telegraf this would enable Icinga2 to be extremely flexible for writing Perfdata.