search

prometheus / client_golang

Prometheus instrumentation library for Go applications
https://pkg.go.dev/github.com/prometheus/client_golang
Apache License 2.0
5.3k stars 1.16k forks source link

Document the bridge that allows for client_golang instrumented code to push with OpenTelemetry OTLP #1495

Open bwplotka opened 4 months ago

bwplotka commented 4 months ago

Thanks to amazing community behind OpenTelemetry Go SDK (special kudos to @dashpole), users of client_golang can optionally import a new Go module, a special bridge [1][2] and push data from prometheus.Registry using OTLP push.

This is useful if you have your application metrics instrumented with Prometheus client_golang, but wish to use OTLP push instead of (or additionally to) promhttp /metrics pull method, because perhaps it's a batch job or you want to use push based collection mechanisms with e.g. OTel collector. Notably, initial benchmarks shows that, currently, this way is more efficient than e.g. pulling metrics using Otel collector Prometheus receiver and converting to OTLP data model. This situation, however, can change as the source of the efficiency issues can be, in theory, optimized away.

Opinionated disclaimer: Using OTLP push (or generally push) has its own disadvantages. Converting Prometheus metrics to OTLP format has another set of disadvantages. Our recommendation is to prefer pull mechanisms for metric pipelines for reliability, centralised discovery, efficiency and simple client failover scenarios. Standard promhttp /metrics pull mechanism remains the most efficient and reliable way to directly get metrics from your application to Prometheus or Prometheus compatible backend.

Nevertheless for those who wants to leverage pure OTLP, this issue is to track adding relevant explanation to our docs. This is also on part with Java Prometheus client offering OTLP

kakkoyun commented 4 months ago

Opinionated disclaimer: Using OTLP push (or generally push) has its own disadvantages. Converting Prometheus metrics to OTLP format has another set of disadvantages. Our recommendation is to prefer pull mechanisms for metric pipelines for reliability, centralised discovery, efficiency and simple client failover scenarios. Standard promhttp /metrics pull mechanism remains the most efficient and reliable way to directly get metrics from your application to Prometheus or Prometheus compatible backend.

I can't emphasize this point more

kakkoyun commented 4 months ago

but wish to use OTLP push instead of (or additionally to) promhttp /metrics pull method, because perhaps it's a batch job or you want to use push based collection mechanisms with e.g. OTel collector

In theory, this approach would eliminate the use of https://github.com/weaveworks/prom-aggregation-gateway?tab=readme-ov-file (or maintained version https://github.com/zapier/prom-aggregation-gateway)

dashpole commented 4 months ago

@bwplotka were you thinking of an example, or just documentation? If documentation, where do you think this belongs?