search

fluent / fluentd

Fluentd: Unified Logging Layer (project under CNCF)
https://www.fluentd.org
Apache License 2.0
12.9k stars 1.34k forks source link

Support for OTLP metrics export #3816

Open gfonseca-tc opened 2 years ago

gfonseca-tc commented 2 years ago

Is your feature request related to a problem? Please describe.

Most modern observability tools use to push metrics instead of letting a server pulling them. It's less prone to error since who is generating the metrics can decide when to send metrics. Also, when we have fluentd running in containers, it's even harder to keep track of all instances and collect metrics from all of them. Having them sending metrics to a centralized gateway on the other side is much easier. On top of that, OpenTelemetry is becoming a standard in the industry and it seems to make sense having support for it in FluentD.

Describe the solution you'd like

Having a way of sending OTLP metrics to a configured OTLP endpoint.

Describe alternatives you've considered

The Prometheus plugin is very versatile and the OpenTelemetry collector has a scraper for prometheus endpoints, making it easy to initially migrate to OTel. But it doesn't solve the problem of having FluentD containerized and collecting metrics from all containers. Enabling FluentD to send those metrics periodically to a gateway seems more reliable to me.

Additional context

No response

cosmo0920 commented 2 years ago

In Fluent Bit side, it will be handled by cmetrics library. If we will support OTEL handling, we have to use and depend on it.

cosmo0920 commented 2 years ago

If we add cmetrics gem as runtime dependency, we can handle OTEL and Prometheus context natively in Fluentd core.

gfonseca-tc commented 2 years ago

I would love to see this feature in place! Not sure how stable the cmetrics library is, but it seems like a good option.

cosmo0920 commented 1 year ago

Or, using otel official Ruby gem would be much better: https://github.com/open-telemetry/opentelemetry-ruby

alrz commented 1 week ago

This is also related to logs as well as metrics and traces, it would simplify ingestion on the app side with readily available otlp exporters, fluent-bit does support it but surprisingly fluentd does not..