search

fluent / fluent-bit

Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX and Windows
https://fluentbit.io
Apache License 2.0
5.71k stars 1.56k forks source link

Sending metrics from node_exporter or scrapers over forward protocol #5520

Open noahbailey opened 2 years ago

noahbailey commented 2 years ago

Currently, metrics gathered from node_exporter or prometheus scrapers must be output via remote write within the same fluentbit instance, becuase they are part of a 'different pipeline' from regular log-formatted data. This could be a limitation for some users who prefer to relay ALL data through one set of fluentbit nodes before splitting data off into different sinks. 

+--------------------+                +--------------------+
| Node A (fluentbit) |  ==forward==>  | Node B (fluentbit) |  ==remote_write output==>   Prometheus
| node_exporter      |                +--------------------+  \==s3 output==> S3 bucket
| journald, etc      |                                         \==> elastic/opensearch, etc
+--------------------+

This would help with a common use case which uses relays per zone to forward data to one centralized system which sends data to their storage back-ends.

edsiper commented 2 years ago

thanks for raising this ticket. This feature will be implemented shortly.

phwissmann commented 1 year ago

Just wanted to express my support for this use case. Is it on any roadmap?

edit: seems to be implemented and working in 2.0.6