fluent / fluent-plugin-prometheus

A fluent plugin that collects metrics and exposes for Prometheus.
Apache License 2.0
257 stars 80 forks source link

aggregated_metrics gives incomplete output #175

Open pranavmarla opened 3 years ago

pranavmarla commented 3 years ago

Details

I have Fluentd set up with multiple workers and multiple plugins. Because of that, there are many Prometheus metrics. When I hit the /metrics endpoint for individual workers, there are no issues. However, when I hit the _/aggregatedmetrics endpoint, I only get partial output -- the output seems to abruptly cut off after a certain point. Does _/aggregatedmetrics have a size limit??

Eg. If I hit the _/aggregatedmetrics endpoint (curl http://localhost:24240/aggregated_metrics), here are the last 3 lines of the resulting output:

fluentd_output_status_slow_flush_count{worker_id="0",plugin_id="object:be0",type="null"} 0.0
fluentd_output_status_slow_flush_count{worker_id="0",plugin_id="object:463c",type="relabel"} 0.0
fluentd_output_status_slow_flush_count{worker_id="0",plugin_id="object:465

Notice that the last line is abruptly cut off mid-way!

Prometheus Config

<source>
  @type prometheus
  port 24240
  metrics_path /metrics
  aggregated_metrics_path /aggregated_metrics
</source>

<source>
  @type prometheus_output_monitor
</source>

Env Info

raviepgithub commented 2 years ago

I'm also facing similar issue. Is this working fine for you now?