search

grafana / alloy

OpenTelemetry Collector distribution with programmable pipelines
https://grafana.com/oss/alloy
Apache License 2.0
1.32k stars 181 forks source link

Make `loki.source.kubernetes_events` clusterable in flow-mode #401

Open braunsonm opened 1 year ago

braunsonm commented 1 year ago

Request

I would like the loki source for kubernetes_events to be able to distribute the logs or only run on a single node when in clustered mode on the Grafana-Agent when deployed with Flow mode.

Unfortunately there doesn't seem to be a way to use loki.source.kubernetes_events if you are only running a cluster of Grafana-Agents in flow mode. If you add this to your configuration you will get duplicate logs.

Use case

For ease of use, we are wanting to run a cluster of grafana-agents in flow-mode to handle our logging and metric ingestion needs. Right now we'd need to deploy some static grafana agents and some flow-mode agents to accomplish this.

rfratto commented 1 year ago

Given the nature of how loki.source.kubernetes_events works, it's unlikely that the work can be distributed, so we might want to solve this via grafana/alloy#399 rather than introducing a clustering block.

cc @tpaschalis

RichardSufliarsky commented 1 year ago

Similar requirement here. We are running grafana-agent in flow mode as a daemonset to be able to collect logs from each node. With 21 nodes running, we are getting additional 20 duplicates for each kubernetes event logged.

We don't need to distribute the work, we just need to be sure that there is exactly one instance of loki.source.kubernetes_events component collecting the data.

andrejshapal commented 3 weeks ago

This looks like good feature as running same component on 20+nodes and then loki have to deduplicate it, is not the best approach.