grafana / alloy

OpenTelemetry Collector distribution with programmable pipelines
https://grafana.com/oss/alloy
Apache License 2.0
1.45k stars 216 forks source link

[Proposal] Log rate limiting globally. #2065

Open cyriltovena opened 1 week ago

cyriltovena commented 1 week ago

Background

Right now the stage limit for log only apply locally to a single alloy instance. see https://grafana.com/docs/alloy/latest/reference/components/loki/loki.process/#stagelimit-block which is a bit impractical to use for rate limiting in large clusters where users want to apply rate limiting globally

Proposal

Not sure if this is possible with cluster management but it would be nice to apply the rate limiting globally across all agents if they form a cluster.

We most likely want to use a different block such as stage.global_limit

wildum commented 1 week ago

One way to solve this with the current solution is to put the component in a module. You can store the module in a git repo and all the Alloy instances will pull it regularly. Not only this allows you to change all the rate limits live across all the instances, but you can also change other parts of the pipeline this way. Another similar approach is to use fleet management