Why Alloy in StatefulSet mode?

[ppawiggers]

The default way to deploy an Alloy controller is DaemonSet mode. Why does this chart override that setting to be StatefulSet mode?

If there's a good reason for it, what would be the suggested way to scale Alloy for metrics scraping? Change to DaemonSet, or increase the number of replicas? How do I make sure metrics aren't scraped twice? Does it filter on node name?

My cluster has ~1000 nodes.

[pioardi]

I have the same questions as @ppawiggers , so just commenting for visibility. @ppawiggers I am going to try to set it to DaemonSet even if I see that the configuration has an @ignored comment on it, hopefully we are able to go for a daemon set.

Update - I have been able to set the controller type to Deployment and also to configure autoscaling and pod resources

[petewall]

This doc should hopefully explain things a bit: https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/Structure.md

TL;DR:

• alloy is a statefulset because it:
  1. Doesn't need to be a daemonset, it's not required to be on every node
  2. It set up easily to have its own storage for write-ahead logging
• alloy-events is a single-instance deployment because it doesn't need storage and if it scales beyond a single instance it'll start sending duplicate events
• alloy-logs is a daemonset because the default way it gathers pod logs is by accessing each individual node's filesystem.

So, it's set up with these footprints because it's the most optimal deployment pattern for the type of work each alloy instance is doing.