search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.89k stars 3.45k forks source link

[canary] Option to disable websocket tailing #11662

Open verejoel opened 10 months ago

verejoel commented 10 months ago

Is your feature request related to a problem? Please describe. It would be nice to to run the Loki canary with spot-checks and metric-checks only, without needing to open a websocket connection.

Describe the solution you'd like A flag allowing one to disable the websocket checks.

Describe alternatives you've considered Run canary on only a subset of devices, to keep the number of websocket connections to a minimum.

Additional context This would allow us to run a canary on every single one of our 10s of thousands of edge devices, where we want to be confident that all telemetry from every device reaches us. Having so many concurrent websocket connections would probably negatively affect our central Loki, but spot checks and metric checks are much more manageable.

dannykopping commented 10 months ago

Seems reasonable!

pjastrzabek commented 1 month ago

To add to that, right now canary always runs as a daemon set. Would be great to have a way of changing this to deployment with certain number of replicas.

We use it as a self monitoring tool that runs on a cluster with many different types of nodes and having canary run on each of them is waste of resources (right now we limit this artificially with node selector)

So in general having more configuration options for various checks AND workload type (daemonSet or deployment) - example how traffic does that here