search

canonical / loki-k8s-operator

This charmed operator automates the operational procedures of ryunning Grafana Loki, an open-source logs backend, in monolithic mode
https://charmhub.io/loki-k8s
Apache License 2.0
10 stars 16 forks source link

warn: Maximum active stream limit exceeded #432

Open marcusboden opened 2 months ago

marcusboden commented 2 months ago

Bug Description

Hi,

we're getting the warning that we exceeded the max active stream limit. This can be configured, but the charm doesn't offer any way of configuring that. Could we either make that configurable or raise it by default?

To Reproduce

Not 100% sure. Maybe we have a larger number of apps here than usual.

Environment

Running on microk8s on a customer environment:

loki                             2.9.5    active      1  loki-k8s                      latest/edge    105

Relevant log output

2024-06-28T07:11:30.436Z [loki] level=warn ts=2024-06-28T07:11:30.378520587Z caller=grpc_logging.go:57 method=/logproto.Pusher/Push duration=489.418µs err="rpc error: code = Code(429) desc = Maximum active stream limit exceeded, reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased, user: 'fake'" msg=gRPC

Additional context

(the user fake was a bit confusing to me, but that's just the default user if no multi-tenancy is used)

lucabello commented 2 weeks ago

We should add the possibility to configure max_global_streams_per_user (which defaults to 5000) via juju config.

marcusboden commented 2 weeks ago

Yeah, that was my thought as well. Or scale it up depending on how many apps/units are sending data.