Maximum active stream limit exceeded

[abhishekgaurav-greytip]

Describe the bug error sending batch, will retry" status=429 error="server returned HTTP status 429 Too Many Requests (429): Maximum active stream limit exceeded, reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased, user: 'fake'"

To Reproduce Steps to reproduce the behavior:

1. Started Loki (SHA or version) Loki version 2.9.2
2. Started Promtail (SHA or version) to tail '...' promtail version 2.6.1
3. Query: {} term

Expected behavior log should be visible in loki but loki dashboards are not showing any logs

Environment:

• Infrastructure: kubernetes
• Deployment tool: helm

Screenshots, Promtail config, or terminal output level=warn ts=2024-01-31T10:07:50.118681851Z caller=client.go:369 component=client host=loki-loki-distributed-gateway.logging msg="error sending batch, will retry" status=429 error="server returned HTTP status 429 Too Many Requests (429): Maximum active stream limit exceeded, reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased, user: 'fake'"

[cyriltovena]

:wave:

You're sending more labels than the default allowed. Take a look at https://grafana.com/docs/loki/latest/send-data/promtail/stages/limit/ and increase those limits.

[abhishekgaurav-greytip]

Default is 10000 and i am sure its more than what i am using.

Changing gcs bucket solve this issue but i lost all old logs.

It has to do something with bucket and log stored. I have added new label in promtail and after applying those changes i started getting this error

[abhishekgaurav-greytip]

msg="error querying storage" err="context canceled"

[JustNZ]

Hey, I have the same issue here and my logs are full of this server returned HTTP status 429 Too Many Requests (429): Maximum active stream limit exceeded, reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased, user: 'swiftlog_itops'

and I also tried every limit change to fix that but also with those limits I still get the issue:

max_cache_freshness_per_query: 10m
split_queries_by_interval: 15m
per_stream_rate_limit: 512M
per_stream_rate_limit_burst: 1024M
cardinality_limit: 200000
ingestion_rate_mb: 10000
ingestion_burst_size_mb: 1000
max_streams_per_user: 100000
max_entries_limit_per_query: 1000000
max_label_value_length: 20480
max_label_name_length: 10240
max_label_names_per_series: 300

[fanchibaoLiu]

Has the problem been resolved? I encountered the same problem.

I think I have a label called traceid, which has too many values.

What parameters should be adjusted to open restrictions？

[fanchibaoLiu]

max_global_streams_per_user: 0 I set this parameter and the error disappeared.

[Marioloez]

max_global_streams_per_user: 0

"Thank you very much, this works very well."