search

grafana / alloy

OpenTelemetry Collector distribution with programmable pipelines
https://grafana.com/oss/alloy
Apache License 2.0
1.35k stars 190 forks source link

prometheus.remote_write: add an upper limit for the number of series kept in-memory #369

Open tpaschalis opened 1 year ago

tpaschalis commented 1 year ago

Currently, prometheus.remote_write will accept new series regardless of how many series are currently tracked in memory. Given enough new series in between WAL truncations, this can cause agents to run out of memory.

It would be nice to have a mechanism in prometheus.remote_write to limit the amount of active series to prevent agents from getting overwhelmed, at the cost of not being able to accept anything new until the next WAL truncation.

When clustering is enabled, this could potentially be paired with a forced truncate whenever the cluster state changes to free up memory sooner and allow new series to be stored.

github-actions[bot] commented 1 year ago

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it. If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue. The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity. Thank you for your contributions!