kayketeixeira
commented
2 years ago Hello @pwlawe, we have the same problem, but we realized that with us this only happens when we perform a search without any filter. When we put a filter the load distribution is more assertive.
Have you already managed to solve this problem?
In our loki-distributed deployment, we have three querier pods. For the last three days at approximately 1:30am EDT, our deployments quickly saturated their working set memory and entered a non-responsive state. RSS memory remained low, and the pods were not OOM killed. The system recovered after manually killing these pods. Memory utilization by one such pod can be seen below, this pattern repeated itself on each of the 3 pods.