Closed brgrz closed 1 month ago
Hi, @brgrz - We have done resource optimizations and @pfrcks working on rolling out the changes.
Do the resource optimizations apply for the ama-logs' ama-logs-prometheus
container's CPU usage? We've noticed consistent CPU throttling on that.
This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.
This issue was closed because it has been stalled for 12 days with no activity.
We have also found this to be an issue. The memory "Limits" to the monitoring pod are set like so:
cpu: 5
memory: 8Gi
Requests:
cpu: 10m
memory: 50Mi
The monitor consistently runs over 1gb.
kubectl top pods -n kube-system --sort-by memory
NAME CPU(cores) MEMORY(bytes)
ama-metrics-operator-targets-cdf598c95-q8kgm 9m 1188Mi
ama-metrics-654dc7bdcc-mxjpl 24m 316Mi
ama-logs-xtx9t 28m 309Mi
ama-metrics-node-grd44 19m 288Mi
ama-logs-rktd7 24m 286Mi
ama-metrics-node-d9k5x 23m 285Mi
ama-metrics-node-4khgz 20m 275Mi
ama-logs-rqtlk 23m 273Mi
ama-logs-rs-86466dbfb8-t8cdz 20m 232Mi
Has anyone been notified of the AKS / Azure Monitor pod memory consumption issue which has been opened for months on the AKS repo?
Cross referencing this here: https://github.com/Azure/AKS/issues/3443
Any work being done on this? kube-system namespace pods taking up more than half of memory footprint in clusters where Azure Monitor/Container Insights are enabled.