search

microsoft / Docker-Provider

Azure Monitor for Containers
Other
138 stars 105 forks source link

Azure Monitor pods memory consumption issue #1293

Closed brgrz closed 1 month ago

brgrz commented 1 month ago

Has anyone been notified of the AKS / Azure Monitor pod memory consumption issue which has been opened for months on the AKS repo?

Cross referencing this here: https://github.com/Azure/AKS/issues/3443

Any work being done on this? kube-system namespace pods taking up more than half of memory footprint in clusters where Azure Monitor/Container Insights are enabled.

ganga1980 commented 1 month ago

Hi, @brgrz - We have done resource optimizations and @pfrcks working on rolling out the changes.

andrewdinunzio commented 1 month ago

Do the resource optimizations apply for the ama-logs' ama-logs-prometheus container's CPU usage? We've noticed consistent CPU throttling on that.

github-actions[bot] commented 1 month ago

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.

github-actions[bot] commented 1 month ago

This issue was closed because it has been stalled for 12 days with no activity.

akakira commented 2 weeks ago

We have also found this to be an issue. The memory "Limits" to the monitoring pod are set like so: cpu: 5 memory: 8Gi Requests: cpu: 10m memory: 50Mi The monitor consistently runs over 1gb. kubectl top pods -n kube-system --sort-by memory NAME CPU(cores) MEMORY(bytes)
ama-metrics-operator-targets-cdf598c95-q8kgm 9m 1188Mi
ama-metrics-654dc7bdcc-mxjpl 24m 316Mi
ama-logs-xtx9t 28m 309Mi
ama-metrics-node-grd44 19m 288Mi
ama-logs-rktd7 24m 286Mi
ama-metrics-node-d9k5x 23m 285Mi
ama-metrics-node-4khgz 20m 275Mi
ama-logs-rqtlk 23m 273Mi
ama-logs-rs-86466dbfb8-t8cdz 20m 232Mi