Closed nemobis closed 3 months ago
Some of the overall settings here are supposed to be configurable in kubernetes, see e.g. https://kubernetes.io/docs/tasks/administer-cluster/reserve-compute-resources/#enforcing-node-allocatable , but they don't seem to be on AKS according to what I heard so far from Azure Support.
@nemobis you can check the configuration of kubelet by yourself by running a debug pod on the node and look at the process snapshot:
➜ ~ kubectl debug node/aks-systempool-21850828-vmss000000 -it --image=mcr.microsoft.com/dotnet/runtime-deps:6.0
Creating debugging pod node-debugger-aks-systempool-21850828-vmss000000-chs4s with container debugger on node aks-systempool-21850828-vmss000000.
If you don't see a command prompt, try pressing enter.
root@aks-systempool-21850828-vmss000000:/# chroot /host
# bash
root@aks-systempool-21850828-vmss000000:/# ps fauxww | grep '/usr/local/bin/kubelet'
I ran into the same "issue"; using a VM with only 4GiB of memory (Standard F2S v2) returns the following:
➜ ~ k describe node aks-systempool-21850828-vmss000000
(...)
Capacity:
cpu: 2
ephemeral-storage: 129886128Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 4025836Ki
pods: 110
Allocatable:
cpu: 1900m
ephemeral-storage: 119703055367
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 2209260Ki
pods: 110
According to the documentation, kubelet will reserve 25% of memory (i.e. 1GiB).
Indeed, using the method described above, you can see kubelet runs with the following flags:
--kube-reserved=cpu=100m,memory=1024Mi,pid=1000
--eviction-hard=memory.available<750Mi,nodefs.available<10%,nodefs.inodesFree<5%,pid.available<2000
So in total 1816576kiB of memory is reserved; and thus: 4025836-1816576=2209260KiB i.e the amount reported by AKS.
Action required from @Azure/aks-pm
Issue needing attention of @Azure/aks-leads
Issue needing attention of @Azure/aks-leads
Issue needing attention of @Azure/aks-leads
Issue needing attention of @Azure/aks-leads
Issue needing attention of @Azure/aks-leads
Issue needing attention of @Azure/aks-leads
Issue needing attention of @Azure/aks-leads
Issue needing attention of @Azure/aks-leads
Issue needing attention of @Azure/aks-leads
Issue needing attention of @Azure/aks-leads
Issue needing attention of @Azure/aks-leads
Issue needing attention of @Azure/aks-leads
Hello, beginning with AKS 1.29 preview and beyond, we shipped changes to the eviction threshold and memory reservation for kube-reserved. The new rate of memory reservations is set according to the lesser value of: 20MB * Max Pods supported on the Node + 50MB or 25% of the total system memory resources. The new eviction threshold is 100Mi. See more information here. These changes will help reduce the resource consumption by AKS and can deliver up to 20% more allocatable space depending on your pod configuration. Thanks!
Describe the bug On AKS with kubernetes 1.24, a node with 4 GB RAM capacity only has 2157 MiB allocatable; yet kube-system alone reserves some 750 MB (of which 550 MB for azure-cns and azure-npm), leaving less than 1400 MiB available for
requests
by others.To Reproduce Steps to reproduce the behavior:
kube-capacity
orkubectl describe node
on a recently created nodeExample node:
Expected behavior A node with 4 GB of RAM should be able to be assigned a pod which requests 1600 MB of RAM (e.g. for Prometheus). (I'm not talking of limits.)
Screenshots![Screenshot_20230309_120410](https://user-images.githubusercontent.com/901528/224020005-d3432655-5015-419d-b9d3-a2da8b6189db.png)
Environment (please complete the following information):
Additional context
There's been a lot of discussion about what the requests and limits should be for various components, but in this case the issue is only with the value of the allocatable memory, so I believe it's orthogonal. If everything in kube-system is requesting way more memory than it needs most of the time, there's no need for such a huge buffer. At very least it should be configurable, or the really available memory should be made clearer so that people can configure their loads and nodepools accordingly, without tinkering with eviction thresholds.
https://github.com/Azure/AKS/issues/1339 https://github.com/Azure/AKS/issues/2125 https://github.com/Azure/AKS/issues/3348 https://github.com/Azure/AKS/issues/3496
I think it's unrelated from https://github.com/Azure/AKS/issues/3443