search

MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure
https://docs.microsoft.com/azure
Creative Commons Attribution 4.0 International
10.07k stars 21.12k forks source link

AKS node OS reservered resources #116624

Open jindrichpilar-kosik opened 8 months ago

jindrichpilar-kosik commented 8 months ago

Hi,

we are working on right-sizing our AKS nodepools. Documentation explains resources reserved for Kubernetes itself, but only mentions that Operating System also reserves CPU/RAM.

In addition to reservations for Kubernetes itself, the underlying node OS also reserves an amount of CPU and memory resources to maintain OS functions.

It is desirable for documentation to explain OS reserved resources as well. Or at least provide link to documentation explaining OS reserved resources.

How much CPU/RAM is reserved for the OS?

Document Details

⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.

RamanathanChinnappan-MSFT commented 8 months ago

@jindrichpilar-kosik Thanks for your feedback! We will investigate and update as appropriate.

SaibabaBalapur-MSFT commented 8 months ago

@jindrichpilar-kosik Thank you for reaching out. Unfortunately, the document you provided does not mention the exact amount of CPU and memory resources reserved for the operating system. However, I can suggest some ways to estimate the amount of resources reserved by the operating system.

The amount of CPU and memory resources reserved by the operating system depends on the operating system itself and the version of Kubernetes you are running. As a general rule of thumb, you can assume that the operating system will reserve at least 10% of the CPU and memory resources on each node.

To get a more accurate estimate, you can use the kubectl top command to monitor the CPU and memory usage of your nodes. This will give you an idea of how much CPU and memory is being used by the operating system and other system processes.

You can also refer to the documentation for the specific operating system you are using to get more information on how much CPU and memory is reserved for the operating system.

I hope this helps! Let me know if you have any other questions.

jindrichpilar-kosik commented 7 months ago

Thanks @SaibabaBalapur-MSFT for swift response.

Sure I can find out using kubectl but it is highly impractical to iterate over all VM series and SKU sizes to find out what is the allocatable CPU/RAM. Could you point me to any documentation for Ubuntu based AKS/K8s nodes?

The amount of CPU and memory resources reserved by the operating system depends on the operating system itself and the version of Kubernetes you are running. As a general rule of thumb, you can assume that the operating system will reserve at least 10% of the CPU and memory resources on each node.

It appears that AKS 1.27.3 with Ubuntu based node Standard_D16d_v5 has allocatable 56.579998 GiB of RAM. With 6.23 GiB reserved for Kubernetes it leaves 1.19 GiB for OS. Far smaller amount than 10 %.

SaibabaBalapur-MSFT commented 7 months ago

@jindrichpilar-kosik Thank you for providing the details. The amount of CPU and memory resources reserved by the operating system depends on the operating system itself and the version of Kubernetes you are running. As a general rule of thumb, you can assume that the operating system will reserve at least 10% of the CPU and memory resources on each node.

Regarding your query about the documentation for Ubuntu-based AKS/K8s nodes, you can refer to the following link: https://docs.microsoft.com/en-us/azure/aks/concepts-clusters-workloads#resource-reservations. This link provides information about the resource reservations for AKS clusters, including the amount of CPU and memory resources reserved by the operating system.

As per the documentation, the amount of memory reserved for the operating system is dependent on the size of the node. For example, for a Standard_D16d_v5 node, the operating system reserves 6.23 GiB of memory, leaving 56.58 GiB of memory allocatable for applications. This is far less than the 10% rule of thumb, but it is important to note that this is a general guideline and the actual amount of memory reserved by the operating system can vary depending on the specific configuration of the node.

I hope this information helps. Let me know if you have any further questions.

jindrichpilar-kosik commented 7 months ago

@SaibabaBalapur-MSFT that is exactly the page this issue refers to. Under the example there is this sentence

In addition to reservations for Kubernetes itself, the underlying node OS also reserves an amount of CPU and memory resources to maintain OS functions.

Which I understand as "For Standard_D16d_v5 node there are extra reserved resources in addition to the 6.23 GiB.". And my question in this issue is - how do I determine these extra reservations without spinning up nodes.

SaibabaBalapur-MSFT commented 7 months ago

@jindrichpilar-kosik Thanks for bringing this to our attention. I'm going to assign this to the document author so they can take a look at it accordingly.

@MGoedtel Can you please check and add your comments on this doc update request as applicable.

rayoef commented 2 months ago

reassign:tamram