Multiple Node Pools

[mathieu-benoit]

• https://azure.microsoft.com/en-us/updates/support-for-multiple-node-pools-in-azure-kubernetes-service-is-now-available/
• https://docs.microsoft.com/en-us/azure/aks/operator-best-practices-advanced-scheduler#provide-dedicated-nodes-using-taints-and-tolerations

Learnings and Gotchas:

• First Node Pool at cluster creation is Linux
• We can't delete the first Node Pool
• HTTP application routing add-on can't be used
• Linux node pools the length must be between 1-12 characters, for Windows node pools the length must be between 1-6 characters
• Maximum of eight node pools
• Maximum of 400 nodes across those 8 node pools
• All node pools must reside in the same subnet (https://github.com/Azure/azure-cli/issues/10984)
• Must use virtual machine scale sets for the nodes
• When upgrading a cluster the new nodes will get the taints
• When you scale a node pool in AKS, taints and tolerations do not carry over by design (https://github.com/Azure/AKS/issues/1061)

Taints and tolerations are used to logically isolate resources with a hard cut-off - if the pod doesn't tolerate a node's taint, it isn't scheduled on the node. An alternate approach is to use node selectors. You label nodes, such as to indicate locally attached SSD storage or a large amount of memory, and then define in the pod specification a node selector. Kubernetes then schedules those pods on a matching node. Unlike tolerations, pods without a matching node selector can be scheduled on labeled nodes. This behavior allows unused resources on the nodes to consume, but gives priority to pods that define the matching node selector.

[mathieu-benoit]

• https://aka.ms/node-labels
• https://aka.ms/aks/nodepool/mode
• https://docs.microsoft.com/azure/aks/operator-best-practices-advanced-scheduler

[mathieu-benoit]

https://github.com/mathieu-benoit/myakscluster/commit/da5897cf1727ad4360193b9e51ee729a07bf4ec7