Use Azure `mode` to separate user & system node pools

[julie-ng]

Problem

Currently the default_node_pool block is used for the system node pool. But this defaults to user mode.

Consequence

AKS has a mode property that can be either system or user which beyond semantics adds the CriticalAddonsOnly=true:NoSchedule taint.

Docs Referender: Manage system node pools in Azure Kubernetes Service (AKS) > System and user node pools

Changes required

• Swap default_node_pool block and the kubernetes_cluster_node_pool resource definitions.
• Will probably destroy and re-create the cluster.

[julie-ng]

Azure was smart enough to recognize it as a system node based on naming 😅 because I see kubernetes.azure.com/mode=system

$ k describe no aks-system-19810993-vmss000000
Name:               aks-system-19810993-vmss000000
Roles:              agent
Labels:             agentpool=system
                    beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=Standard_B2ms
                    beta.kubernetes.io/os=linux
                    failure-domain.beta.kubernetes.io/region=norwayeast
                    failure-domain.beta.kubernetes.io/zone=0
                    kubernetes.azure.com/agentpool=system
                    kubernetes.azure.com/cluster=cloudkube-dev-i7iw-managed-rg
                    kubernetes.azure.com/mode=system
                    kubernetes.azure.com/node-image-version=AKSUbuntu-1804gen2containerd-2022.03.20
                    kubernetes.azure.com/os-sku=Ubuntu
                    kubernetes.azure.com/role=agent
                    kubernetes.azure.com/storageprofile=managed
                    kubernetes.azure.com/storagetier=Premium_LRS
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=aks-system-19810993-vmss000000
                    kubernetes.io/os=linux
                    kubernetes.io/role=agent
                    node-role.kubernetes.io/agent=
                    node.kubernetes.io/instance-type=Standard_B2ms
                    storageprofile=managed
                    storagetier=Premium_LRS
                    topology.disk.csi.azure.com/zone=
                    topology.kubernetes.io/region=norwayeast
                    topology.kubernetes.io/zone=0
                    workloadType=system
Annotations:        csi.volume.kubernetes.io/nodeid:
                      {"disk.csi.azure.com":"aks-system-19810993-vmss000000","file.csi.azure.com":"aks-system-19810993-vmss000000","secrets-store.csi.k8s.io":"a...
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 31 Mar 2022 22:24:37 +0200

Nevermind - no taints 😓

CreationTimestamp:  Thu, 31 Mar 2022 22:24:37 +0200
Taints:             <none>
Unschedulable:      false

[julie-ng]

TIL 😒

Adding taints, labels, or tags to nodes should be done for the entire node pool using az aks nodepool. Applying taints, labels, or tags to individual nodes in a node pool using kubectl is not recommended.

Docs: Specify a taint, label, or tag for a node pool