Adding node taints

[wonderhoss]

I am trying to create a cluster with a tainted node to test tolerations. From the documentation it should be possible to register a node with taints by using a kubeadm InitConfiguration patch, but this doesn't seem to work.

My cluster config looks like this:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: foo

nodes:
- role: control-plane
- role: worker
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      taints:
        - key: "class"
          value: "limited"
          effect: "NoSchedule"
      kubeletExtraArgs:
        node-labels: "class=limited"

This will successfully create a cluster, but all information in the kubeadm patch is not reflected once the node is created. Debug (-v 5) log shows the patch being used:

[...]

DEBUG: config/config.go:96] Using the following kubeadm config for node foo-worker:
apiServer:
  certSANs:
  - localhost
  - 127.0.0.1
  extraArgs:
    feature-gates: KubeletInUserNamespace=true
    runtime-config: ""
apiVersion: kubeadm.k8s.io/v1beta3
clusterName: foo
controlPlaneEndpoint: foo-control-plane:6443
controllerManager:
  extraArgs:
    enable-hostpath-provisioner: "true"
    feature-gates: KubeletInUserNamespace=true
kind: ClusterConfiguration
kubernetesVersion: v1.31.0
networking:
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/16
scheduler:
  extraArgs:
    feature-gates: KubeletInUserNamespace=true
---
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- token: abcdef.0123456789abcdef
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 10.89.0.23
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock
  kubeletExtraArgs:
    node-ip: 10.89.0.23
    node-labels: class=limited
    provider-id: kind://podman/foo/foo-worker
  taints:
  - effect: NoSchedule
    key: class
    value: limited
skipPhases:
- preflight
---
apiVersion: kubeadm.k8s.io/v1beta3
discovery:
  bootstrapToken:
    apiServerEndpoint: foo-control-plane:6443
    token: abcdef.0123456789abcdef
    unsafeSkipCAVerification: true
kind: JoinConfiguration
nodeRegistration:
  criSocket: unix:///run/containerd/containerd.sock
  kubeletExtraArgs:
    node-ip: 10.89.0.23
    node-labels: ""
    provider-id: kind://podman/foo/foo-worker
skipPhases:
- preflight
---
apiVersion: kubelet.config.k8s.io/v1beta1
cgroupDriver: systemd
cgroupRoot: /kubelet
evictionHard:
  imagefs.available: 0%
  nodefs.available: 0%
  nodefs.inodesFree: 0%
failSwapOn: false
featureGates:
  KubeletInUserNamespace: true
imageGCHighThresholdPercent: 100
kind: KubeletConfiguration
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
conntrack:
  maxPerCore: 0
  tcpCloseWaitTimeout: 0s
  tcpEstablishedTimeout: 0s
featureGates:
  KubeletInUserNamespace: true
iptables:
  minSyncPeriod: 1s
kind: KubeProxyConfiguration
mode: iptables

[...]

But the node so created looks like this, with no taint and also missing the extra label:

apiVersion: v1
kind: Node
metadata:
  annotations:
    kubeadm.alpha.kubernetes.io/cri-socket: unix:///run/containerd/containerd.sock
    node.alpha.kubernetes.io/ttl: "0"
    volumes.kubernetes.io/controller-managed-attach-detach: "true"
  creationTimestamp: "2024-11-05T16:13:25Z"
  labels:
    beta.kubernetes.io/arch: arm64
    beta.kubernetes.io/os: linux
    kubernetes.io/arch: arm64
    kubernetes.io/hostname: foo-worker
    kubernetes.io/os: linux
  name: foo-worker
  resourceVersion: "489"
  uid: 641567d7-29cf-4a23-809d-cae8aa1327fd
spec:
  podCIDR: 10.244.1.0/24
  podCIDRs:
  - 10.244.1.0/24
  providerID: kind://podman/foo/foo-worker
status:
  addresses:
  - address: 10.89.0.23
    type: InternalIP
  - address: foo-worker
    type: Hostname
  allocatable:
    cpu: "5"
    ephemeral-storage: 104266732Ki
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    hugepages-32Mi: "0"
    hugepages-64Ki: "0"
    memory: 1995028Ki
    pods: "110"
  capacity:
    cpu: "5"
    ephemeral-storage: 104266732Ki
    hugepages-1Gi: "0"
    hugepages-2Mi: "0"
    hugepages-32Mi: "0"
    hugepages-64Ki: "0"
    memory: 1995028Ki
    pods: "110"
  conditions:
  - lastHeartbeatTime: "2024-11-05T16:13:37Z"
    lastTransitionTime: "2024-11-05T16:13:25Z"
    message: kubelet has sufficient memory available
    reason: KubeletHasSufficientMemory
    status: "False"
    type: MemoryPressure
  - lastHeartbeatTime: "2024-11-05T16:13:37Z"
    lastTransitionTime: "2024-11-05T16:13:25Z"
    message: kubelet has no disk pressure
    reason: KubeletHasNoDiskPressure
    status: "False"
    type: DiskPressure
  - lastHeartbeatTime: "2024-11-05T16:13:37Z"
    lastTransitionTime: "2024-11-05T16:13:25Z"
    message: kubelet has sufficient PID available
    reason: KubeletHasSufficientPID
    status: "False"
    type: PIDPressure
  - lastHeartbeatTime: "2024-11-05T16:13:37Z"
    lastTransitionTime: "2024-11-05T16:13:37Z"
    message: kubelet is posting ready status
    reason: KubeletReady
    status: "True"
    type: Ready
  daemonEndpoints:
    kubeletEndpoint:
      Port: 10250
  images:
  - names:
[...]
  nodeInfo:
    architecture: arm64
    bootID: 5aaf2513-cfa1-4611-a785-664fd60c6c02
    containerRuntimeVersion: containerd://1.7.18
    kernelVersion: 6.11.3-200.fc40.aarch64
    kubeProxyVersion: ""
    kubeletVersion: v1.31.0
    machineID: 7c44d1c32c2f4241a8b838ee29a6f201
    operatingSystem: linux
    osImage: Debian GNU/Linux 12 (bookworm)
    systemUUID: 2288f19c-4537-4ade-9605-866cba5555e3

What am I missing?

[BenTheElder]

InitConfiguration is only used for the first node in a kubeadm cluster, JoinConfiguration is used for all other nodes.

[wonderhoss]

🤦‍♀️

Yeah, that would make sense, wouldn't it? Thank you.