search

kubernetes-sigs / kind

Kubernetes IN Docker - local clusters for testing Kubernetes
https://kind.sigs.k8s.io/
Apache License 2.0
13.42k stars 1.55k forks source link

kubeadm labelling failing using kubeletExtraArgs #3355

Open doublethink13 opened 1 year ago

doublethink13 commented 1 year ago

What happened:

Tried to create a cluster with additional labels on the nodes using nodeRegistration.kubeletExtraArgs.node-labels but it didn't work.

I ran kind create cluster --wait 5m --config config.yml (config below).

What you expected to happen:

kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"

Should add labels as expected.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Following https://kind.sigs.k8s.io/docs/user/ingress/, I tried to create a cluster with this config file:

---
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - image: kindest/node:v1.27.2@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba72
    role: control-plane
  - extraPortMappings:
      - containerPort: 80
        hostPort: 80
        protocol: TCP
      - containerPort: 443
        hostPort: 443
        protocol: TCP
    image: kindest/node:v1.27.2@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba72
    kubeadmConfigPatches:
      - |
        kind: InitConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "ingress-ready=true"
    labels:
      nginx-controller: true
    role: control-plane
  - image: kindest/node:v1.27.2@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba72
    role: worker
  - image: kindest/node:v1.27.2@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba72
    role: worker

The control-plane node does not have a label ingress-ready=true. However, it does have a label nginx-controller=true. If I create a cluster without the labels configuration, it stills doesn't add the expected label. It works, ie, k get node --selector=ingress-ready=true returns the expected node, if I use:

labels:
  nginx-controller: true
  ingress-ready: true

I've also tried the above methods in a worker node but k get node --selector=ingress-ready=true still returns no node.

Logs: logs.tar.gz

Environment:

Server: Containers: 29 Running: 7 Paused: 0 Stopped: 22 Images: 313 Server Version: 24.0.5 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Using metacopy: false Native Overlay Diff: true userxattr: false Logging Driver: json-file Cgroup Driver: systemd Cgroup Version: 2 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 runc Default Runtime: runc Init Binary: docker-init containerd version: 8165feabfdfe38c65b599c4993d227328c231fca runc version: v1.1.8-0-g82f18fe init version: de40ad0 Security Options: apparmor seccomp Profile: builtin cgroupns Kernel Version: 6.2.0-31-generic Operating System: Ubuntu 22.04.3 LTS OSType: linux Architecture: x86_64 CPUs: 12 Total Memory: 15.3GiB Name: eduardo-ThinkPad-E15-Gen-4 ID: fce466fa-00bb-48e1-8ebf-0ee2b71af203 Docker Root Dir: /var/lib/docker Debug Mode: false Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false 

- OS (e.g. from `/etc/os-release`): 
```txt
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
doublethink13 commented 1 year ago

I could have a look into this :slightly_smiling_face: (if really a bug and not a misstep on my part)

wongcheechun commented 1 year ago

Your config.yml is not correctly configured, the - on extraPortMappings is not needed. Here's what works for me 

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - image: kindest/node:v1.27.2@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba72
    role: control-plane
    kubeadmConfigPatches:
    - |
      kind: InitConfiguration
      nodeRegistration:
        kubeletExtraArgs:
          node-labels: "ingress-ready=true"
    labels:
      nginx-controller: true
    extraPortMappings:
      - containerPort: 80
        hostPort: 80
        protocol: TCP
      - containerPort: 443
        hostPort: 443
        protocol: TCP
  - image: kindest/node:v1.27.2@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba72
    role: worker
  - image: kindest/node:v1.27.2@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba72
    role: worker

Control plane should show ingress-ready label

$ kubectl get node test-control-plane -o json | jq '.metadata.labels'                                                                                                                                                                                            ⎈ kind-test
{
  "beta.kubernetes.io/arch": "amd64",
  "beta.kubernetes.io/os": "linux",
  "ingress-ready": "true",
  "kubernetes.io/arch": "amd64",
  "kubernetes.io/hostname": "test-control-plane",
  "kubernetes.io/os": "linux",
  "node-role.kubernetes.io/control-plane": "",
  "node.kubernetes.io/exclude-from-external-load-balancers": ""
}

edit: Forgot to mention also that your config declared 2 control planes. Not sure if this is intended.

doublethink13 commented 1 year ago

Your config.yml is not correctly configured, the - on extraPortMappings is not needed. Here's what works for me

What - do you mean?

Your config looks exactly the same as mine, you just ordered them differently.

Although your config seems to work.

Because of that, I tried switching the order of the control plane node.

Using:

---
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
  - extraPortMappings:
      - containerPort: 80
        hostPort: 80
        protocol: TCP
      - containerPort: 443
        hostPort: 443
        protocol: TCP
    image: kindest/node:v1.27.2@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba72
    kubeadmConfigPatches:
      - |
        kind: InitConfiguration
        nodeRegistration:
          kubeletExtraArgs:
            node-labels: "ingress-ready=true"
    labels:
      nginx-controller: true
    role: control-plane
  - image: kindest/node:v1.27.2@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba72
    role: control-plane
  - image: kindest/node:v1.27.2@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba72
    role: worker
  - image: kindest/node:v1.27.2@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba72
    role: worker

Works. So apparently the control-plane order is important? Is there a good reason for it?

stmcginnis commented 1 year ago

The original comment is correct of having an extra -, so it is not just the order off the information in the yaml. You have a syntactically different structure in your first example that makes it invalid.

doublethink13 commented 1 year ago

Where exactly is the extra dash?

stmcginnis commented 1 year ago

~In your original example:~

- extraPortMappings:

Edit: Sorry, ignore that. The display formatting was throwing me off and now I see you had two control-plane nodes defined. I can't see a difference with the actual node definition content now that I look close.

doublethink13 commented 1 year ago

I'm defining four nodes, two control planes and two workers. I don't see the yaml syntax error.

It's an array with four elements.

stmcginnis commented 1 year ago

I'm not sure, but with having two control plane nodes, you may need to include the node label configuration for both nodes. Changing the order may make a difference in that now the first node is getting that configuration, so it could be the previous config caused the first control plane node to spin up without it, and therefore fail to apply any of the labels.

BenTheElder commented 1 year ago

Aside: two control plane nodes isn't recommended. It isn't realistic "HA" behavior, and it's extra overhead locally.

More generally multiple nodes isn't recommended unless you have specific relevant use case like debugging Kubernetes's rollout behaviors, since they're sharing all the hosts's resources leakily and adding more overhead.

That said, the reason this doesn't work is you're trying to configure InitConfiguration on node # 2, but kubeadm uses join for all but the first node.

BenTheElder commented 1 year ago

If you use JoinConfiguration or use the first control plane node this will work as expected.

You can also configure both Join and Init if you need to write the patch once for any sort of node and refactor in the future, kubeadm will ignore the irrelevant one.

BenTheElder commented 1 year ago

https://kubernetes.io/docs/reference/config-api/kubeadm-config.v1beta3/#kubeadm-k8s-io-v1beta3-InitConfiguration

InitConfiguration contains a list of elements that is specific "kubeadm init"-only runtime information. kubeadm init-only information. These fields are solely used the first time kubeadm init runs. After that, the information in the fields IS NOT uploaded to the kubeadm-config ConfigMap that is used by kubeadm upgrade for instance. These fields must be omitempty.

The docs should maybe have an additional note that init is only used for one node, but there are other docs about using kubeadm and the kubeadm config docs are somewhat reasonably expecting those are followed. kubeadm join and JoinConfiguration are used for all non-initial nodes. Init is special to the bootstrap node.