search

kubernetes-sigs / kind

Kubernetes IN Docker - local clusters for testing Kubernetes
https://kind.sigs.k8s.io/
Apache License 2.0
13.01k stars 1.51k forks source link

Cluster creation hangs or fails on "joining worker nodes" if custom node role is set #3657

Closed carreter closed 2 weeks ago

carreter commented 2 weeks ago

What happened: Creation of a cluster hangs or fails on the "Joining worker nodes" step if node-role.kubernetes.io/worker: "" is added as a label. It seems kubelet is failing to start:

I0613 17:42:08.596458     154 kubelet.go:158] [kubelet-start] Checking for an existing Node in the cluster with name "bug-test-worker" and status "Ready"
I0613 17:42:08.599004     154 round_trippers.go:553] GET https://bug-test-control-plane:6443/api/v1/nodes/bug-test-worker?timeout=10s 404 Not Found in 2 milliseconds
I0613 17:42:08.599295     154 kubelet.go:173] [kubelet-start] Stopping the kubelet
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-check] Waiting for a healthy kubelet. This can take up to 4m0s
[kubelet-check] The kubelet is not healthy after 4m0.000408189s

Unfortunately, an error has occurred:
        The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' returned error: Get "http://localhost:10248/healthz": context deadline exceeded

What you expected to happen: Cluster should start and label nodes with appropriate role.

How to reproduce it (as minimally and precisely as possible): kind create with the following config:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  ipFamily: ipv4
nodes:
- role: control-plane
- role: worker
  labels:
    node-role.kubernetes.io/worker: ""

Anything else we need to know?: I believe it's something specifically about the node-role.kubernetes.io label that's breaking it. kind create with another label succeeds:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  ipFamily: ipv4
nodes:
- role: control-plane
- role: worker
  labels:
    test: test

Environment:

carreter commented 2 weeks ago

kind-bug.tar.gz

Log dump here!

BenTheElder commented 2 weeks ago

Yes, this isn't supported, you're using a namespace reserved for Kubernetes and Kubelet will not allow you to do this if it's not an approved value.

You should use a different namespace for your custom labels / values.

https://kubernetes.io/docs/reference/labels-annotations-taints/

Kubernetes reserves all labels and annotations in the kubernetes.io and k8s.io namespaces.

/remove-kind bug /kind support /close

k8s-ci-robot commented 2 weeks ago

@BenTheElder: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/kind/issues/3657#issuecomment-2166670848): >Yes, this isn't supported, you're using a namespace reserved for Kubernetes and Kubelet will not allow you to do this if it's not an approved value. > >You should use a different namespace for your custom labels / values. > >https://kubernetes.io/docs/reference/labels-annotations-taints/ > > >/remove-kind bug >/kind support >/close > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
BenTheElder commented 2 weeks ago

(also https://github.com/kubernetes-sigs/apiserver-network-proxy/pull/631#discussion_r1638814663, and if you check the kubelet log kubelet should have a message about this)

carreter commented 2 weeks ago

Got it, makes sense. Thanks for the fast response!

BenTheElder commented 1 week ago

this is also #3536, we may attempt to validate these earlier