Add kubelet instance configuration to configure CRI socket for each node

[HirazawaUi]

Is this a BUG REPORT or FEATURE REQUEST?

FEATURE REQUEST

Anything else we need to know?

This issue wants to remove container runtime interface (CRI) socket annotations from Node objects in Kubernetes and no longer added --container-runtime-endpoint args in kubeadm-flags.env ( --container-runtime-endpoint args deprecated in kubelet), this annotation and args are used to specify the CRI socket endpoint on each node that the kubelet uses to communicate with the container runtime.

Add kubelet-instance-config.yml locally to store container runtime interface (CRI) sockets, and in kubeadm init, join and upgrade, override the global kubelet configuration through kubelet-instance-config.yml.

More detailed design at: https://hackmd.io/@r2Yq-PVoRcK2OaGypnwPMg/rJFChACCp/edit

### Tasks

[HirazawaUi]

/cc @neolit123 @pacoxu

[neolit123]

@HirazawaUi we need a KEP for this, ket's see what @pacoxu can comment about the previous work done on this.

@pacoxu can you please post links to your previous KEP work.? also didn't we already have a tracking issue here in k/kubeadm?

maybe we can close KEP PRs and let @HirazawaUi to take over.

[neolit123]

also are you willing to work on this @HirazawaUi there are a lot of things to update around this change across multiple releases. and i don't think me or anyone else will have a lot of time for it.

• KEP design how is it going to work exactly, do we need a feature gate
• k8s.io docs update
• k/k code update
• do we need a e2e test in k/kubeadm

too many questions

[HirazawaUi]

also are you willing to work on this @HirazawaUi there are a lot of things to update around this change across multiple releases. and i don't think me or anyone else will have a lot of time for it.

• KEP design how is it going to work exactly, do we need a feature gate
• k8s.io docs update
• k/k code update
• do we need a e2e test in k/kubeadm

too many questions

Yes, I'll try to finish it.

[HirazawaUi]

@pacoxu What do you think about this? I would love to hear your comments :)

[pacoxu]

The KEP

• https://github.com/kubernetes/enhancements/issues/3929
• https://github.com/kubernetes/enhancements/pull/3930

My current proposal would be something like https://github.com/kubernetes/enhancements/pull/3930#issuecomment-1931657825

i meant a generic patches approach where a local config file overrides the global config downloaded from the cm. not the --patches feature.

this local file must be stored somewhere. perhaps in the same dir as config.yaml, but called config-instance.yaml. IMO it has a number of tricky aspects that need to be covered in the design doc for init, join, upgrade.

Above is @neolit123 your proposal in comment here https://github.com/kubernetes/enhancements/pull/3930/files#r1177682829.

I prefer this solution.

BTW, /var/lib/kubelet/kubeadm-flags.env will only have --container-runtime-endpoint later. I would like to make this cri-socket-annotation and runtime endpoint configuration to config-instance.yaml under /var/lib/kubelet.

[HirazawaUi]

Thanks, I have missed this comment, I will resurrect kep as soon as possible.

[neolit123]

note if this ever becomes part of kubeadm, we can close this ticket: https://github.com/kubernetes/kubeadm/issues/1924