search

kubernetes-sigs / kubespray

Deploy a Production Ready Kubernetes Cluster
Apache License 2.0
15.71k stars 6.38k forks source link

Kube-Proxy is enabled when upgrading #11227

Open ehsan310 opened 2 months ago

ehsan310 commented 2 months ago

What happened?

I have recently upgraded my cluster to 2.25.0 and noticed kube-proxy is pushed and enabled again which caused my cluster goes into a problem. I have calico eBPF enabled and when cluster is deployed checks are correctly done so no kube-proxy is pushed but when doing upgrade kube-proxy DS is pushed and then mess up the cluster with iptables rule and ipvs interface!

What did you expect to happen?

I was expecting the cluster upgrade to ignore installing/upgrading kube-proxy when ebpf is enabled.

How can we reproduce it (as minimally and precisely as possible)?

Install the cluster with calico ebpf enabled. Remove Kube-proxy DS Remove IPVS (if enabled) run upgrade-cluster.yaml

OS

debian 12

Version of Ansible

I am using kubespray docker image (2.25.0)

Version of Python

I am using kubespray docker image (2.25.0)

Version of Kubespray (commit)

2.25.0

Network plugin used

calico

Full inventory with variables

.

Command used to invoke ansible

ansible-playbook -i inventory/ansible.hosts -u ehsan --become --become-user=root -k -K upgrade_cluster.yml

Output of ansible run

It works without issue, just check is missing for ignoring kube-proxy.

Anything else we need to know

No response

VannTen commented 2 months ago

Install the cluster with calico ebpf enabled. Remove Kube-proxy DS

kube-proxy should not be deployed in the first place when calico_bpf_enabled is true, see https://github.com/kubernetes-sigs/kubespray/blob/4b9349a052ba181b110dd4d751c8ccc153e22207/roles/kubespray-defaults/defaults/main/main.yml#L41 (kubeadm handle the kube-proxy part, and we skip that kubeadm phases in some cases, including calico ebpf)

Can you check the value of calico_bpf_enabled on your nodes ? And ideally of kubeadm_init_phases_skip as well.

ehsan310 commented 2 months ago

I get that and that but I haven't enabled ebpf in the beginning I enabled when the cluster were up and running and removed kube-proxy , so during the inital phase kube-proxy was in the cluster. I was hoping that when ebpf is enable and i removed kube-proxy DS , kube-proxy is not getting pushed again.

is there anywhere in node that keep kubeadm configs ?

VannTen commented 2 months ago

Hum, not sure. We should check if it's a template for kubeadm config, but presumably we should renew those on upgrade. Maybe kubeadm create a config map in the cluster ? That needs some investigation

ehsan310 commented 2 months ago

sounds like there is a kubeadm-config configmap in kube-system namespace.

https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-reconfigure/#applying-kube-proxy-configuration-changes

poblahblahblah commented 1 month ago

We ran into this problem as well, albeit with a different CNI, and it was because we deleted the kube-system/kube-proxy DaemonSet but we did not delete the kube-system/kube-proxy ConfigMap.

Once we removed both of those the issue went away on upgrade.

ehsan310 commented 1 month ago

We ran into this problem as well, albeit with a different CNI, and it was because we deleted the kube-system/kube-proxy DaemonSet but we did not delete the kube-system/kube-proxy ConfigMap.

Once we removed both of those the issue went away on upgrade.

I'll try to test this, hopefully this will fix the issue, then might be good to document it.