search

gardener / machine-controller-manager

Declarative way of managing machines for Kubernetes cluster
Apache License 2.0
257 stars 117 forks source link

Reduce load on etcd/kube-apiserver on pod eviction #949

Closed thiyyakat closed 1 month ago

thiyyakat commented 1 month ago

What this PR does / why we need it:

The PR changes the way pods are listed before eviction, by using a PodInformer which uses a local cache rather than directly querying the kube-apiserver/etcd. The pods are listed only after the cache has synced.

Which issue(s) this PR fixes: Fixes #703

Special notes for your reviewer:

Impact of changes tested by running pods configuring a PDB with maxUnavailable set to 0, on a cluster with 20 machines, and deleting all the machines, thereby initiating drain. Without change, peak traffic recorded was 8.51 MB/s. With change, peak traffic recorded was 1.48 MB/s.

Screenshot 2024-10-21 at 1 10 05 PM

Integration tests run for providers AWS and Azure completed successfully.

Additionally, to manually check if cache for podInformer syncs successfully before the podLister.List() call,time.Sleep( 30 * time.Second) was introduced before calling RunCordonOrUncordon(), and a new pod (default/nginx-pod2 ) was deployed during the sleep period. Logs were added to print the names of all pods on the node returned by the podLister. After triggering the deletion of the machine, the machine entered the drain flow, and after the sleep the new pod's name was logged.

I1023 16:13:08.044770   91923 machine_util.go:1182] (drainNode) Invoking RunDrain, forceDeleteMachine: false, forceDeletePods: false, timeOutDuration: 5m0s
I1023 16:13:08.044894   91923 drain.go:238] ABOUT TO SLEEP for 30s
I1023 16:13:43.861554   91923 drain.go:369] Found pod default/nginx-pod2

Release note:

MCM will use an `informer` instead of the`clientset` to list pods in the drain logic. This will reduce the load on etcd/kube-apiserver.
gardener-robot-ci-3 commented 1 month ago

Thank you @thiyyakat for your contribution. Before I can start building your PR, a member of the organization must set the required label(s) {'reviewed/ok-to-test'}. Once started, you can check the build status in the PR checks section below.