search

eksctl-io / eksctl

The official CLI for Amazon EKS
https://eksctl.io
Other
4.92k stars 1.41k forks source link

drain nodegroup on deletion #370

Closed errordeveloper closed 5 years ago

errordeveloper commented 5 years ago

After #369 lands we should add drain capability on deletion, to cater for upgrades nicely (#100, #357, #348). It seems plausible to add --drain and keep on by default.

whereisaaron commented 5 years ago

@errordeveloper kube-aws installs a cool 'auto-drainer' that runs on each node in the cluster. It detects both ASG terminations and Spot Instance terminations. When detected, the node automatically drains itself before signalling it is ready to be terminated. This speeds up upgrades and in-scaling of ASGs, because nodes terminate more cleanly and faster by signalling they are ready to stop, rather than waiting the full ASG termination notice timeout.

The cool thing about this self-drain approach is that it doesn't matter how the termination happens, whether ekcctl or the user just updating the ASG size directly, or instance upgrades, or instance type changes, or auto-scaling, the node still self-drains cleanly.

This system is a based on couple of bash scripts my cool friend @mumoshu developed:

https://github.com/kubernetes-incubator/kube-aws/blob/bdeadab166e728dbcc130ac3124affc66cc4d682/builtin/files/userdata/cloud-config-controller#L2333-L2543

For ASG changes, one script in a Deployment polls the ASG API for pending terminations and then signals any to-be-terminated node via a ConfigMap. Then a script runs on each node in a DaemonSet, polls the ConfigMap for ASG terminations, and polls the instance metadata URL to detect Spot instance terminations. Spot instances don't technically get notice, but @mumoshu found this backdoor way to detect termination by watching the instance metadata.

errordeveloper commented 5 years ago

This sounds like a great add-on! I think to begin with, we should add a simple '--drain' flag in 'eksctl delete nodegroup' command, that will help for automated upgrades.

On Fri, 4 Jan 2019, 3:25 am Aaron Roydhouse, notifications@github.com wrote:

@errordeveloper https://github.com/errordeveloper kube-aws installs a cool 'auto-drainer' that runs on each node in the cluster. It detects both ASG terminations and Spot Instance terminations. When detected, the node automatically drains itself before signalling it is ready to be terminated. This speeds up upgrades and in-scaling of ASGs, because nodes terminate more cleanly and faster by signalling they are ready to stop, rather than waiting the full ASG termination notice timeout.

The cool thing about this self-drain approach is that it doesn't matter how the termination happens, whether ekcctl or the user just updating the ASG size directly, or instance upgrades, or instance type changes, or auto-scaling, the node still self-drains cleanly.

This system is a based on couple of bash scripts my cool friend @mumoshu https://github.com/mumoshu developed:

https://github.com/kubernetes-incubator/kube-aws/blob/bdeadab166e728dbcc130ac3124affc66cc4d682/builtin/files/userdata/cloud-config-controller#L2333-L2543

For ASG changes, one script in a Deployment polls the ASG API for pending terminations and then signals any to-be-terminated node via a ConfigMap. Then a script runs on each node in a DaemonSet, polls the ConfigMap for ASG terminations, and polls the instance metadata URL to detect Spot instance terminations. Spot instances don't technically get notice, but @mumoshu https://github.com/mumoshu found this backdoor way to detect termination by watching the instance metadata.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/weaveworks/eksctl/issues/370#issuecomment-451344950, or mute the thread https://github.com/notifications/unsubscribe-auth/AAPWSytB-y3pFXloNP0pmKZ66B5VovRLks5u_snHgaJpZM4ZfbUs .

mumoshu commented 5 years ago

@errordeveloper --drain sounds great if what it does is basically calling kubectl drain for each node in the nodegroup being deleted!

Adding a node-drainer addon just makes the flag optional, and may also adds the possibility to handle instance termination external to eksctl, like spot instance termination. Am I correct?

errordeveloper commented 5 years ago

Looks like we could try using github.com/kubernetes/autoscaler/cluster-autoscaler/utils/drain package, or otherwise we can refactor kubectl code into a library that we can then call.

errordeveloper commented 5 years ago

I've started working on https://github.com/kubernetes/kubernetes/pull/72827.

errordeveloper commented 5 years ago

To facilitate for upgrades, I will try and get this implemented before I finish kubernetes/kubernetes#72827.

whereisaaron commented 5 years ago

Discussion and builds of automatic node draining for ASG in-scaling and Spot Fleet shutdowns, similar to kube-aws https://github.com/awslabs/amazon-eks-ami/issues/66

errordeveloper commented 5 years ago

@whereisaaron that seems to be more related to #586, don't you think?

whereisaaron commented 5 years ago

Oh yes, didn’t spot it. Moved it over.