search

maksim-paskal / aks-node-termination-handler

Gracefully handle Azure Virtual Machines shutdown within Kubernetes
Apache License 2.0
34 stars 6 forks source link

Feature request: Run in Queue mode similar to AWS NTH #86

Open heschlie opened 3 weeks ago

heschlie commented 3 weeks ago

I'm mostly asking this to see if there is interest for this, I haven't dug deep enough to know how complex this might turn out. If there is interest I'd be happy to assist with development of this feature.

I think the requirements are pretty simple?

maksim-paskal commented 3 weeks ago

@heschlie, thank you for bringing up this issue. It's an interesting feature. However, could you elaborate on the problem this feature would solve?

heschlie commented 3 weeks ago

I'll admit I'm still exploring this and AKS, but with the AWS NTH running in Daemonset mode there were sometimes issues with the node being properly cleared out as it was draining the node it was on, depending on how your network stack and resources were allocated. In the end it always sorted itself out as that is a strength of the eventual consistency of k8s. Running in Queue mode mostly removed that problem as you ran one pod to manage all of the nodes tainting and draining. There would still be the occasion that the node that it was running on would be the one getting drained, in which case you would still be relying on the eventual consistency of the cluster but this happened much less.

There are other minor benefits in that it runs less pods, so less resources (not just CPU and memory, but IPs, loggging, etc.), easier to troubleshoot as you can debug in one log stream, less clutter, and if you run a service mesh less pods to include in the mesh if you need to encrypt all comms. You cloud also carve out which ASGs you wanted to monitor based on some configuration, though that feature wasn't as helpful unless you wanted to ignore some ASGs for some reason and you could accomplish the same via taints and tolerations with this too.

Again this is something I would consider a nice to have not a requirement by any means. I have plans to start using this to manage our node pools that will run spot instances so I'll have more of an opinion when I get some time with this.

Thanks for the quick reply btw.

maksim-paskal commented 3 weeks ago

Thank you for the details. So, in essence, this feature does not solve any problems with taint/drain of nodes in AKS and we do not know for certain if it will improve reliability.

On the other hand, you're correct that theoretically, moving from DaemonSets to Deployment could use fewer resources. I think this would be a great enhancement. I need to explore some options in Azure Cloud. If you have any insights on how to implement this, I would greatly appreciate the information. Upon a quick review of the documentation, I didn't find any options on how to create this queue in Azure.

heschlie commented 3 weeks ago

yea that is where my hangup is, in AWS you can pipe the VM lifecycle events to an arbitrary location including SQS. Assuming there is a convenient way to do that in Azure it should be just a matter to watching those events.

I'll see if I can find a way to do that in a way that doesn't feel super hacky, providing I find a way to do that I let you know and see if I can mock up a prototype that we can discuss a bit more.