Karpenter support for cluster autoscaling

[alexisbel1]

Karpenter is an open-source node provisioning project built for Kubernetes. Its goal is to improve the efficiency and cost of running workloads on Kubernetes clusters. Karpenter works by:

• Watching for pods that the Kubernetes scheduler has marked as unschedulable
• Evaluating scheduling constraints (resource requests, nodeselectors, affinities, tolerations, and topology spread constraints) requested by the pods
• Provisioning nodes that meet the requirements of the pods
• Scheduling the pods to run on the new nodes
• Removing the nodes when the nodes are no longer needed

Karpenter has many advantages over cluster autoscaler. One prerequisite would be that AKS can manage multiple instance types without defining multiple node pools.

Currently the only cloud provider which support Karpenter is AWS.

It would be awesome to have AKS support it.

[sftim]

If anyone wants to work on this and would like tips, ask in #karpenter-dev on the Kubernetes Slack workspace. Visit https://slack.k8s.io/ for a (CAPTCHA gated) invitation.

[stevehipwell]

@sftim I don't think it'd be possible to implement Karpenter on AKS (and have it be useful) without either AKS support for Flexible VMSS node pools or AKS support for adding a non-VMSS machine as a node.

[sftim]

I think I can sketch out how to design this within existing constraints, but:

• most of my consultancy work is aimed at AWS
• it feels big enough that I'd prefer to frame it as paid work, if I did take it on (more likely: I don't)

[dnitsch]

I haven't gotten too much into it - but this seems to suggest you can.

presumably some caveats will need to be made to AKS users that they must provision multiple nodepools with different VMs in order for it to work. There is also the possibility to add nodepools on their behalf programatically 🤷‍♂️, if people are happy to allow Karpenter managed identity to also make changes to AKS itself.

The changes inside karpenter would be quite big as well but should be possible.

[stevehipwell]

I don't think it'd be possible to implement Karpenter on AKS (and have it be useful)

I don't think the current node provisioning model in AKS would support the Karpenter workflow for anything functional let alone be close to the fleet API it uses in AWS to offer similar capabilities.

[dnitsch]

@stevehipwell are you thinking response times?

[stevehipwell]

@dnitsch the following list of concerns is representative but not exhaustive.

• Node ready response time
• Node customisation (lack of user data support)
• Node type/size variance (how many VMSSs can be created)
• Availability metrics (what node types should be requested based on availability)
• General API limitations

[wasabii]

I think it's pretty clear that adopting Karpenter requires the AKS team itself.

There is not really a need for node pools at all. That's an Autoscaler concept adopted into AKS. There's some sort of other abstraction that would need to be developed.

[palma21]

This thread became a bit dormant so apologies for not providing an update sooner. We have been working on this though.

We're committed to provide all the use cases from vertical scaling, SKU/size optimization and many others folks are requested, please do let us know what scenario you're interested in if it's not listed above We expect to have news for this very soon, in the upcoming weeks.

Wrt to karpenter specifically it's a single-entity governed project at the moment so we cannot commit one way or another to supporting its interfaces and the project itself in AKS. We are happy to consider it if it ever becomes a community governed project. But until then we will be providing the implementation and functionality for these requirements

[bplasmeijer]

@palma21 showing off Karperter at #kubecon 🔥🔥🔥🔥 Preview ETA @palma21?

[dynek]

What about https://azure.microsoft.com/en-us/updates/karpenter-support-in-aks/ ?

[Flawke]

What about https://azure.microsoft.com/en-us/updates/karpenter-support-in-aks/ ?

They removed it..

[stevehipwell]

@Flawke it certainly looks like it.

[dynek]

Google cached it, hopefully something's being worked on behind the scene :-)

[conallprendergast]

I am told that this was announced accidentally, but should be coming out in a week or two.

On Thu 20 Apr 2023, 11:48 Pierrick Brossin, @.***> wrote:

Google cached it https://webcache.googleusercontent.com/search?q=cache:Y_-RP7rnjqAJ:https://azure.microsoft.com/en-us/updates/karpenter-support-in-aks/, hopefully something's being worked on behind the scene :-)

— Reply to this email directly, view it on GitHub https://github.com/Azure/AKS/issues/2712#issuecomment-1516038536, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE5GOHJ7DD77O4COXYYTEHDXCEA6TANCNFSM5K27BV6Q . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[Sabarish-ram]

waiting for the karpenter feature in AKS

[bplasmeijer]

any update @palma21 ?

[bplasmeijer]

Please vote! We need to get Karpenter on all cloud providers to save the world!

https://github.com/aws/karpenter-core/issues/756

@palma21 @ellistarn

[ppodevlabs]

do we have an ETA for this one?? would be really appreciated :P

[hronix]

+1

[Lavanya-Anbalagan]

+1

[warolv]

+1

[Anhui-tqhuang]

+1

[kethahel99]

+1

[kevin-bates]

+1

[swapnil-gohil]

+1

[SudhamshBachu]

+1

[dunefro]

+1

[bplasmeijer]

Choosing between smaller and larger node types in Kubernetes depends on factors like workload needs, costs, and efficiency.

Here are the pros and cons:

Smaller Nodes:

Pros:

1. Cost-Efficiency: Lower expenses for lightweight workloads.
2. Resource Use: Efficient resource allocation for less demanding tasks.
3. Scaling: Easier fine-tuned scaling.
4. Isolation: Better workload separation.

Cons:

1. Limited Resources: Potential performance issues for demanding tasks.
2. Complexity: More management for numerous small nodes.
3. Network Impact: Higher network load.

Bigger Nodes:

Pros:

1. Performance: Handles heavy workloads effectively.
2. Simplicity: Easier management.
3. Consolidation: Efficiently accommodates multiple smaller tasks.

Cons:

1. Costly: Higher expenses, especially if resources need to be fully used.
2. Resource Use: Wastage if not optimized.
3. Scalability: Overprovisioning risks.

A mix of small and large nodes could suit diverse workloads. Kubernetes features like resource management aid practical usage, regardless of node size. Base decisions on workload, performance, cost, and operations, with regular monitoring for optimization.

⚠️Save cost and optimize the compute workload, and not have wasted compute.

[asahnovskiy-deloitte]

+1

[geekyshameem]

+1

[ppodevlabs]

I am told that this was announced accidentally, but should be coming out in a week or two. … On Thu 20 Apr 2023, 11:48 Pierrick Brossin, @.> wrote: Google cached it https://webcache.googleusercontent.com/search?q=cache:Y_-RP7rnjqAJ:https://azure.microsoft.com/en-us/updates/karpenter-support-in-aks/, hopefully something's being worked on behind the scene :-) — Reply to this email directly, view it on GitHub <#2712 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE5GOHJ7DD77O4COXYYTEHDXCEA6TANCNFSM5K27BV6Q . You are receiving this because you are subscribed to this thread.Message ID: @.>

well, it is taking a bit longer.... Any news?

[bplasmeijer]

Any update @palma21???

[palma21]

All updates for this are now on/depending of: https://github.com/kubernetes/org/issues/4258

AKS does have an autoprovision item that will be in preview before the end of the year. https://github.com/Azure/AKS/issues/2442

[lupass93]

+1

[ronmegini]

+1

[sravanakinapally]

+1

[palma21]

https://twitter.com/jorgefpalma/status/1721944779011858729

https://github.com/Azure/karpenter

[stevehipwell]

@palma21 I don't see any docs for how to actually use Karpenter in AKS? 👀

[PixelRobots]

@palma21 I don't see any docs for how to actually use Karpenter in AKS? 👀

Currently only the open source option that's linked above is available. Hopefully a built-in option such as an AKS add-on will be out soon.

You can check out this blog post on how to use the open source method.

https://massimocrippa.com/blog/f/karpenter-provider-for-aks

[stevehipwell]

Thanks for the link @PixelRobots. I'm all for deploying my own K8s components, but I don't think we're really at the stage for an announcement given that there isn't an official build of the image yet. Or any official docs as to how it works like there is for EKS.

I'd also like to know how AKS plans to maintain their provisioner without keeping a fork of the whole code base?

[palma21]

Not yet, we just announced the OSS provider. The add-on and respective docs will come soon.

The provider code is cloud specific, core code is shared by all providers. We're not planning to make any forking of the core code (same as with all OSS projects we use).

[stevehipwell]

@palma21 I'm not looking for an add-on, but I'd like an OCI image and some docs explaining how the AKS implementation works to go with the announcement. Last time I checked the repo seems to contain the docs for the AWS version and point to that site.

[justindavies]

@stevehipwell The documentation for the AKS provider will be released with the add-on, and the Open Source repository will be updated with a link to that documentation for customers to reference, as well as updating the AKS specifics in the repo itself. As Jorge mentioned, the repository we announced contains the AKS Karpenter provider, and will be the home for any work we do that is provider specific

[0dragosh]

@justindavies is there a roadmap we can all follow so we prevent questions like "what's the ETA" ?

[justindavies]

Good morning all, just to let everyone know we announced Node Autoprovision earlier today: https://learn.microsoft.com/en-gb/azure/aks/node-autoprovision?tabs=azure-cli

[bplasmeijer]

@justindavies any update on windows containers?

[tppalani]

Hi All

just want to know the update is that azure will support karpenter autoscaler or still it is in discussions state?

[zioproto]

This issue should probably be closed. Please check the following links:

• https://azure.microsoft.com/en-us/updates/provider-for-running-karpenter-on-azure-kubernetes-service-aks/
• https://github.com/Azure/karpenter-provider-azure

[bplasmeijer]

No windows support yet. Please keep it open.

On Fri, 17 May 2024 at 08:38, Saverio Proto @.***> wrote:

This issue should probably be closed. Please check the following links:

- https://azure.microsoft.com/en-us/updates/provider-for-running-karpenter-on-azure-kubernetes-service-aks/

• https://github.com/Azure/karpenter-provider-azure

— Reply to this email directly, view it on GitHub https://github.com/Azure/AKS/issues/2712#issuecomment-2116856160, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAA4N4ZAIIIELMPTRWWHJD3ZCWQWTAVCNFSM5K27BV62U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJRGY4DKNRRGYYA . You are receiving this because you commented.Message ID: @.***>