Discussion: Extending Cloud Providers

jonathan-innis commented 1 year ago

Tell us about your request

There's a growing number of requests to extend the cloud providers that Karpenter supports from AWS to other cloud providers (Azure, GCP, Orcale, etc.). This issue is intended to cover the discussion of extending the number of cloud providers broadly.

Additional Context

Related Issues

aws/karpenter#2507 aws/karpenter#936

Tracking Tasks

[ ] #754

Attachments

No response

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

ellistarn commented 1 year ago

I'd like to explore two models for this: 1: Karpenter as a Library (as currently implemented here: https://github.com/bwagner5/karpenter-k3d) 2: Karpenter as an API: https://github.com/kubernetes-csi

A few table stakes in my mind:

Vendors should be able to define their own API surface (e.g. AWSNodeTemplate) that's separately versioned
Vendor code should live in a separate repository from Karpenter core (k8s is doing this across the board w/ in-tree / out-of-tree)

debugger24 commented 1 year ago

Happy to work on this. Let me know how we can collaborate.

jonathan-innis commented 1 year ago

@debugger24 Are you interested in helping/building a particular provider? GCP? Oracle? Digital Ocean?

tonymarkel commented 1 year ago

I'd like to help with the Oracle portions of this in whatever capacity I can. If I read @ellistarn correctly, I'm okay to create a separate repo like @bwagner5 did. I haven't been able to find an equivalent open-source project on the 'net, so hopefully I'm not re-inventing a wheel. If I don't see anything in this thread in a day or so, I'll follow Brandon's example and put what little I have in a repo and share it here.

jonathan-innis commented 1 year ago

Nope! @tonymarkel. Feel free to take a stab at the Oracle cloudprovider and share the repo in this issue!

ellistarn commented 1 year ago

The best example you'll find is to look at aws/karpenter as the aws implementation. You'll need your provider to rely on aws/karpenter-core.

sftim commented 1 year ago

Should we label this with neutrality?

Anhui-tqhuang commented 10 months ago

seems that if we want to make a cloud provider, we cannot reuse the existing repo as it is tied to AWS implementation, we need to use karpenter-core to create our own karpenter ....

many implementation is not interfaced like

ellistarn commented 8 months ago

For an example of another provider implementation, check out: https://github.com/Azure/karpenter

jackfrancis commented 8 months ago

For an example of another provider implementation, check out: https://github.com/Azure/karpenter

Note the source attribution section:

https://github.com/Azure/karpenter#source-attribution

Which is to say: for folks looking to rapidly prototype a new provider the AKS provider above is proof that you can just take the existing AWS bits and replace the relevant AWS-specific stuff with your own stuff.

cc @Anhui-tqhuang

There are valid points about how we can improve the portability of the various provider interfaces, but it shouldn't be a blocker per se for folks who are eager to build now.

jonathan-innis commented 6 months ago

Looking over this issue: I'm realizing that what this issue really is is a discussion issue around a bunch of things, providing a common space for any cloud providers that want to implement a Karpenter cloud provider to come to have an active discussion about how to do it.

There's no true exit criteria for this issue, so I'm changing this to be prefixed with "Discussion:"

sftim commented 6 months ago

We can get this converted to a GitHub Discussion if you like, @jonathan-innis.

jonathan-innis commented 6 months ago

We can get this converted to a GitHub Discussion

@sftim My one reservation about converting this to a discussion is routing. Do we have a good feeling which issues will get routed as discussions and which ones will just be bare-bones issues. I worry if we start having separate discussions for some one-off issues over in the Discussions tab in Github that we are going to lose people who are used to going to the issues section.

I also don't want this to be the only issue that ends up converting into a Discussion. Realistically, I see a lot of "discussion-y" issues that probably still make sense classified as issues so I wonder if we just keep this as-is for now.

Happy to hear any additional insight that you might have here.

Bryce-Soghigian commented 5 months ago

Another Datapoint this thread might be interested in: https://github.com/kubernetes/autoscaler/issues/5394#issuecomment-1485331666

k8s-triage-robot commented 2 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can: