Azure Support

[chrislovecnm]

Problem Statement

Support the running a kops cluster on Azure. This implementation would model AWS and GCE, and would be awesome!

How

Azure includes full API support for the creation of virtual machines and their scaling groups. There are multiple examples of the 'hard way' but we first can model there managed version.

Why

Why not ? Having AWS, GCE, Azure is the trifecta

Design

TBD

[robinpercy]

+1 Cloud-agnosticism is a consistent priority I hear from customers deploying K8s. Being able to use Kops across all three would solve a lot of pain.

[chrislovecnm]

@robinpercy I have gotten asks from my clients that they want to run on the big three. We got two down, one to go

[vendrov]

Great idea! I believe that our first step should be, adding windows nodes support for kops :)

[chrislovecnm]

Yes we do need windows node support as well. Will open a separate ticket. I have talked with Microsoft shops who would love that,

[vendrov]

No need to open a new one, https://github.com/kubernetes/kops/issues/2422

[bgeesaman]

I've looked at over a dozen K8s installation tools and managed K8s services in the past few months on all major providers. What was painfully clear was the level of fragmentation in the tools and offerings as well as what features are supported. It required a ton of mental energy to translate terms and patterns to get similar output.

We are at a point where focused efforts like this can provide that next level of true abstraction so that administrators/operators can see and manage Kubernetes uniformly--regardless of cloud provider. I think bringing all the advanced features and capabilities (HA, rolling-upgrades, etc) battle-tested in AWS and GCE to Azure is a no-brainer.

[chrislovecnm]

@bgeesaman well put - fortunately the coming machine api should help with this, but the installers do have different use cases

[geojaz]

We should be building out azure support regardless of windows nodes. We can definitely consider building out windows node support in kops, but I would start with the basics. That is to say, let's build out linux support first because that's what we support on AWS and GCE. That will bring some level of feature parity to the 3 biggies via kops.

I too have clients that need 1 tool to manage all the clusters across these providers and kops is a great way to go. We now have much of the background work done to build a new provider so I think it's quite doable. I'm personally not that up to speed on Azure, but I'd like to see a good "azure-native" implementation to complement aws and gcp. Excited to see this get attention and I'm happy to help.

[robertojrojas]

IMHO, Azure support is a must have! I agree with @geojaz windows support would be nice, but Linux should come first. I'll be interesting to see what changes would be needed in the CLI and API to accomplish this. I'm happy to help as well.

[brandoncole]

This would be great. There are definitely some companies who prefer Azure to AWS in foreign countries which is where this comes into play a lot. In the U.S.A. there's indifference to some degree in the cloud vendor, but there's certain places globally where the choice has been made for you which we have to respect.

[chrislovecnm]

/assign

[justinsb]

Yes, I'd love to see us work on this. Azure is doing some very cool stuff, and it would be great to help people to try out Kubernetes on Azure as easily as they can try out AWS & now GCE.

[hubt]

This would be excellent. Windows support isn't important at least initially. People need this for cloud portability(usually because of cost), not OS portability.

[chrislovecnm]

@hubt we have had one request for windows which mean x100 have wanted it. But KISS right? And that work can be completed in ||.

[perryao]

@chrislovecnm Is this work active or still awaiting the machine api?

[chrislovecnm]

@perryao nobody has started on this, would love someone to pick this up!

[perryao]

I'd like to try my hand at this. I'll work on a rough version over the coming weeks and hopefully come back to discuss design with you @chrislovecnm

[chrislovecnm]

@perryao let me know, if you have questions happy to chat with you.

[rajeevb2]

@perryao any news on kops for azure?

[perryao]

@rajeevb2 unfortunately not too much. I have started and have a pretty good feel for what needs to be done, but haven't had enough time to make meaningful progress on it yet. But still a priority for me.

At this point I've just stubbed out the tasks I'll need in cloudup and starting implementing a few interfaces.

[chrislovecnm]

@perryao if you want to add a feature flag, please start pushing code. Love to have more than one person working on this if you feel it would help.

Sometimes more people don't help 😆

[zarakay]

Happy to help with testing if you need it

[marc-sensenich]

@perryao If you would like to have some help on this from the testing and development front, I'd be willing to help out

[perryao]

@chrislovecnm @marc-sensenich thanks guys. I just pushed the initial stuff I did to layout the changes I'd need to make. Here's the diff of where I'm at now: https://github.com/kubernetes/kops/compare/master...perryao:feature/%233957azure-support?expand=1. I'm wrapping up a deliverable at work this week, but I think I should be back to working on this early next week.

[chezbut]

Is there any plan for being able to run this on local Hyper-V hosts, with multiple VM's?

[Cittarasu]

@chrislovecnm @perryao -- great start on Azure front. I can help in testing/development front. Let me know how we can sync-up on this front. We have been using KOPS for AWS extensively but we are exploring KOPS for vsphere and Azure.

[luisdavim]

Any updates on this?

[thangchung]

+1

[surenm]

Great start @perryao @chrislovecnm. Happy to help with the testing effort!

[wandergeek]

+10

[rajivchodisetti]

+10

[billpratt]

Hello everyone. I have started to work on this during free time. You can track progress here: https://github.com/billpratt/kops/commits/azure

So far I have basic state-store functionality working with Azure blob storage.

[dazdaz]

Awesome to see the beginning of the work to implement Azure support with kops. May I ask what the current status is. Have you considered creating a roadmap of features/comaptibility to highlight where we're at and what requires developer effort.

[billpratt]

@dazdaz I havent had as much time lately to work on this. I plan to start up again soon. I originally created this project on my fork. It needs a little bit of cleanup. I also ran into a blocker where a downstream dependency of kops requires a specific version of the azure-sdk-for-go. I will need to update to use a newer version. I need to also evaluate what version of k8s to start supporting this in Azure. Kubernetes 1.12 released support for Azure VM Scale Sets, which gives similar functionality as AWS scale sets. If kops is going to support < 1.12 for Azure, I'll have to use Azure Availability Sets which does not give you automatic cluster scaling.

[argiris-mat]

@billpratt hello there. Can we do anything to support this? Is there anything I could test or document?

[billpratt]

Not yet. I have some free time coming up and I plan to continue to work on this. I will let you know when something is ready.

[igormusinov]

any updates?

[innlouvate]

Any chance this might be coming in the next 6 months or so? Would happily support if I can.

[bzuelke]

i need this in my life so bad. I hope in the near future this can become a reality. I hate having to force myself to have a snowflake in Azure from our AWS clusters.

[DG4227]

Would love to see this come to fruition!

[bzuelke]

@billpratt I've looked over your fork and looks like you've done a ton of leg work. Is there anything that anyone here could do to try and get this in a PR? Didn't want to reinvent the wheel if I didn't have to. I'd love to have something in place for this before end of year. Would be incredibly useful.

[bryaneburr]

MS loves to give away free time on azure, and higher-ups love to take it. This would make devops' lives like a billion times easier 😀

[billpratt]

Hi everyone,

This was strictly a personal project of mine that I worked on at night and no work was done as a Microsoft employee. Unfortunately I have had no time finish. @bzuelke, if you or anyone else on this thread would like to fork what I've done and take over, please do as I would also love to see this feature added 😃

[bzuelke]

Thanks for getting back to the thread @billpratt I totally get where you're at. I'm kind of in the same boat time wise which is frustrating naturally. I know you have a decent amount of leg work put into the code and if the universe is willing to grant me some time to check it out I'll 100% fork it and muck with it. Or if someone else jumps on it I'm totally open to collab to work this out.

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[pierluigilenoci]

@billpratt any news?

[bzuelke]

Chances are there won't be any support for Azure unless someone really jumps on it. But I think more people use other forms of cluster management until Cluster API goes GA or simply uses AKS. I would have like to take a stab at this but simply don't have resources available to follow up on Bills work :/

[christianh814]

Chances are there won't be any support for Azure unless someone really jumps on it. But I think more people use other forms of cluster management until Cluster API goes GA or simply uses AKS. I would have like to take a stab at this but simply don't have resources available to follow up on Bills work :/

+1 to this comment and the ":/" ...realistically; when Cluster API matures, KOPS would/should use that as the core of the tool.

[prksu]

/remove-lifecycle stale

[rendhalver]

Just catching up on this. I see cluster-api has a non pre-release release in 0.28.0. Is that what people were waiting for? I am also interested in helping out where I can. I have been looking for reason to learn more go and we use kops to deploy our clusters and having the ability to manage all of them with kops would be grand.