Tracking Issue: Concise outline for using cloud providers

[srflaxu40]

I was informed to post an issue here outlining various issues setting up kubeadm. The underlying issue I found is not having a clear and concise document, and for those that are looking for a walk-through on enabling cloud-provider features in any cloud the task is very daunting and under-documented with the exception of one or two blog posts that are for openstack or other providers (not AWS).

Could we please get documentation involving the following steps and elaboration with clear/concise explanations pertaining to specific providers:

1. Modifying / creating a cloud-config file alongside adding the --cloud-provider / --cloud-config flags in the apiServer and kube-controller files for kube-master is not sufficient.
2. Additionally, one has to update kubelets to contain the --cloud-provider / --cloud-config flag in "extra args env variable". Does this require restart, or any stateful order? Start master, then start kubelets, etc.
3. This is not sufficient for enabling the cloud provider; AWS requires specific IAM permissions in order to interact with Amazon as well as specific tags 'KubernetesCluster=' (from cloud-config file on all nodes). However, there is no documentation that explains the exact resources - it is assumed that you have to tag everything. Even then, after tagging and after giving full access to S3, EC2, VPC (just for testing), etc, I still received could not update routes errors. I could create EBS volumes in Amazon using PVs / PVCs, but could not mount them because my pods could not longer be scheduled with strange tolerations/taints that seemed to prevent them from getting on any nodes for scheduling.
4. Even after 1-3 steps this is still insufficient to get kubeadm running with --cloud-provider on AWS. It additionally broke my calico installation, which leads me to wonder how possible this is.

I think if someone created a step-by-step, fail-proof document showing setting up a master with kubeadm using the kubeadm init command, and then corresponding joins with underlying kubelet configuration for a cluster it would be very helpful. This is a very special use-case because It allows dynamically joining nodes without having to do a universal re-apply, and it also enables/allows-for AWS provider-centric features in kube. This particular use-case is not available in KOPS / kubespray, and other tools I have seen.

Current documentation I have gone through that appears lacking / insufficient:

kubeadm on open-stack k8s-cloud-provider-notes

I feel like kubeadm is exciting and although I am using it without provider enabled it would be nice to have some of the features working for things like persistent disk. This could use some TLC I think as having to go through code is really time consuming for those unfamiliar with the code-base.

[arthur0]

I think it's can be resolved soon. The approach using config file is interesting because you can pass the kube-api-server and kube-controller-manager extra args in the yaml definition, e.g.:

apiServerExtraArgs:
    cloud-provider: "openstack"
    cloud-config: "/etc/kubernetes/cloud.conf"
controllerManagerExtraArgs:
    cloud-provider: "openstack"
    cloud-config: "/etc/kubernetes/cloud.conf"

Reference: https://kubernetes.io/docs/admin/kubeadm/#config-file

It's not possible to configure volumes mount for the /etc/kubernertes/cloud.conf file yet, however, it's already implemented here: https://github.com/kubernetes/kubernetes/pull/49840/

Recently, the cloud-config file documentation for OpenStack was added and I think the other cloud-providers can do the same: https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/#openstack

Another point about cloud provider integration is the in-progress refactoring to support out-of-tree and out-of-process cloud providers (a.k.a pluggable cloud providers), see the proposal: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/cloud-provider/cloud-provider-refactoring.md

So, the cloud providers integration still in an experimental phase, but I think the kubeadm is going on the right way: https://kubernetes.io/docs/admin/kubeadm/#cloud-provider-integrations-experimental

I think it's more urgent is to create/adapt documentation about:

• cloud-config file for the other providers
• cloud provider integration itself
• tutorials using the kubeadm config-file

cc @luxas

[tamalsaha]

@srflaxu40, have you looked into https://github.com/appscode/pharmer . These sort of issues are addressed out of the box in Pharmer.

[srflaxu40]

No @tamalsaha I am trying to use something under the tree at the moment, and my use-case is very specific - does pharmer allow the ability to dynamically join a mode via kubeadm without re-applying the entire cluster.

[tamalsaha]

Does pharmer allow the ability to dynamically join a mode via kubeadm without re-applying the entire cluster.

Yes.

You can check the flags that we pass around for cloud-config/provider in pharmer for aws.

[srflaxu40]

So, I can bring nodes up one at a time through their userdata?

[tamalsaha]

We use a NodeGroup concept. All nodes in that group has same UserData using LaunchConfiguration. Then you can scale up/down that group. If you want different configuration, you will need to create a different NodeGroup.

[srflaxu40]

Here's the thing though - the key component of my use-case is it has to run in SpotInst?

[tamalsaha]

ok. spot instances are not supported currently.

[luxas]

@srflaxu40 Thanks for this issue. We know this is a pain point and we're in an in-between state currently, the community is in the middle of moving these cloud providers out of tree. As harsh as it may sound, it is in the end a responsibility of the cloud provider in question to provide documentation on what is needed for each case. kubeadm can't do anything for you here (e.g. setting up IAM roles for you-name-it-cloud-provider).

If you're interested in the cloud provider refactoring and how the process will get better, look at the out of tree docs on kubernetes.io and join #wg-cloud-provider on Slack. The group is meeting 10am PT today and every Weds.

@tamalsaha @srflaxu40 Let's not get distracted :)

Thanks for the comment @arthur0!

TL;DR; What actual action items do we see here right now that might affect kubeadm?

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten /remove-lifecycle stale

[timothysc]

/assign @liztio

[aisensiy]

Trying to make kubeadm installed cluster to work with EBS in aws but failed. Hope there is some good instruction for that.

[timothysc]

/cc @luxas - this is definitely needed for 1.11 release.

[Bradamant3]

@timothysc what's changed since last November? sig-docs is still working on getting cloud provider docs out of tree; nothing has changed there. I can see the value of documenting more clearly whatever new features in kubeadm support cloud provider installs, but should be common to kubeadm, not specific to cloud providers. Clarification would help, thanks!

[neolit123]

it the kubeadm office hours meeting today (06.06.2018) it was decided the following:

We don’t have cloud provider instructions for kubeadm - the docs are here, but we should probably only link to them + some brief details, as we don’t want to expand on cloud provider specifics: https://kubernetes.io/docs/getting-started-guides/cloudstack/

https://docs.google.com/document/d/130_kiXjG7graFNSnIAgtMS1G8zPDwpkshgfRYS0nggo/edit#

^ the above is open to discussion. @timothysc @Bradamant3 @srflaxu40

[timothysc]

I'll take this one.

[timothysc]

So we discussed this today on the sig call. It would be ideal to have a limited set of overrides outlined then link to a separate doc that sig-cloud-provider maintains.

/cc @andrewsykim

[andrewsykim]

FYI @hogepodge re: https://github.com/kubernetes/website/issues/8304

[timothysc]

We are working with @kubernetes/sig-cloud-provider-misc to help get better refs in place, but is may lag the v1.11 milestone so we are punting out, but actively working on the details.

[timothysc]

@andrewsykim could you cross-link the proposal you were talking about, please.

[andrewsykim]

@timothysc @hogepodge is currently working on putting something together, will cross-link as soon as it's out.

[hogepodge]

I don't want to leave this without a response.

Each cloud provider will be expected to produce documentation on how to enable, configure, and develop their provider. With the move towards external providers, this issue is only going to get more complicated, especially with automated installation tools. More details about the challenges of external providers can be found here:

https://kubernetes.io/docs/tasks/administer-cluster/running-cloud-controller/

cloud-provider-openstack has some documentation in place, but it doesn't completely address the issue you're describing.

https://github.com/kubernetes/cloud-provider-openstack/pull/67 https://github.com/kubernetes/cloud-provider-openstack/pull/224 https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/using-controller-manager-with-kubeadm.md

Coming up with a base set of instructions that are widely applicable, with individual provider instructions for both in-tree (which sig-cloud-provider is hoping to deprecate in time) and external that are validated is essential. I'm working with @d-nishi on this.

[neolit123]

we hand two reports the last couple of days, about users not understanding how to pass the cloud provider to the control plane and the kubelet from kubeadm.

[NeilW]

I've been writing the Brightbox cloud controller manager over the last few weeks and I've built up a list of questions and things that I've had to work out to integrate it into kubernetes and in particular into a kubeadm built cluster.

The challenges are rather more extensive than the ones listed in the example above - particularly if you want to talk or serve securely with certificates in the kubeadm fashion.

Implementing the cloud controller interface is relatively straightforward. It is working out what is hidden away in the cloud controller manager wrapper application and how to get that working that is the difficult bit.

How the cloud controller manager operates and what it needs shouldn't change between provider plugins, and the provider plugins generally just need a secret and/or a configmap file in addition to that.

I can relate my experience on this or other relevant ticket if that helps. Let me know.

[andrewsykim]

@NeilW if you can create an issue in kubernetes/kubernetes and tag sig-cloud-provider we'd be happy to discuss it in our next meeting.

[timothysc]

@andrewsykim @d-nishi - are PRs up for cleaner cloud-provider integration now?

/cc @Bradamant3

[timothysc]

xref - https://github.com/kubernetes/community/blob/master/keps/sig-cloud-provider/0019-cloud-provider-documentation.md

[neolit123]

hello, @andrewsykim @hogepodge

what is the state of https://github.com/kubernetes/community/blob/master/keps/sig-cloud-provider/0019-cloud-provider-documentation.md in terms of k/website docs follow ups?

would we be able to close this issue for 1.12? /cc @timothysc

[timothysc]

@neolit123 We simply need to point to a page. Ideally, it would be a post-deployment step similar to CNI but for out-of-tree cloud providers. https://docs.google.com/spreadsheets/d/1zqqh_S8qZBJx6DPTXrjElN1RJXiQGnIymPuVL4PUczo/edit#gid=0

[neolit123]

@timothysc understood, once the k/website (?) page is created we are going to link to it.

[timothysc]

Yes.

[timothysc]

It doesn't look like we can link out until 1.13 at this point.

[andrewsykim]

Apologies for not getting this completed in time for v1.12.

[andrewsykim]

As we work through docs in the next release, if you have anything specific you want included as part of provider docs (i.e. steps for kubeadm install) please propose those in https://github.com/kubernetes/community/blob/master/keps/sig-cloud-provider/0019-cloud-provider-documentation.md

[timothysc]

@andrewsykim what's the state of this for 1.13?

[andrewsykim]

@timothysc sorry for the late response, was out last week on a company offsite.

We're still executing KEP-0019 and tracking progress of docs from each provider in the Cloud Provider Tracking Sheet (need to be in kubernetes-sig-cloud-provider mailing list to access). As for consolidating the docs together @chenopis is leading that effort and will have some more updates soon.

[timothysc]

/assign @dims

dragging you in ;-)

[timothysc]

@andrewsykim @chenopis - Where do the docs stand for 1.13?

[dims]

@timothysc Boo! :)

[hogepodge]

@timstclair Rounding up the state of documentation for all the providers. Looking to make this consistent across all the providers.

https://docs.google.com/document/d/1yj8z2ctFZKqUGF40U_MB_7VZD8VuEz0eLFVg_CQYvzI/edit?usp=sharing

[NeilW]

@hogepodge

The Brightbox cloud-controller-provider documentation is here.

Design and development: https://github.com/brightbox/brightbox-cloud-controller-manager/blob/master/README.md

Installation and configuration: https://github.com/brightbox/brightbox-cloud-controller-manager/blob/master/config/README.md

if you want to add that into the list. Once there is a template to work to I'll make the Brightbox docs fit.

[onitake]

@hogepodge

We're still in the early alpha phase with our factored-out Cloudstack CCM, so the documentation only consists of what's written in the README: https://github.com/swisstxt/cloudstack-cloud-controller-manager/blob/master/README.md#use

Should we reformat this into something more in line with other cloud provider documentation or will this be ok?

[dims]

/unassign

[timothysc]

I'm going to close here now as https://github.com/kubernetes/cloud-provider/issues?q=is%3Aissue+is%3Aopen+label%3AP0 is tracking their updates.