search

kubernetes / enhancements

Enhancements tracking repo for Kubernetes
Apache License 2.0
3.42k stars 1.47k forks source link

Dramatically Simplify Kubernetes Cluster Creation (kubeadm umbrella issue). #11

Closed jbeda closed 5 years ago

jbeda commented 8 years ago

Description

Creating a new Kubernetes cluster is too hard. We need to simplify the number and types of actions to get a production cluster up and running.

Note that this is different from bringing up a development cluster (single node ala monokube or minikube) or automation around cluster creation (https://github.com/kubernetes/community/wiki/Roadmap:-Cluster-Deployment).

If we do this right, the number of manual steps to get a cluster running should be minimal. This will have the added benefit of making other deployment scenarios (dev cluster, cluster automation) simpler and smaller.

As part of this, we should make simplifying assumptions and have opinionated defaults. An example would be embedding etcd and picking an easy to use network technology. Certificates and trust should be established automatically.

Progress Tracker

_FEATURESTATUS is used for feature tracking and to be updated by @kubernetes/feature-reviewers. FEATURE_STATUS: IN_DEVELOPMENT

More advice:

Design

Coding

Docs

jbeda commented 8 years ago

Other related efforts/prior art:

[I'll update this comment with new links as they come in]

jbeda commented 8 years ago

@mikedanese -- I know this is a lot of what you've been working on. I'd love to get that reflected here and scoped for 1.4. Do you mind shooting some pointers over?

philips commented 8 years ago

cc @derekparker @aaronlevy @pbx0 from the CoreOS team working on https://github.com/coreos/bootkube and the self-hosted stuff with @mikedanese to realize a k8s driven creation and update story.

jbeda commented 8 years ago

To be extra clear -- I'm proposing that we make this experience part of core kubernetes. The fact that core k8s is a set of things that have to be set up together is a powerful thing but it makes things look very very complex. We should be willing to have a sane set of defaults and embedded solutions built in to the main distribution.

Right now our "manual install" page is incredibly daunting. We should aim to reduce that (at least for a given set of integrated services) to a single screen without the crutch of automation tools that paper over the complexity.

mikedanese commented 8 years ago

There is a class of infrastructure (that doesn't currently exist) that would benefit all deployment automations. We should try to enumerate what these items are, give them relative priorities and advocate for them in v1.4 planning. I started to create a list a couple days ago: https://github.com/kubernetes/kube-deploy/issues/123 cc @justinsb @errordeveloper @bgrant0607

pires commented 8 years ago

@jbeda https://github.com/kubernetes/kubernetes/issues/5472, https://github.com/kubernetes/kubernetes/pull/16077 and https://github.com/kubernetes/kubernetes/issues/5754 are related to this.

thockin commented 8 years ago

Something that crossed my mind with docker 1.12 - built-in kvstoreeans that libnetwork's overlay driver might be viable for us. Having a built-in network mode for Docker installs that works anywhere and doesn't require extra components might be nice.

Might require some work to not assume prefixes per node.

klizhentas commented 8 years ago

Just so I understand it better embedded etcd will be optional right? As in production we would want to still deploy etcd separate from the API/Scheduler/Controller

smarterclayton commented 8 years ago

Network is the hardest part - you can ignore security and edge cases as long as pods can talk to each other. Leveraging libnetwork seems like a practical choice where possible (or just have a daemonset that drops in your favorite network auto provisioner via CNI). Once the node is started we can run any code.

jbeda commented 8 years ago

@klizhentas Yes! The idea is to make small clusters super easy. Folks looking for large clusters will want to manage etcd independently. Users can choose to take on the complexity of breaking everything out but it'll be an advanced move.

jbeda commented 8 years ago

@smarterclayton I think we just need to pick something to get going. The easiest zero-config option would be the way to go.

derekwaynecarr commented 8 years ago

I think the model we present should not look that different from the production model. I am a fan of making it easy to launch a Kubelet that then has a static manifest with sensible defaults to launch control plane that is not hidden in a mess of salt. On that model, etcd can still be a pod as well as other parts of our control plane.

On Wednesday, June 22, 2016, Joe Beda notifications@github.com wrote:

@smarterclayton https://github.com/smarterclayton I think we just need to pick something to get going. The easiest zero-config option would be the way to go.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/kubernetes/features/issues/11#issuecomment-227897347, or mute the thread https://github.com/notifications/unsubscribe/AF8dbA8VZgUidBorolDZ7ajk8w7C0O6xks5qObecgaJpZM4I8A4G .

klizhentas commented 8 years ago

@jbeda so in the simple case kube-controller-manager, kube-apiserver,kubelet + etcd will be one go binary?

jbeda commented 8 years ago

@klizhentas @derekwaynecarr I don't know what the binaries will be that we ship. I do know that we have to make it dead easy to download a thing and get it up and running. If we can get stuff self hosted on the cluster in a container, that would be a good solution. The number of steps needs to be reduced to ~1 per node.

Let's start with the ideal set of things we want the end user to type. From there we can figure how to get there in a sustainable way (and with the opportunity to do everything in a more explicit way for advanced users).

derekwaynecarr commented 8 years ago

I think we can demonstrate the composable nature of the platform without having to build monolithic binaries that are contrary to the spirit of microservice architecture.

There are two separate but related topics: ability to create a node (and bring up control plane), ability to have a new node join an existing cluster easily. I worry moving to monolithic binaries don't necessarily help either cause.

I also think if we want to advocate being agnostic about a particular container runtime, the setup process should follow suite. This is why I like @mikedanese ideas in the space since they start with the Kubelet (which could work with any container runtime it's pointed against) rather than starting with a particular container runtime launching the Kubelet.

On Wednesday, June 22, 2016, Alexander Klizhentas notifications@github.com wrote:

@jbeda https://github.com/jbeda so in the simple case kube-controller-manager, kube-apiserver,kubelet + etcd will be one go binary?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kubernetes/features/issues/11#issuecomment-227901304, or mute the thread https://github.com/notifications/unsubscribe/AF8dbMrm4XRDpultZxQyXkAXkGxMpg13ks5qObyVgaJpZM4I8A4G .

derekwaynecarr commented 8 years ago

@jbeda - agree on focusing on desired ux command first

On Wednesday, June 22, 2016, Derek Carr decarr@redhat.com wrote:

I think we can demonstrate the composable nature of the platform without having to build monolithic binaries that are contrary to the spirit of microservice architecture.

There are two separate but related topics: ability to create a node (and bring up control plane), ability to have a new node join an existing cluster easily. I worry moving to monolithic binaries don't necessarily help either cause.

I also think if we want to advocate being agnostic about a particular container runtime, the setup process should follow suite. This is why I like @mikedanese ideas in the space since they start with the Kubelet (which could work with any container runtime it's pointed against) rather than starting with a particular container runtime launching the Kubelet.

On Wednesday, June 22, 2016, Alexander Klizhentas < notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

@jbeda https://github.com/jbeda so in the simple case kube-controller-manager, kube-apiserver,kubelet + etcd will be one go binary?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kubernetes/features/issues/11#issuecomment-227901304, or mute the thread https://github.com/notifications/unsubscribe/AF8dbMrm4XRDpultZxQyXkAXkGxMpg13ks5qObyVgaJpZM4I8A4G .

aaronlevy commented 8 years ago

Biased, because I'm already working on it, but I'd advocate for putting effort into self-hosted (k8s-on-k8s) installations as a means of simplifying both cluster creation, and lifecycle management.

If the installation contract becomes "I can run a kubelet" and everything else is built atop that in containers -- then installation criteria could become as simple as "does your node pass node-e2e tests?"

More or less this is already possible in a simple case with all core components being run as static pods. This is how many installations work, and is well understood. The problem with this approach is that it becomes difficult to transition from "this was easy to install" to "I want to customize/modify/update this installation".

As soon as we want to make modifications to this cluster, we're back to some kind of "modify files on disk" configuration management (salt/ansible/chef/etc). In this case it doesn't preclude us from having a "simple" installation and other "production" deployment tools. Or even decide on standardization/contract where a more complex tool can take over from the other static installation (e.g. kube-apiserver.yaml will exist in /etc/kubernetes/manifests)

Alternatively, in the self-hosted scenario, the static installation can remain simple on first install (in concept, replace your static pod definitions with deployments/daemonsets). But then can be modified / extended without relying on external configuration management (or external deployment tooling that needs to evolve in lock-step with your cluster) -- everything is by definition just a kubernetes object.

Updates to the cluster can become api-driven / or even an update-controller application. Assets (tls, configuration, flags) travel with their components as they are also just kubernetes objects (secrets / configMaps). We get all the niceties of kubernetes lifecycle management.

Now all that being said, networking really is a hard part. Maybe this comes down to figuring out the coordination at the kubelet level + cni (e.g. how do I self-host flannel-client + allow it to configure networking for subsequent pods).

klizhentas commented 8 years ago

For this particular feature I think it would help to go backwards - not from implementation to UX, but vise-versa. If we figure out user experience with this right, the implementation will follow.

Here's the ideal scenario that users can see on k8s quickstart page:

Starting single node k8s

wget https://kuberenetes.io/releases/latest/kube
# starts both node, API, etcd, all components really
kube start

This will let users to explore kubernetes, start containers

Adding node

Then there's a use case when users want to get to run smaller clusters to play with failover, HA and so on.

On first node, execute:

# adds provisioning token to securely add new nodes to the cluster
kube token add
<token1>

On any node to be added in the cluster:

kube start --token=<token1> --cluster=https://<ip of the first node>

That's the minimum amount of steps I can imagine to bootstrap the cluster in dev mode. If we figure out this UX first, everything else will follow.

bgrant0607 commented 8 years ago

This has become a discussion issue rather than a tracking issue. It's great that lots of people are interested in this topic. We could use help. I created a github team (sig-cluster-lifecycle) and googlegroup (kubernetes-sig-cluster-lifecycle), which you can request to join. I'm going to rename the sig-install slack channel to sig-cluster-lifecycle. We should brainstorm in the googlegroup rather than generate more github notifications.

Also, a number of people have been working in this area for a while. We're going to summarize the current state and make a prioritized list of work items that have already been identified.

metral commented 8 years ago

I took the liberty of creating a post that summarizes the generic expectations stated here in the googlegroup (kubernetes-sig-cluster-lifecycle) to continue the brainstorming: https://groups.google.com/forum/#!topic/kubernetes-sig-cluster-lifecycle/LRMygt2YNrE

philips commented 8 years ago

On Wed, Jun 22, 2016 at 2:02 PM Tim Hockin notifications@github.com wrote:

Something that crossed my mind with docker 1.12 - built-in kvstoreeans that libnetwork's overlay driver might be viable for us. Having a built-in network mode for Docker installs that works anywhere and doesn't require extra components might be nice.

Relying on libnetwork for bootstrapping sounds like a mess longterm. I would much rather figure out bootstrap, reconfiguration, etc of CNI, which we need to figure out anyways, than make some compromise that puts a new dependency on the Docker engine.

aronchick commented 8 years ago

Reminder - these issues (ideally) should be for only discussion about the flow of the work items related to the feature. If you would like to discuss (please do!) please do it in the Google group.

idvoretskyi commented 8 years ago

Please, take a look at this proposal - https://github.com/kubernetes/kubernetes/issues/27948

jbeda commented 8 years ago

@aronchick @bgrant0607 @metral Agreed -- let's take this to the mailing list.

https://groups.google.com/forum/#!forum/kubernetes-sig-cluster-lifecycle

jimmycuadra commented 8 years ago

@jbeda For your list of related work: https://github.com/InQuicker/kaws

lukemarsden commented 8 years ago

Update on the plan for this feature: https://docs.google.com/presentation/d/17xrFxrTwqrK-MJk0f2XCjfUPagljG7togXHcC39p0sM/edit?ts=57a33e24

lukemarsden commented 8 years ago

Proposal: https://github.com/kubernetes/kubernetes/pull/30360

lukemarsden commented 8 years ago

Update on this: all the dependencies we needed for this landed before the 1.4 feature freeze. We've agreed in the SIG that kubeadm experimental packages can be included in the next few weeks in time for the release (it doesn't need to be in the release branch). So this is on track to be in alpha for 1.4.

See also our WIP demo: https://www.youtube.com/watch?v=Bv3JmHKlA0I&feature=youtu.be

cc @mikedanese @pwittrock

janetkuo commented 8 years ago

@jbeda Are the docs ready? Please update the docs to https://github.com/kubernetes/kubernetes.github.io, and then add PR numbers and have the docs box checked in the issue description

jaredbhatti commented 8 years ago

Ping. Any update on docs?

jaredbhatti commented 8 years ago

Another ping on docs. Any PRs you can point me to?

lukemarsden commented 8 years ago

@janetkuo @jaredbhatti so sorry for the delay replying to this, I was travelling. I'll be working on docs imminently. When do you need them by, at the latest?

devin-donnelly commented 8 years ago

Launch date is Tuesday, 20 September. Ideally we'd like to get them merged in ahead of time, say by Friday 16 September if possible.

jimmycuadra commented 8 years ago

Docs should ideally include clarification on the relationship between the various cluster deployment solutions, because it's getting very confusing for users as to why there are so many and which one is "official" and/or "recommended," especially since there are many under the kubernetes organization. Specifically:

idvoretskyi commented 8 years ago

@lukasredynk @jbeda yet another ping on docs.

mikedanese commented 8 years ago

The docs PR is out for review https://github.com/kubernetes/kubernetes.github.io/pull/1265

devin-donnelly commented 8 years ago

Thanks. I've got it in tracking now.

On Wed, Sep 21, 2016 at 3:36 PM, Mike Danese notifications@github.com wrote:

The docs PR is out for review kubernetes/kubernetes.github.io#1265 https://github.com/kubernetes/kubernetes.github.io/pull/1265

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kubernetes/features/issues/11#issuecomment-248762839, or mute the thread https://github.com/notifications/unsubscribe-auth/ARmNwEuVjs5RvAnhKKVyrkeZcFj5X9FFks5qsbF0gaJpZM4I8A4G .

bogdando commented 7 years ago

The subfeature request https://github.com/kubernetes/features/issues/138

idvoretskyi commented 7 years ago

@jbeda @mikedanese can you confirm that this item targets beta in 1.6?

mikedanese commented 7 years ago

Yes it does.

bogdando commented 7 years ago

@mikedanese could you please link the proposal with the amended https://github.com/kubernetes/kubeadm/issues/22 feature flags proposal? I'm lost in tracking its status

jbeda commented 7 years ago

Updated the checkboxes above. This is largely implemented for 1.6. The last bugs and and UX issues are getting ironed out now.

The biggest missing thing now for beta for this is documentation. Much of the UX here is hidden from users but should still be documented.

idvoretskyi commented 7 years ago

@jbeda please, provide us with the release notes at https://docs.google.com/spreadsheets/d/1nspIeRVNjAQHRslHQD1-6gPv99OcYZLMezrBe3Pfhhg/edit#gid=0

Also, please, select the valid checkpoints at the Progress Tracker.

jbeda commented 7 years ago

@lukemarsden Is pulling together release notes.

@luxas -- do you know if anyone is on the line for doc updates for kubeadm for 1.6? If not, I can probably take a pass.

jbeda commented 7 years ago

Doc update for Beta: https://github.com/kubernetes/kubernetes.github.io/pull/2829

luxas commented 7 years ago

This stays in beta for the v1.8 release. There are some dependent work we are pushing instead (Bootstrap Tokens and easy Upgrades for instance). Once we have all dependencies on beta level or higher, we can push this last overall piece (kubeadm) to GA

idvoretskyi commented 6 years ago

@roberthbailey @luxas @jbeda @mikedanese is it still beta for 1.9?

Also, please update the feature description using https://github.com/kubernetes/features/blob/master/ISSUE_TEMPLATE.md

roberthbailey commented 6 years ago

kubeadm will still be beta in 1.9.

zacharysarah commented 6 years ago

@jbeda :wave: Please indicate in the 1.9 feature tracking board whether this feature needs documentation. If yes, please open a PR and add a link to the tracking spreadsheet. Thanks in advance!

zacharysarah commented 6 years ago

@jbeda Bump for docs ☝️

/cc @idvoretskyi