look into self-hosted kubelet

[janwillies]

The system-kubelet is currently used as the main kubelet. For a true self-hosted platform, the kubelet should be self-hosted as well. The system-kubelet should only take over when things are falling apart. See the design document: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/self-hosted-kubernetes.md

This can be accomplished when running bootkube with --self-host-kubelet

Related:

• https://github.com/kubernetes-incubator/bootkube/issues/275
• https://github.com/kubernetes-incubator/bootkube/issues/96

[alanmeadows]

I'm thinking we punt on this for now until it becomes "well tested"

It seems risky and we have many other moving parts. Also, removing the kubelet from the bucket of "system things" doesn't empty the bucket. We still have docker and the kernel so the advantages are small.

OpenStack has a mandate to support a seamless upgrade and side-by-side running of N-1 and N to support upgrading the core components followed by agents on what may be thousands of hosts. What is the equivalent Kubernetes claim to support upgrading Kubernetes core components, followed by rolling kubelet updates? At what point is the gap too large?

[janwillies]

The idea here could be to update the self-hosted kubelet via kubernetes primitives, and maybe even update the system-kubelet via this mechanism. The gap for the system-kubelet shouldn't get too wide, that's for sure.

In the end this doesn't free us from providing a path for infrastructure updates, but what's the majority of these updates? Is it kernel, os, libs, docker, kubelet? It could be a benefit to have a "speedway" to rollout kubelet updates.

[stealthybox]

Love the conversation -- The containerized kubelet is definitely maturing.

Base images like CoreOS, Ubuntu Core, Project Atomic, Photon, and ContainerOS manage those 'system level' updates for you automatically.

The reason CoreOS encourages HA clusters is because any 1 node is supposed to be able to go down.

CoreOS's and ContainerOS do an active-passive upgrade for the kernel & etc. Atomic uses rpm-os-tree which is sort of git-like.

Do you guys know anybody who has traction using this stuff?

[v1k0d3n]

@stealthybox we've actually looked into CoreOS and Atomic, and they are considerations if/when we can get past the dependency for OS-level requirements for Ceph (this is a requirement for our openstack-helm repository). CoreOS' locksmith+etcd feature is extremely useful for auto-draining hosts during kernel-level upgrades, etc. i'm not sure I understand your question though on "who has traction on using" these things. for our part, this is part of our internal roadmap, but we are working directly with those vendors and the community to gauge and document these general use-cases. docs are going to be spread across several repos within the https://www.github.com/att-comdev namespace. but again, these are all side/parallel tasks in an effort to build a stable underlay for our much larger effort: openstack-helm. this repo just happened to be an open PoC, so community discussions and input are very welcome. :+1: