search

kubernetes-retired / kubernetes-anywhere

[EOL] {concise,reliable,cross-platform} turnup of Kubernetes clusters
Apache License 2.0
532 stars 196 forks source link

design and implement phase2 and phase3 handoff (a.k.a. deploy addons) #126

Closed mikedanese closed 6 years ago

mikedanese commented 8 years ago

Right now addon configuration can be generated from the cluster config object. We need a way to run it automatically after phase2 on the master completes. kube-up uses the kube-addon-manager.sh which is a mess which we should avoid. Open questions:

errordeveloper commented 8 years ago

Where are templates expanded?

I wonder whether this is really needed.

Looking at SkyDNS right now, it has the following:

  1. DNS_REPLICAS and DNS_SERVER_IP, which are attributes in a config file
  2. DNS_DOMAIN, which is passed as an argument

I am pretty sure that a config map can be used for 2, not quite sure about 1.

I'd guess a simple and flexible approach would be to use a git repo, and make updates optional through git mechanics. User could override the repo location, if they really have to set different parameters to add-ons they can fork and edit. However, I'm only familiar with SkyDNS add-on, and the DNS_SERVER_IP is something that I've had to care about, as the default (10.0.0.10), is too vulnerable to clash, so I'd rather have a separate discussion of that...

Do we support upgrades? If so how?

As I suggested, using a git repo may be a simple way to start. I've also whether user might want to freeze certain addons, and in that case they would just have to make sure to not commit any changes... Well, this now becomes a little separate project, some folks might like using such a thing for doing simple CI/CD stuff. Also, would be good to check the status of updates for daemon sets, last time I checked, it didn't work yet, although it does't sound too complicated to handle daemon sets in different fashion. I'm happy to think of a simple design/prototype for this, if the general idea of using git sounds plausible.

Is this a continuously running and reconciling process or does it only run once?

With the approach described above, it would have to run constantly or, at least periodically.

I am also thinking that some user might prefer to perform explicit updates to the set of addons, which may be a kind of thing they'd do with Ansible (or other such thing)... That's just for the record. I'd go with a single easy solution for the start.

mikedanese commented 8 years ago

It would be great if the configuration required was minimal enough to not require templating. I think for the MVP, we need to be able to apply addons to the control-plane once (probably at the end of the deployment process). Versioning through git also sounds good. We should investigate https://github.com/kubernetes/helm and see if it has the capabilities we require from an addon applier. It would be great not to have to roll our own. Explicit updates sound good for the initial implementation.

errordeveloper commented 8 years ago

Yes, I was wondering if Helm would be appropriate to provide out of the box.

errordeveloper commented 8 years ago

See https://github.com/kubernetes/helm/issues/838#issuecomment-227484145 for details of my investigation so far, basically I didn't get very far with it. Hacking a PoC version of Helm that would work with kube-system namespaces seems doable, but let's see if Helm maintainers are planning to work on this any time soon.

errordeveloper commented 8 years ago

Ok, actually https://github.com/kubernetes/helm/pull/852 fixes the issue, I'll carry on trying out what it takes to actually use Helm for managing addons.

errordeveloper commented 8 years ago

Where are templates expanded? On the local machine or the master?

Helm solves templating question, and I'd be happy convert addons to charts with the obvious parameters (ones that are currently substitute by kube-up.sh and image versions that are currently bumped manually).

I've converted dashboard addon to illustrate how this might look like.

Do we support upgrades? If so how?

Helm currently doesn't have an upgrade mechanism https://github.com/kubernetes/helm/issues/690, but we haven't decided if that is required, so may be it's okay to wait for it to appear in Helm eventually, as it's most likely will.

Is this a continuously running and reconciling process or does it only run once?

Regarding this item, I suppose Helm would provide enough flexibility to how addons are managed, except for upgrades being an outstanding feature...

errordeveloper commented 8 years ago

@mikedanese good catch on the title, I though something was odd with it 👍

mikedanese commented 8 years ago

Thanks for exploring this. In your opinion is it to early to move to helm or should we go for it? Would it be easy to selectively enable addons? https://github.com/kubernetes/kubernetes-anywhere/blob/master/phase3/Kconfig

errordeveloper commented 8 years ago

I haven't used Helm super extensively to judge how well it works in the long term, but appears to be fairly simple right now. The Helm team are on this usecase, so that's good. Regarding selecting add-ons, we can do a simple loop over selected ones in Ansible, and Helm would take care of where the definitions are located and do the template rendering in a non-adhoc fashion. I am not hugely impressed with usability of Go templates, but it has a linter that can be improved.

On Wed, 22 Jun 2016, 19:41 Mike Danese, notifications@github.com wrote:

Thanks for exploring this. In your opinion is it to early to move to helm or should we go for it? Would it be easy to selectively enable addons? https://github.com/kubernetes/kubernetes-anywhere/blob/master/phase3/Kconfig

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kubernetes/kubernetes-anywhere/issues/126#issuecomment-227838671, or mute the thread https://github.com/notifications/unsubscribe/AAPWS0EPh_3wgXwrXJ8LQ7yUeb8Tj7_Dks5qOYHOgaJpZM4IzX_Z .

fejta-bot commented 6 years ago

Issues go stale after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta. /lifecycle stale

fejta-bot commented 6 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta. /lifecycle rotten /remove-lifecycle stale

errordeveloper commented 6 years ago

/close