Allow modular addons

[bergerx]

Currently there are some pre-defined addons in gardener that are used when creating a shoot.

We want to provision some resources as part of our shoot creation process. I'm curious if it could be possible to be able to extend the addons provided out of the box in different gardener installations?

I'm not sure if this is easy to implement, but it makes sense to be able to allow our users pick some addons among our internal set of addons when provisioning shoots. I think this worth discussing maybe we can follow up with a design doc.

[rfranzke]

/cc @vasu1124

[bergerx]

A recent discussion in slack can be found here: https://kubernetes.slack.com/archives/CB57N0BFG/p1530630996000284

Summary:

@bergerx:

So, if user of a Shoot deploys all the pods with best effort QoS (no request) there wont be any auto-scale actions taken on that cluster.

@mvladev:

I believe this is the case. In most cases you need to set https://kubernetes.io/docs/tasks/administer-cluster/manage-resources/memory-default-namespace/ If a pod cannot be scheduled -> the the cluster autoscaller can react

@bergerx:

But since we provide clusters as they are, our consumers are free to use LimitRanges or not. We’d like to create some pre-set resources in the clusters we provide. In our case, most of our organisation is at the very beginning with k8s adoption, if we tell them about LimitRanges and ResourceQuotas, it will just cause frustration, but in the meantime we still want to get advantage of autoscale and dont want our users to mess with scaling when they start. We thought about providing a pre-deployed environment with some basic apps, pod-presets, LimitRanges, ResouceQuotas and also app-level metric collection (prometheus operator) and app-level central logging.

@mvladev:

In general we want to get rid of the .spec.addons section of the Shoot specification. We want to make it more generic. In fact the Shoot resource IMHO should be changed to make the whole API more pluggable - right now if you want to add new provider you have to fork - which is bad.

[vlerenc]

Exactly, we would rather not like to extend the add-on concept in-tree (similar request showing the effort if we continue the in-tree approach is #272). This would not be sustainable (which is why we didn't add any other add-ons anymore) and we need another approach here, if anything at all. Meaning, most cluster installers focus on the cluster installation itself - content can be added independently and the field is moving fast.

An extension concept could also be realised via an independent controller, e.g. deployed into the garden (or seed) cluster watching shoot resources (or namespaces) and acting upon them. This way we would decouple the logic and not turn the Gardener into an unmanageable beast that does too much. That is also @mvladev point for long and in regards to add-ons I fully agree, that some better way must be found.

[vlerenc]

See also #269 that asks for better configurability of the ingress controller. Out of band, but the same add-on, users have asked for multiple replicas for the ingress controller. These are requests that illustrate how difficult the add-on concept is right now. If we provide them, we need to provide a lot more than we have today. So either we go this path, then we need to improve the concept or we drop add-on altogether.

[bergerx]

@mvladev mentioned using helm-crd but it's still experimental. I briefly did a research if there is an existing good-enough solution for this specific problem, but seems like it's being discussed all over the place and no foreseeable mainstream solution yet. I found this thread the closest possible with Helm, and seem like no one knows where to go: https://groups.google.com/forum/#!topic/kubernetes-sig-apps/nHk_Z-zb_fg

So seeing the landscape I do also agree to leave the addons out of Gardener's scope until there is a clear winner, and let individuals solve the problem. But even though Gardener doesn't address this problem, It would make sense to host an example solution under the Gardener GitHub organization, but given the fuzzy environment, I'm not sure even about that.

[luisdavim]

What about a way to reference a list of charts to be deployed in the shoot/seed definitions and then the controller would just loop through and install them... This would enable out-of-the-tree addons and could be implemented as a separate controller watching the shoots/seeds resources.

[luisdavim]

just looked at helm-crd mentioned by @bergerx and @mvladev and that seams like an interesting alternative.

[luisdavim]

https://github.com/weaveworks/flux could also be an alternative

[bergerx]

From https://github.com/gardener/bouquet:

Bouquet is a draft addon manager for the Gardener. It incorporates some of the requested features of the community but not yet all of them. Caution: This software is early alpha. It is not meant for production use and shall (currently) only serve as a possible outlook of what is possible with pre-deployed software on Gardener Kubernetes clusters.

[luisdavim]

bouquet seems interesting but using configmaps instead of helm charts might be limiting.

[rfranzke]

We won't work on this one in the foreseeable future, hence, closing it for now.