Proposal: Switch from shared to standalone model

[enisoc]

This is a proposal to make a fundamental change to the user experience of Metacontroller as part of going from alpha to beta. I'm posting this to try to get feedback from users and contributors as to whether this is a good idea, as well as ideas for what the end result should look like.

The proposed change is to switch from a shared service (one Metacontroller server for the whole cluster) to a non-shared, "standalone" model (run your own Metacontroller server for each controller or set of related controllers).

Background

Metacontroller was originally conceived as a shared service for two main reasons:

Lower barrier to entry

My thinking was that, once you have Metacontroller installed (which someday your cluster admin might have already done for you), the fact that it's a shared service would create a very low barrier for you to add a new controller. You could just submit a new CompositeController object to the API server.

However, it turns out this causes a different barrier to appear. Since Metacontroller is shared, you have to worry about other users of the cluster when you decide or request your cluster admin to install, upgrade, or reconfigure the Metacontroller server. In the proposed standalone model, you just run your own Metacontroller server scoped to only your controllers, so you don't have to worry about what anyone else is doing in the cluster.

In addition, the ability to simply submit a CompositeController turned out to not provide much benefit. I originally envisioned that webhooks for use with Metacontroller would most commonly be served through FaaS frameworks, meaning the hook author wouldn't need to run an actual webserver. In that case, it would save effort to let them avoid having to run the Metacontroller server. Instead, it seems that most webhooks are hosted as standalone Pods, which means there's not much additional effort required to add a Metacontroller server as a sidecar container to that Pod, configured to talk to the webhook over localhost.

This proposal doesn't preclude running webhooks in FaaS frameworks, either. You could simply run a standalone Metacontroller server that's configured to call out to a URL that happens to be served by the FaaS framework rather than a container within the same Pod.

Scalability

My other reason for making Metacontroller a shared service was to be able to share informers (in-memory caches of k8s API objects) across controllers. Since these caches are in-memory, each separate process that needs one adds additional load on the API server, as the informer subscribes to a watch stream for every resource it caches. To mitigate this, all core k8s controllers share informers by living in one process (kube-controller-manager). My thought was to use this model for custom controllers as well, ensuring that the maximum additional watch load on the API server would be the same as adding just one additional kube-controller-manager process, no matter how many custom controllers you add.

Given the success people have been having with controllers hosted in individual processes with their own in-memory caches and their own watch streams, it seems that my concern was not justified. This proposal doesn't preclude sharing informers across controllers, either. You can still group controllers at whatever level of granularity you want, and have them all hosted by one Metacontroller server.

Proposed Changes

What would change

• No more CompositeController or DecoratorController CRDs.
  • The config will be in the same format, but not stored in CRDs because it only applies to one particular Metacontroller process.
  • One option is to have Metacontroller load these as YAML files from a directory, which could be a mounted ConfigMap/Secret. Updating the ConfigMap would cause the Metacontroller server to reload the files and restart controllers with the new config.
• We will drop the plan for Metacontroller to support per-controller service accounts.
  • You should run a separate Metacontroller Pod as each service account instead.

What would NOT change

• Capabilities of controllers hosted by Metacontroller will not change.
  • A controller could still reach out and process objects at the cluster level, or in namespaces other than the one in which the controller Pod is running.
• The fields inside APIs that configure how hosted controllers behave (CompositeController, DecoratorController) will not change.
  • Only the place where this configuration is stored will change (no longer in a cluster-wide CRD).
• The API for hook requests/responses will not change.

Impact

This section summarizes the effects of the proposed changes.

Benefits

• Metacontroller will stop being a huge privilege escalator. Run your own Metacontroller under your own service account, which has only the permissions needed by your controller(s).
• End users of controllers developed with Metacontroller will no longer need to install Metacontroller; there will be no such thing as "installing" Metacontroller. They will just install the desired controller, which happens to use Metacontroller under the hood.
• The mental model of putting together your hook code and a Metacontroller sidecar matches that of Go frameworks like kubebuilder. From the outside, a Pod with a webhook+metacontroller is indistinguishable from a single Go binary built with kubebuilder, so you can deploy either in the same way.
• Debugging the parts of your controller that live inside Metacontroller will be easier since it runs as you (instead of under a cluster-level account), often in the same Pod as your webhook container.
• Securing the metacontroller<->webhook connection can be much easier (don't need mTLS) if you run it as a sidecar, since you can restrict communication to localhost.

Costs

• All current users of Metacontroller alpha will need to refactor their deployments in order to upgrade to beta or beyond.
• Anyone who would have preferred to keep Metacontroller as a shared, cluster-wide service will have to switch to the standalone/sidecar model instead.

Why not both?

It's conceivable that we could support both cluster-wide CRD mode and standalone config file mode. However, that would significantly increase future maintenance costs and slow down progress towards beta. My perception is that standalone mode will work better for all the use cases I've seen so far in the wild. If that's wrong, please leave a comment describing why cluster-wide mode is important to you.

[phs]

I found this issue while reading metacontroller's source to see if I could do just as you are proposing: run several, limited metacontroller instances (one per desired controller).

As you anticipated, my concerns with the shared model surround security requirements. We won't run a single, very powerful controller that acts on behalf of arbitrary web hooks. Instead we want to enumerate the powers required by each controller on a per-case basis and add them on the default service account in a namespace dedicated to that controller. Our current use of metacontroller (as part of our evaluation) limits its powers to merely creating jobs in certain namespaces, with the hope that those jobs (using their local default service accounts) could hold their required permissions instead of metacontroller. Of course since RBAC won't let us restrict the images used in created jobs, metacontroller still retains the ability to start arbitrary pods in many namespaces and so is too powerful.

Using a web hook to communicate between side cars is a reasonable way for us to dictate policy to the generic controller implementation. While evaluating metacontroller we were concerned about controlling the possible targets of the web hook; fixing the web hook at controller deploy time (or otherwise locking it to pod localhost) would be welcome. I would also be just as happy mounting a binary that we supply to a well-known location in the metacontroller pod and having it call us to speak json over stdin/stdout. For us metacontroller would ideally be a framework, though I appreciate the effort to remain language agnostic.

[moredure]

So, new api will looks like configuration file per statefulset of metacontroller?

[enisoc]

@mikefaraponov wrote:

So, new api will looks like configuration file per statefulset of metacontroller?

Yes. Also with the option to keep controller configs in multiple files and specify them as command-line args to metacontroller.

[nikhilk]

In our use case we have 1 top-level CRD + 3 implementation detail CRDs. We would want a single metacontroller deployment to handle all these, and avoid the cost of running multiple deployments of metacontroller.

Update on re-read: If there is a way to accomplish the same, i.e. single metacontroller deployment that reads CRDs/controllers to handle via mounted configuration, rather than a set of CRDs that is fine.

FWIW, for us, it doesn't matter if metacontroller needs additional permissions, as we are running a single service workload per GKE cluster.

[willnewby]

My use case involves Metacontroller managing external resources, so this is actually much better of a pattern for me (primarily due to the least-privilege changes).

As long as controllers are represented in a way that's still easy-to-understand (despite the CRDs seemingly going away), I'm all for it.

👍

[enisoc]

@willnewby wrote:

As long as controllers are represented in a way that's still easy-to-understand (despite the CRDs seemingly going away)

The current plan is to keep the configuration format the same, meaning you can just take the same CompositeController or DecoratorController spec you have and dump it into a file/ConfigMap as JSON or YAML.

[luisdavim]

For me, the shared informers is one of the reasons for choosing metacontroller

[enisoc]

@luisdavim Do you mean it's important that informers are shared across all metacontroller-hosted controllers installed by any user? Or would it be enough that all of your own controllers share informers?

[luisdavim]

Per user should be enough, as long as one instance of the metacontoller managing multiple composite and/or decorator controllers.

[reegnz]

+1 from a security pov I prefer standalone controllers over shared ones. For me metacontrollers biggest value is not the shared cache but it's high-level controller semantics and webhook-style behaviour.

[dirty49374]

That's exactly what I want !!!.

[bsctl]

Awesome! +1

[jetztgradnet]

I very much like this new approach as it keeps the metacontroller an implementation detail of a custom controller when used in a common pod with the controller logic (metacontroller as sidecar). If one was ever to change the controller implementation to a fully coded custom controller, one would still simply deploy the (updated) pod/deployment only without the sidecar.

Also, "local" privileges are much better then the "global" ones.

Any idea when this beta release is going to happen?

[jetztgradnet]

Also, are there any plans to propose this model for inclusion into K8s core? This is a low-barrier approach compared to write a fully custom controller similar to the admission web hooks.

[floriankoch]

@enisoc i like the idea of a shared matacontroller - but i understand the arguments against.

With the proposed solution , what is the default way to run a controller? Run the metacontroller in a container, and the webhook in another container in the same pod?