Multi-cluster support

[dflemstr]

Describe your problem

At Spotify, we have a use-case where we're quickly outgrowing the capacity of a single Kubernetes cluster. For more details about the rationale, see the internal issue (202701502).

We are considering sharding our control plane across multiple Kubernetes clusters, on a per-namespace basis. The sharding strategy itself is probably something we want to retain control of ourselves and could solve using an existing kpt extension point, e.g. a kpt fn.

We control how kpt gets invoked through our custom CI/CD system. If we consider it to be in scope of kpt to talk to multiple clusters, we would like to see that feature added. If not, we'd need to resort to something like invoking kpt several times with different kubectxs, but this means it'll be less likely in the long run that we'll be able to expose kpt directly to our customers.

It gets tricky to figure out how to deal with multiple contexts that map to different clusters and the authentication lifecycle of those contexts. We could probably get away with an approach where handling contexts stays out of scope of kpt, and we're rather concerned with sharding resources by routing them to the right context per KRM resource.

As a strawman suggestion (not at all optimal, but just as something to convey the idea), I could imagine there being something that configures kpt for how to use multiple contexts, that doesn't require a lot of per invocation overhead (like passing lots of extra CLI flags...), e.g. a config file (maybe in ~/.kube/kpt-config or whatever):

apiVersion: kpt.../v1
kind: Config
contexts:
  - context: foo_bar_baz_00
    resourceSelector:
      spotify.com/cluster-shard: 00
  - context: foo_bar_baz_01
    resourceSelector:
      spotify.com/cluster-shard: 01

[droot]

@dflemstr multi cluster support requests have come up in the past but I don't think we have made serious progress on this front.

/cc @mortent @karlkfi who might be familiar with the prior art in the k8s eco-system.