High Level CAPN Goals

[christopherhein]

Prelim Goals:

• To configure new resource types for nested pod based control planes
• To enable declarative orchestrated control plane upgrades (minus etcd)
• To enable independent scaling of each of the control plane components
• To enable configuring the control plane components
• To enable the use of custom implementations for all components

Prelim Non-Goals:

• To manage etcd, it's expected that etcd can be run separately with an integration point
• To provide CNI configuration this is left to the management or super cluster

[christopherhein]

@salaxander @Fei-Guo @weiling61 @charleszheng44

[christopherhein]

Please add more Goals/Non-Goals for CAPN

[charleszheng44]

Though manging Etcd is not our goal, to increase the development velocity, can we implement a basic NestedEtcd controller that creates in-mem Etcds for NestedControlPlanes? In the future, we can make CAPN compatible to some third-party Etcd-operators.

[enxebre]

Some high level goals I'd like to include:

• To reinforce responsibility boundaries between cluster provider and cluster consumer / Service plane vs data plane.
• To shift content out of consumer-visible (kubeconfig) cluster data plane to service control plane. Externalizes etcd, kube-{apiserver|controller}, scheduler, auth, identity to management cluster.
• To reduce cost per cluster by deploying control planes as pods to the management cluster rather than target traditional virtual/physical/infrastructure hosts.
• To enable heterogeneous architectures for data plane compute nodes. E.g Virtual nodes (as in https://github.com/kubernetes-sigs/multi-tenancy/tree/master/incubator/virtualcluster) vs virtual/physical/infrastructure hosts.

cc @ironcladlou @csrwng

[christopherhein]

Though manging Etcd is not our goal, to increase the development velocity, can we implement a basic NestedEtcd controller that creates in-mem Etcds for NestedControlPlanes? In the future, we can make CAPN compatible to some third-party Etcd-operators.

Yes, agree @charleszheng44.

[christopherhein]

@enxebre these are great and I agree should be added.

One thing, with "reduce cost per cluster"? I'm concerned that might be all dependent on where the control planes are provided. At least w.r.t physical "$". Am I misinterpreting what you mean?

[christopherhein]

/kind design

[kfox1111]

I wonder if this helm chart might help with an initial bootstrapping implementation: https://github.com/kvaps/kubernetes-in-kubernetes/tree/master/deploy/helm/kubernetes

It is pretty easy to map crd values to a helm chart values file and deploy that way. This can allow iterating with the api to get it right much more quickly then trying to write it from scratch in something like go. Once the api is nailed down, then it can potentially be replaced by a go operator that does the templating itself?

[christopherhein]

Thanks @kfox1111, yeah. I was taking a look at that recently as it was shared on the SCL mailing list. Looks like a neat implementation. I like that it's orchestrated with kubeadm at parts even though the templating is done upfront.

We currently have a similar approach which is backed by a controller that we're in essence reimplementing the API in this project. https://github.com/kubernetes-sigs/multi-tenancy/tree/master/incubator/virtualcluster

Probably useful to share things between these two projects.

[christopherhein]

/milestone v0.1.0

[christopherhein]

/close

[k8s-ci-robot]

@christopherhein: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-nested/issues/4#issuecomment-805079046): >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.