Explore: CRDs for orgs and spaces

[gcapizzi]

Background

We have drafted a proposal to support CF orgs and spaces across multiple clusters. The current proposals is based on an Org CRD, which would include a list of Kubernetes namespaces belonging to it.

After discussing this with the broader team, the idea of having a Space CRD pointing to its parent Org came up.

Questions

• Let's explore different ways of designing CRDs to solve the org/space problem, discuss pros and cons and add our conclusions to the document.
• Security-wise, how do we make sure the credentials used by the orgs/spaces controllers are safe?

[kieron-dev]

We noted that the Miro board shows an 'org' resource, and possible 'space' resource.

We realised that the prototype did not need to setup Dex to work. Users supplied by Dex are no different to impersonated users supplied by an admin user (kubectl --as ). So we ripped out the dex setup, simplifying the demo.

The prototype did not support space roles, so we looked at extending it to support roles at both org and space level.

Initially, we intended to have separate org and space resources. This felt a bit unbalanced with the org reconciler doing not a lot apart from having to duplicate org user role information onto the space resources. Then the space reconciler would do the bulk of the namespace and rolebinding work.

So instead, we looked at including the space definition in the org resource. This seems a reasonable thing to do. See commit.

We realised that space names cannot be used directly as namespace names, as two orgs can have the same space name. So we are currently prefixing the namespace name with the org name. This could still be problem with orgs/spaces: org-a/space, org/a-space. So GUIDs might be required.

We also noted a potential requirement for a global object where we can add users for the global roles of admin, admin-readonly and global-auditor. We could use a global-roles CRD with lists for the 3 global roles. The reconciler would ensure all the Orgs have the global users with their roles added at the org level, and then the org reconciler would take over to created the required role bindings. This way we avoid ClusterRoleBindings which might give permissions on non-CF namespaces, particular on things like secrets which are not CF-only custom resources.

We would still like to modify the org reconciler to handle updates, so it can keep namespaces in sync, and also role bindings in sync after updates. At the moment, it only really supports create and delete actions. We should also experiment with the global role CRD and controller.

[danail-branekov]

While looking for ways to securely store cluster configurations we asked ourselves how we should react in the event of configuration credentials expiry. We thought that we need a cross-cluster service accounts.

We looked for inspiration in the admiralty multicluster service account but the project seems abandoned as the installation failed with certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0. Disappointing...

We also stumbled upon this article discussing two implementations of cluster federation. And while reading it, we realised that our current proposal does not deal with scheduling, instead it relies on a static org-to-cluster allocation (via the Org CRD spec). The CF admin would have to decide how to allocate cluster resources.

In an ideal world, CF users should see a CF deployment as a single entity, no matter how many k8s clusters it runs on. This kind of sounds like we need a federated cluster but then kubectl users would be left stranded (in federated clusters they would have to deal with federated proxy objects and even dedicated federation CLI)

Maybe we should abandon the idea that kubectl users would be able to still use kubectl in both single-cluster and multi-cluster deployments:

• If you are brave enough to use kubectl on a single k8s cluster, you should be as brave as using kubefedctl on federated clusters and be aware of the federated proxy objects.
• If you are happy to use a single-cluster deployment, then using just kubectl is fine.
• The third option would be to have lots of CF deployments, each of them in a dedicated cluster.

We could make the decision of single vs federated cluster to be a matter of a deployment option. However, in case the deployer wants to use a federated cluster, we should probably come up with a "federated" eirini controller that creates "federated" statefulsets instead of regular ones.

Another benefit of using a federated cluster hopefully might be that it solves the problem of securely storing remote cluster configs, credentials expiry and so on.

TL;DR we should probably have an explore on federated clusters and see whether we could make use of them.

[danail-branekov]

Coming back to our prototype, a viable direction to take is to replace the admin-credentialed kubeconfig that we use as remote cluster config with service account secrets that we get from each workloads cluster. In more details:

• Create a service account on every workloads cluster (it has a secret with a token/certificate/whatever bound)
• Copy the service account tokens into the "system" cluster secrets
• When creating the clients for the remote cluster, somehow use the service account credentials

Maybe we could get inspired from admiralty, e.g.: https://github.com/admiraltyio/multicluster-service-account/blob/a5a9073c13562189e695650e819c6c9c3a1cc531/pkg/importer/importer.go#L58