georgethebeatle
commented
3 years ago As stated in the issue the 1:1 org to cluster mapping might not be the best option for several reasons
- Org size: An org can too small to have a dedicated cluster, or too large to fit in a single cluster
- Creating org implies creating a new cluster which is a complex thing for CF-on-k8s to do
- Users might want to put a cf instance on a cluster they are already using for other purposes as well. In this case the cf will take control of the whole cluster potentially messing up current RBAC rules. This will make cf look less native.
- Some of our stakeholders have a requirement of running multitenant workloads on the same cluster
- Some of our stakeholders have a requirement of running cf on multiple clusters since clusters don't scale well with too many nodes
In the CF model an org is a collection of spaces, and currently spaces map to k8s namespaces, so in order to address the points above we are proposing a new Org CR that declares the following:
- A list of namespaces that the org consists of grouped by the cluster they belong to
- A list of users that are member of the org being defined along with their roles
This Org CR is going to be reconciled by an orgs controller into namespaces and role bindings. This controller holds multiple kubeconfigs of a number of clusters that can be used to acommodate orgs.
This design makes org-to cluster mapping more transparent and allows for several deployment options:
- A single "system" cluster that holds the org definitions and orchestrates them on a set of "workload" clusters
- A single cluster holding both the org definitions as well as the workload namespaces
- Multi cluster deployment where one of the workload clusters is used to hold the org definitions
We have a working prototype in our explorations repo. It creates two kind clusters, configured to work with dex as OIDC provider (dex is running on the first cluster, tokens are valid on both).
In k8s there is no stored User object. Users are coming from the tokens issued by the IDP and are valid on any cluster configured to trust that IDP. This means that we can make a user an org member by creating an RBAC RoleBinding to a well-know ClusterRole that models one of the CF roles (i.e. space developer)
Background
In https://github.com/cloudfoundry/cf-crd-explorations/issues/35 we tried to map CF roles and permissions to RBAC rules on the current proposed CRDs, assuming CF spaces would map to Kubernetes namespaces and CF orgs would map to Kubernetes clusters.
While the Space-Namespace mapping seems to fit quite well, the Org-Cluster mapping might be problematic, as CF orgs can be both much smaller in scope than a Kubernetes cluster, or much larger! Some users might want small orgs that would easily fit in a single cluster, while some others might want very large orgs that span across multiple clusters (as Diego clusters can get much larger than the maximum size of a Kubernetes cluster).
On the other hand, the Org-Cluster mapping works quite well from an RBAC perspective, as the only types of roles available on Kubernetes are scoped to either namespaces (
Role) or clusters (ClusterRole).Question
What would happen to our proposed RBAC roles if we removed the Org-Cluster mapping assumption? Can we come up with different ways of implementing CF orgs on Kubernetes that could still work well with RBAC?