[Enhancement Proposal][RFD] Integrating with Hierarchical Namespace Controller / Matching Namespace by label

[BronzeDeer]

Summary

Provide label matching for namespaces in applications.namespaces, AppProject.spec.sourceNamespaces and AppProject.spec.destinations[].namespaces.

Primarily this will allow us to match on the tree-labels of the hierarchical-namespace-controller (HNC), which allows for fuller, but secure self-service around namespaces for empowered developers. It would also expressing the permission to deploy apps in certain additional namespace in a more visible and coherent way that maintains clean and automatically right-scoped RBAC, reducing load on argoCD and cluster operators

Motivation

Currently, as part of the apps-in-any-namespace pattern used to empower individual developer teams to work with minimal or no reliance on a cluster/argo-admin, we are relying on name-globbing to provide dynamism for matching future namespaces, which do not exist at time of configuration. This minimizes the issue where on-boarding of new teams requires an ArgoCD Admin to manually expand the list in the configuration of the argoCD controllers, and potentially creating new role bindings as well. Similarly, if a team wants to deploy to a new namespace (either for internal grouping, or security boundaries, or ephemeral environments), a platform operator has to create said namespace and then manually update the destinations defined in the AppProject. If certain common namespaces are requested and later deleted to remove the team's permissions, manually tracking that change back into relevant AppProjects is required to not leave a dangling permission that might allow the team to intentionally or unintentionally breach into a separate team's namespace if they later start using said commonly named namespace. Globbing allows us to minimize this churn by creating an implied hierarchy through a system of nested prefixes like "argocd-apps-", and "argocd-apps-team-a-", however this hierarchy is not represented in Kubernetes anywhere and relies on out-of-band transmission of information ("Oh, that didn't work, because you didn't prefix your namespace correctly, it's in the docs somewhere)

Allowing namespaces to be matched by label would allow for a much clearer association of the semantic connection between "ArgoCD will read Applications from here", "This AppProject can deploy applications here and have them picked up by ArgoCD" and "ArgoCD/This appProject can deploy here". and a namespace. It also avoids over-matching potentially unintended namespaces through careless globbing.

However, this concept becomes most powerful for developer self-service, when combined with the Hierarchical Namespace Controller (HNC), which allows certain empowered users to create a hierarchy of sub-namespaces in a namespace, without ever breaching the Kubernetes isolation boundary enforced by namespaces.

An ArgoCD + HNC setup could even minimize the permission that need to be granted to ArgoCD, which normally needs to run in cluster-mode to allow for management of not yet created namespaces. Such a setup could look like the following:

Example ArgoCD + HNC Setup

There are 4 relevant cluster roles, mainly used for namespaced rolebindings:

• argocd-manage-namespace (This can be represented by the default cluster-admin CR or a custom CR scoped down to a subset of all Resources that ArgoCD is allowed access to and is used to deploy into a namespace)
• argocd-manage-application (Grants full read/write permissions on Applications in bound namespaces)
• argocd-manage-cluster-resources (Optional, Custom role granting access to a subset of cluster-wide resources)
• manage-subnamespaces (Optional, Grants permission to create and destroy subnamespaces of a namespace through the `SubnamespaceAnchor; This is optional if any deployments by a team should happen through argoCD or mechanisms other than project member creating subnamespaces directly)

At cluster creation, a cluster-admin:

• Installs argocd in a dedicated namespace in namespace mode, creating a roleBinding for argocd-manage-namespace in said namespace with a optional clusterRoleBinding for argocd-manage-cluster-resources if it should also manage some or all cluster-scoped resource
• Creates 2 namespaces "argocd-apps" and "argocd-destination"
• Create a role-binding for argocd's serviceaccount to argocd-manage-applications in the first and argocd-manage-namespace in the latter. They then add a label match criterion for the existence of argocd-apps.tree.hnc.x-k8s.io/depth in applications.namespace.

(Note that all but the optional clusterRole for cluster-wide resources can be replaced with a copy of the role into the relevant namespace, however this creates higher churn and is potentially less clear, but this would allow for truly namespaced, but dynamic ArgoCD permissions)

At this point, these settings do not need to be touched anymore. ArgoCD will automatically be able to deploy into all children of argocd-destination and read applications from all children of argocd-apps creation of sub-namespaces can be delegated via K8S RBAC as needed. When a new team joins, the administrator creates a subnamespace of argocd-apps, called team-a-apps and a subnamespace of "argocd-destination" called "team-a-destination". Finally they create an AppProject, with sourceNamespace label matching on team-a-apps.tree.hnc.x-k8s.io/depth and a destination matching on the tree label for the latter.

Now, if this configuration also needs no further touches, no matter how many namespaces team-a creates in their destination sub-hierarchy. If either team-a "top"-level namespace is removed, all their permissions vanish, and even if another namespace with the same name is created outside of the hierarchy, argoCD will not inherit K8s permissions on it, nor will apps be read from the app name-space since they now exist outside of the containing hierarchy.

In summary, the permission model is fully and explicitly defined as hierarchical relations and permission sub-sets that allow, once setup for teams to fully self-manage, including creating namespace sub-isolation for ephemeral environments.

Proposal

matchLabels: selectors are nothing new or spectacular api or codewise and well established patterns inside the kubernetes ecosystem, so their implementation would be rather straight forward. What I mainly want to discuss is whether this would allow unaware operators to create hard-to-forsee security problems through misconfiguration that might be unacceptable in ArgoCD.

I can see a few areas of potential concern that arise from the fundamental lifecycle difference between name matching and label matching. The name of a resource cannot ever change, as such, whether a specific namespace is captured by the existing model can be decided statically and without contacting kube-api-server. On the other hand, namespaces can gain and shed labels during the lifetime, which makes authorizing decisions for a certain namespace depend on the current state of the cluster. This means:

• previously a certain AppProject could not get access to an existing namespace full of resources without editing the AppProject itself (which should always be forbidden for non-argo/cluster admins)
• A different ability, that of labeling a namespace, is now used to get access to a namespace of a certain name, previously, this was limited to the ability to create a namespace, however, there is no "label" verb in Kubernetes RBAC, just update, which is an incredibly powerful verb bestow on a low privilege user/role, so it can be debated whether this actually enables higher exploitability. Many operators already use label matching to decide if they should affect a certain namespace (including ArgoCD-operator), so it seems a common view that both update and create namespace permissions are only given out to highly trusted nigh-admins or cluster-admins.
• ArgoCD now needs to make explicit checks whenever it syncs into a namespace (atleast if the namespace is not covered by explicit name rules and a label match destination selector exists for the relevant AppProject).

Note that if the HNC is installed, running and only tree-labels are used, it is impossible to gain access to an existing namespace without having the ability to update namespaces and also set the parentage of an existing namespace within HNC's RBAC, otherwise the controller will immediately reconcile the attempted label change to conform to the correct hierarchy.

I would be very glad to supply the necessary PR to make this a reality, especially because I would like to use this pattern in IDPs I am building/maintaining. First, however. I would like to know whether this change in the security/permission model for applications in any namespace is tenable at all.

I can see two approaches:

• Create an explicit synergy with HNC, allowing label matching only for tree-labels and hiding the implementation details from the user via a "includeSubNamespaces" field on relevant name selectors. This would have good ergonomics and a higher security level, atleast while the HNC is deployed and running, however this seem overly inflexible and reliant on "magic values", despite using an extremely well established concept of label-matching. If HNC were ever to be included into base K8s by default however, this might be preferable as an alternate, less risky configuration option.
• Allow matching on existence or values of arbitrary labels. This creates great flexibility and interoperability with other operators. On the other hand, it allows for easier privilege escalation through careless configuration, however, so does sourceNamespaces/destinations[].namespace = *.

There are some other detail design considerations for example:

• churn of api-server requests, however a cache and a watch on relevant labels should allow us to make the impact negligible
• how to structure label match syntax in cli flag

But I think they are secondary to whether the general enhancement is reasonable

[andrii-korotkov-verkada]

Prefix matching can be tricky. What if team B creates an app with a name for team A? How do we avoid them getting extra permissions?