Enabling cross-namespace resource refs, with minimal service account management

[toumorokoshi]

Hello!

In namespace mode, Config Connector uses a different service account for every namespace that it manages. In that model, when there is a need to reference a resource in another namespace, the permissions to read resource from that namespace must also be granted to the corresponding KSA as well.

Is there a simple way to enable these cross-namespace reads? One cumbersome strategy is to enumerate all apiGroup / resource combinations into a ClusterRole, then attach the binding to the specific service account:

# project-viewer is needed for IAMMemberPolicy projectRef
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: project-viewer
rules:
- apiGroups:
  - resourcemanager.cnrm.cloud.google.com
  resources:
  - projects
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: project-viewer-networking
  namespace: projects
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: project-viewer
subjects:
- kind: ServiceAccount
  name: cnrm-controller-manager-networking
  namespace: cnrm-system

But I believe this would require:

• one ClusterRole per apiGroup that Config Connector owns (not to mention new ones added later)
• one RoleBinding per ClusterRole / namespace combination (since there's a separate KSA per namespace)

Is there a simple set that could be applied once, and scale to new versions of KCC and linear per namespace?

[karlkfi]

Is there any benefit to using resourceRef dependencies across namespaces vs external resourceRefs?

My experience with external resourceRefs has been that their specifying syntax isn't really standardized yet and often requires a substitution with a prefix for the resource type, which is annoying. But I'm wondering if there's any sort of dependency ordering concern or efficiency from depending on a k8s resource (resourceRef), rather than a GCP resource (external resourceRef).

Creating roles and rolebindings for namespaced KCC instances is pretty tedious, especially if you're composing multiple kpt packages that each need to either supply those Roles & RoleBindings (with unique names) or assume the user has provisioned them (since there's no kpt conditionals to optionally enable/disable them).

[toumorokoshi]

I do agree that the rolebinding / roles are tedious. @kibbles-n-bytes had a suggestion that we could automate this creation with the operator. The risk there may be giving even broader permissions to the Config Connector operator or it's various pods.

Is there any benefit to using resourceRef dependencies across namespaces vs external resourceRefs?

Currently I think there's scenarios where the unique identifier for a resource isn't specifyable (e.g. folder ID). For those category of resources, we would have to include more complex logic to resolve such references. It's hard to automate and standardize, so making external references a complete replacement for resourceRef would take time.

You could imagine the logic of retrieving a resource from the api server is much more standard that resolving a unique identifier from a GCP API.

[karlkfi]

I think the easiest way to get around having multiple packages try to micro-manage overlapping permissions is gonna be to just grant KCC in each namespace permission to get/list/watch all KCC resources on the cluster. This will allow for cross-namespace dependencies without user intervention.

[tonybenchsci]

So we've recently started to capture and manage Org level GCP resources (e.g. Projects) with KCC. This opens the door to referencing kind: Projects using resourceRef.{name: ${PROJECT_ID}, namespace: company-org} instead of resourceRef.{external: ${PROJECT_ID}}

So I'm wondering if there can be some elaboration on the benefits of this?

1. external resource Ref syntax is not standard so it's annoying
2. "resource from the api server is much more standard than resolving a unique identifier from a GCP API" (could someone expand on this? As in does it benefit the reconciliation efficiency?)
3. Anything else???

[karlkfi]

cnrm-viewer ClusterRole was added in KCC v1.45.0 to help with using cross-namespace resource references. https://github.com/GoogleCloudPlatform/k8s-config-connector/issues/407

It seems like users will have to create the RoleBindings themselves for new namespaces tho.