AKS managed RBAC should cover AKS maintained CRDs

[denniszielke]

What happened: Scenario is about multi-tenant cluster in which each team gets a dedicated namespace to deploy their apps. AKS AD Auth and RBAC are used to secure access to the namespace and the objects for the developer teams. As stated in the limitations (https://docs.microsoft.com/en-us/azure/aks/manage-azure-rbac) AKS managed RBAC does not cover custom CRDs but this will also block the creation of SecretProviderClass and therefore does not allow the developer team to manage access to KeyVault. I get that is is probably not possible to manage access to all CRD but AKS should at least allow the management of AKS maintained CRDs like:

• SecretProviderClass
• ScaledObjects
• FluxCD

I would also expect that the builtin AKS RBAC roles especially Azure Kubernetes Service RBAC Reader and Azure Kubernetes Service RBAC Writer would also include read and write access to these CRD in a dedicated namespace scope.

What you expected to happen: When impersonating an identity that has only access to a namespace I would expect to be able to create/read/update SecretProviderClass when impersonating Azure Kubernetes Service RBAC Writer role granted by AKS managed RBAC.

How to reproduce it (as minimally and precisely as possible): Create Namespace, grant AAD Identity access to namespace only, try to create/read/update a SecretProviderClass

[ghost]

Hi denniszielke, AKS bot here :wave: Thank you for posting on the AKS Repo, I'll do my best to get a kind human from the AKS team to assist you.

I might be just a bot, but I'm told my suggestions are normally quite good, as such: 1) If this case is urgent, please open a Support Request so that our 24/7 support team may help you faster. 2) Please abide by the AKS repo Guidelines and Code of Conduct. 3) If you're having an issue, could it be described on the AKS Troubleshooting guides or AKS Diagnostics? 4) Make sure your subscribed to the AKS Release Notes to keep up to date with all that's new on AKS. 5) Make sure there isn't a duplicate of this issue already reported. If there is, feel free to close this one and '+1' the existing issue. 6) If you have a question, do take a look at our AKS FAQ. We place the most common ones there!

[ghost]

Triage required from @Azure/aks-pm

[ghost]

Action required from @Azure/aks-pm

[MarkTopping]

I second this as a feature request and I have a very similar use case whereby we want to empower teams to manage certain CRD objects within their own namespaces; but not to gain access to all CRDs.

I note in the documentation that it says this is a feature currently being looked into but I was unable to spot a ticket which captures the work.

I doubt you'll implement this to only accommodate a fixed range of known values, but should that be the case then I'd like to append the following CRD's to Dennis's list, or at least backup his request with additional examples of CRDs which I'd welcome granular control over:

• AzureIdentities
• AzureIdentityBindings
• AzurePodIdentityExceptions
• VirtualServers (nginx)
• VirtualServerRoutes (nginx)

[MarkTopping]

I've come to a further realization just now which I feel should add some weight to the important of supporting granular RBAC over CRD's...

If you need to empower a User or a Service Account to deploy into the cluster and at least one of those objects that needs to be deployed is a CRD then you need to grant the permission "Microsoft.ContainerService/managedClusters/*" which is basically cluster admin. That's way over-powered.

[goenning]

I found this on the AKS docs:

If you have CRDs and are making custom role definitions, the only way to cover CRDs today is to provide Microsoft.ContainerService/managedClusters/*/read. AKS is working on providing more granular permissions for CRDs. For the remaining objects you can use the specific API Groups, for example: Microsoft.ContainerService/apps/deployments/read.

It seems like they are working on it, is this the issue we can track for updates?

[denniszielke]

@miwithro can you add this issue to the backlog? Or is there another issue for this?

[ghost]

Triage required from @Azure/aks-pm

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[aidacloud]

I've come to a further realization just now which I feel should add some weight to the important of supporting granular RBAC over CRD's...

If you need to empower a User or a Service Account to deploy into the cluster and at least one of those objects that needs to be deployed is a CRD then you need to grant the permission "Microsoft.ContainerService/managedClusters/*" which is basically cluster admin. That's way over-powered.

This has been a major issue for us as well. We've restricted access to CRDs, but it creates a ton of friction between the ops and development teams.

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[CocoWang-wql]

Checking this one with internal team and will post here once we have any update.

[ghost]

Triage required from @Azure/aks-pm

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[8-bit-fox]

@aidacloud, @MarkTopping I found a work-around by creating a new role that provides global read permissions (from the documentation) and remove permissions that I do not want to grant. E.g., a CustomReader role would look like

permissions {
    actions = [
      "Microsoft.Authorization/*/read",
      "Microsoft.Resources/subscriptions/operationresults/read",
      "Microsoft.Resources/subscriptions/read",
      "Microsoft.Resources/subscriptions/resourceGroups/read"
    ]
    data_actions = [
      "Microsoft.ContainerService/managedClusters/*/read",
    ]
    not_data_actions = [
      "Microsoft.ContainerService/managedClusters/admissionregistration.k8s.io/*",
      "Microsoft.ContainerService/managedClusters/api/*",
      "Microsoft.ContainerService/managedClusters/apiregistration.k8s.io/apiservices/*",
      "Microsoft.ContainerService/managedClusters/apis/*",
      "Microsoft.ContainerService/managedClusters/authentication.k8s.io/*",
      "Microsoft.ContainerService/managedClusters/authorization.k8s.io/*",
      "Microsoft.ContainerService/managedClusters/bindings/*",
      "Microsoft.ContainerService/managedClusters/certificates.k8s.io/*",
      "Microsoft.ContainerService/managedClusters/componentstatuses/*",
      "Microsoft.ContainerService/managedClusters/coordination.k8s.io/*",
      "Microsoft.ContainerService/managedClusters/discovery.k8s.io/endpointslices/*",
      "Microsoft.ContainerService/managedClusters/endpoints/*",
      "Microsoft.ContainerService/managedClusters/extensions/podsecuritypolicies/*",
      "Microsoft.ContainerService/managedClusters/flowcontrol.apiserver.k8s.io/*",
      "Microsoft.ContainerService/managedClusters/groups/impersonate/*",
      "Microsoft.ContainerService/managedClusters/healthz/*",
      "Microsoft.ContainerService/managedClusters/livez/*",
      "Microsoft.ContainerService/managedClusters/metrics/*",
      "Microsoft.ContainerService/managedClusters/metrics.k8s.io/*",
      "Microsoft.ContainerService/managedClusters/networking.k8s.io/ingressclasses/*",
      "Microsoft.ContainerService/managedClusters/node.k8s.io/*",
      "Microsoft.ContainerService/managedClusters/nodes/*",
      "Microsoft.ContainerService/managedClusters/openapi/v2/*",
      "Microsoft.ContainerService/managedClusters/persistentvolumes/*",
      "Microsoft.ContainerService/managedClusters/podtemplates/*",
      "Microsoft.ContainerService/managedClusters/policy/podsecuritypolicies/*",
      "Microsoft.ContainerService/managedClusters/rbac.authorization.k8s.io/*",
      "Microsoft.ContainerService/managedClusters/readyz/*",
      "Microsoft.ContainerService/managedClusters/resetMetrics/*",
      "Microsoft.ContainerService/managedClusters/scheduling.k8s.io/*",
      "Microsoft.ContainerService/managedClusters/secrets/*",
      "Microsoft.ContainerService/managedClusters/storage.k8s.io/*",
      "Microsoft.ContainerService/managedClusters/swagger-api/*",
      "Microsoft.ContainerService/managedClusters/ui/*",
      "Microsoft.ContainerService/managedClusters/users/*",
      "Microsoft.ContainerService/managedClusters/version/*"
    ]
  }

(if I did not miss anything).

Of course, this does not allow for granular CRD access, but at least you can circumvent other major pain-points. And please note: This solution is fragile in terms of future changes to the data-actions available through Azure: If a new action is added, it will have "allow" as default permission and not "deny".

The (hopefully complete) list of data-actions is found in the docs.

[ghost]

Issue needing attention of @Azure/aks-leads

[shashankbarsin]

@denniszielke the ask makes sense, we are currently looking into ABAC support in AAD for supporting CRDs as part of Azure RBAC for AKS too. Will get back with an ETA on this

[ghost]

Action required from @shashankbarsin, @Anumita.

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[ghost]

This issue will now be closed because it hasn't had any activity for 7 days after stale. denniszielke feel free to comment again on the next 7 days to reopen or open a new issue after that time if you still have a question/issue or suggestion.

[denniszielke]

Not stale

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[sharebear]

Definitely not stale, hit this problem a couple of weeks back.

[Mohid-A]

we have a similar use case for camelk based workloads, where we want to provide read-and-write access to certain CRDs that camelk relies on, so that different app teams can read and update the CRD's as needed. Support for granular role assignments for CRD's will definitely help such usecases. Thanks

[elmolinnell]

This is still very much a live issue. As with a lot of the other guys here I've run into the same issue.

Our use case is to allow development teams to uninstall Helm Charts which include CRD's from the specific namespaces which they have rights delegated on. As suggested by @8-bit-fox, I have created custom role which allows enough rights to delete everything except the CRD for the secretproviderclasses.secrets-store.csi.x-k8s.io. Giving Cluster Admin rights scoped to the namespace seems like complete overkill.

[petersmallegange]

Also ran into this permissions issue for CRD's. Solved this by creating custom role and added permissions for customresourcedefinitions.

 data_actions = [
      "Microsoft.ContainerService/managedClusters/apiextensions.k8s.io/customresourcedefinitions/read",

[antoineozenne-at-leocare]

This is a problem for us too. We would give the permission to read/write azconfig.io/azureappconfigurationproviders to a managed identity (which is used by GitHub to deploy our Helm charts). I thought about several solutions:

• As indicated in the documentation, we can't explicitly give permissions on a CRD with an Azure custom role (because we cannot use an unknown DataAction).
• Add this identity in an AAD group and assign a Kubernetes RoleBinding to this group. But this identity is deployed with Azure Service Operator which doesn't allow adding an identity to a group.
• Change the --oidc-username-claim parameter of the Kubernetes apiserver to map the name of the managed identity instead of the ObjectId (from sub). This way, I could assign a Kubernetes RoleBinding directly to the managed identity because I know its name (while I don't know its ObjectId before it is created). This solution seems to be overkilled, and anyway, AKS doesn't allow customizing the --oidc-username-claim parameter (see https://github.com/Azure/AKS/issues/2861).
• Use Kubernetes aggregated ClusterRole to aggregate a custom ClusterRole to the built-in edit ClusterRole. But I realized AKS does not seem to use the built-in Kubernetes roles with its Azure built-in roles.

So the only solution we have for the moment is to use the @8-bit-fox -like solution. But it is a big security issue to give an admin role to an identity used by our CI.

The irony is that azconfig.io/azureappconfigurationproviders is a resource provided by Azure. 😄

[MarkTopping]

Just chipping in again to say that this limitation is causing friction for me once more.

My use-case this time is that I need to deploy GitHub Actions self-hosted runners which will be orchestrated by the Actions Runner Controller (ARC).

The 1st deployment for the Controller creates a crd called AutoscalingRunnerSet. I believe I can grant permissions for the crd to be created by using the '"Microsoft.ContainerService/managedClusters/apiextensions.k8s.io/customresourcedefinitions/*' data action. So far so good.

But the 2nd and subsequent RunnerSet deployment needs to then create one of these custom AutoscalingRunnerSet objects and therefore the Role needs to allow for resources of this particular crd type to be created. The friction point is that I cannot grant a data_action which permits creating AutoscalingRunnerSet objects and therefore to work around this I need to offer up admin level permissions by granting Microsoft.ContainerService/managedClusters/*. And then like others in this ticket have noted, I additionally need to manage a tonne of permissions under the not_data_actions property as to peel back many of the most sensitive resource types

permissions {
  actions = [ ]
  data_actions = [
    "Microsoft.ContainerService/managedClusters/*" 
  ]
  not_actions = [ ]
  not_data_actions = [ 
    # Insert 10's or 100's of rows here to remove permissions that we didn't want to give
  ]
}

I know the ticket was opened asking for just AKS maintained CRDs to be covered by Azure RBAC but I would very much welcome a more flexible solution which allows AKS administrators/operators to use RBAC for any CRD type that exists in our environments.

[rajagopalan-trimble]

I am also waiting for a solution on this. Require a custom role that can manage the CRDs in a cluster.

[MarkTopping]

@shashankbarsin

Hi,

You mentioned ~1 year back that you were looking into the options and would get back in touch with an ETA. I was just wondering if there were any updates to be had?

Thanks

[microsoft-github-policy-service[bot]]

This issue has been automatically marked as stale because it has not had any activity for 21 days. It will be closed if no further activity occurs within 7 days of this comment.

[denniszielke]

Not stale and being worked on.

[microsoft-github-policy-service[bot]]

This issue has been automatically marked as stale because it has not had any activity for 21 days. It will be closed if no further activity occurs within 7 days of this comment.

[antoineozenne-at-leocare]

/no-stale please

[microsoft-github-policy-service[bot]]

This issue has been automatically marked as stale because it has not had any activity for 21 days. It will be closed if no further activity occurs within 7 days of this comment.

[denniszielke]

Not stale

[anroots-by]

This is dearly needed by any organization who wants to give self-service but guardrailed access to engineering teams.

Microsoft/Azure promotes various advantages of using the managed AKS cluster (tie-in with Entra, KeyVault for secret management), but these are not securely usable without support for CRD RBAC-s.

I want to give engineering teams full access to resources in their namespaces, including CRD based PrometheusRule and SecretProviderClass. My options right now are to either ditch the principle of least privilege and grant everyone admin access (and attempt to viably limit it with not_data_actions); or gate deployment operations to a central "deployment" team (who are admins), or CI (which will have a big blast radius if admin and compromised). This does not align with good security design. A preferred solution would be ability to specify per CRD-class based RBAC similarly to other in-cluster resources.

Microsoft recently said that security is prioritized above all else. Please live up to this commitment, and prioritize resolving this gap in AKS security model. CRD-s are an integral part of any K8s deployment.