Unable to deploy on GKE Autopilot

[yokomotod]

What steps did you take and what happened:

On GKE Autopilot cluster, gatekeeper-mutating-webhook-configuration MutatingWebhookConfiguration resource failed to deploy.

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/v3.10.0/deploy/gatekeeper.yaml

namespace/gatekeeper-system created
...
validatingwebhookconfiguration.admissionregistration.k8s.io/gatekeeper-validating-webhook-configuration created
Error from server (GKE Admission Webhook Controller): error when creating "https://raw.githubusercontent.com/open-policy-agent/gatekeeper/v3.10.0/deploy/gatekeeper.yaml": admission webhook "admissionwebhookcontroller.common-webhooks.networking.gke.io" denied the request: GKE Admission Webhook Controller: the following (group,resource) pairs are not allowed in webhook rules: ('*','*')

I believe this is the problem: https://github.com/open-policy-agent/gatekeeper/blob/v3.10.0/deploy/gatekeeper.yaml#L3433
If I comment out gatekeeper-mutating-webhook-configuration, then other resources will be deployed successfully.

What did you expect to happen:

All resources are deployed successfully.

Anything else you would like to add:

Deployment succeeds on GKE Standard cluster.

Environment:

• Gatekeeper version: v3.10.0
• Kubernetes version: (use kubectl version):

  $ kubectl version --short
  Client Version: v1.22.14-dispatcher-dirty
  Server Version: v1.22.12-gke.2300

And it is GKE Autopilot cluster

[sozercan]

@maxsmythe @acpana any ideas about this? looks like a GKE specific admission webhook?

[acpana]

thanks for reporting @yokomotod ; is this a private GKE cluster?

do the suggestions from this issue help at all? https://github.com/elastic/cloud-on-k8s/issues/1437 ;

in particular adding firewall rules: https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#add_firewall_rules

[yokomotod]

@acpana

No, it's not private cluster.

I could confirm that if I modify the mutating webhook to specific resource then it succeeds to deploy.

So private cluster/firewall doesn't matter I think.

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: gatekeeper-mutating-webhook-configuration
webhooks:
 - admissionReviewVersions:
  # ...
  rules:
  - apiGroups:
    - '*'
    apiVersions:
    - '*'
    operations:
    - CREATE
    - UPDATE
    resources:
-    - '*'
+    - pod

$ kubectl apply -f gatekeeper.yaml
Warning: AdmissionWebhookController: mutated namespaceselector of the webhooks to enforce GKE Autopilot policies.
mutatingwebhookconfiguration.admissionregistration.k8s.io/gatekeeper-mutating-webhook-configuration created

As error message says, the GKE autopilot seems to have specific rule that disallow mutation webhooks which targeting all resources. ( I couldn't find any documents about this though)

the following (group,resource) pairs are not allowed in webhook rules: (' * ',' * ')

[acpana]

hey @yokomotod thanks for your patience. Yes, the firewall suggestion was not applicable/ wrong / a red herring -- sorry about that.

Upon more research, the GKE docs call out:

Note: you cannot enable mutation on Autopilot clusters.

as per https://cloud.google.com/anthos-config-management/docs/how-to/mutation .

So it seems that mutations are not a supported usecase on GKE Autopilot as of now unfortunately. You could send in a feature request if this is something really important for you. In the meantime, IIRC, you can use mutations on GKE non autopilot cluster

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[shannonxtreme]

FWIW Autopilot allows mutating webhooks as long as no wildcards (*) are present. kube-system is also rejected.

https://cloud.google.com/kubernetes-engine/docs/concepts/autopilot-security#webhooks has the full restrictions for mutating webhooks