Gatekeeper constraints not correctly evaluating `PriorityClass` objects

imbgar commented 2 months ago

What steps did you take and what happened: I have a gatekeeper deployment that correctly evaluates policies for other kinds, but not PriorityClass.

Deploy gatekeeper via Argo with a manifest like this(slightly modified for obscurity):

---
# gatekeeper.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: "gatekeeper-obscured"
  namespace: argocd
  finalizers: ["resources-finalizer.argocd.argoproj.io"]
  labels:
    app: gatekeeper
    helm.sh/chart: gatekeeper-0.2.1 # correlates to v3.16.3 release
    app.kubernetes.io/managed-by: Helm
    accountName: stage
    version: "0.2.1" # correlates to v3.16.3 release
    bundle: gatekeeper
    region: us-west-2
spec:
  project: default
  source:
    repoURL: obscured.dkr.ecr.us-west-2.amazonaws.com/charts
    targetRevision: "3.16.3"
    chart: gatekeeper
    helm:
      releaseName: "gatekeeper"
      values: |
        audit:
          nodeSelector:
            node-pool: platform-service-pool
          tolerations:
          - effect: NoSchedule
            key: platform-service-pool
            operator: Equal
            value: "true"
        auditInterval: 43200
        controllerManager:
          nodeSelector:
            node-pool: platform-service-pool
          tolerations:
          - effect: NoSchedule
            key: platform-service-pool
            operator: Equal
            value: "true"
        crds:
          nodeSelector:
            node-pool: platform-service-pool
          tolerations:
          - effect: NoSchedule
            key: platform-service-pool
            operator: Equal
            value: "true"
        image:
          repository: obscured.dkr.ecr.us-west-2.amazonaws.com/gatekeeper
        podAnnotations:
          ad.datadoghq.com/manager.checks: |
            {
              "openmetrics": {
                "instances": [
                  {
                    "openmetrics_endpoint": "http://%%host%%:%%port_metrics%%/metrics",
                    "namespace": "opagatekeeper",
                    "metrics": [".*"]
                  }
                ]
              }
            }
          obscured.backstage.com/entity_ref: component:gatekeeper
        podLabels:
          deployed-by: argocd-infra
          generated-by: helm
        postInstall:
          labelNamespace:
            image:
              repository: obscured.dkr.ecr.us-west-2.amazonaws.com/gatekeeper-crds
          nodeSelector:
            node-pool: platform-service-pool
          probeWebhook:
            image:
              repository: obscured.dkr.ecr.us-west-2.amazonaws.com/curl
          tolerations:
          - effect: NoSchedule
            key: platform-service-pool
            operator: Equal
            value: "true"
        postUpgrade:
          nodeSelector:
            node-pool: platform-service-pool
          tolerations:
          - effect: NoSchedule
            key: platform-service-pool
            operator: Equal
            value: "true"
        preInstall:
          crdRepository:
            image:
              repository: obscured.dkr.ecr.us-west-2.amazonaws.com/gatekeeper-crds
  destination:
    name: "a-test-cluster"
    namespace: gatekeeper
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
  ignoreDifferences:
    - group: admissionregistration.k8s.io
      kind: MutatingWebhookConfiguration
      jqPathExpressions:
        - ".webhooks[].matchConditions"
    - group: admissionregistration.k8s.io
      kind: ValidatingWebhookConfiguration
      jqPathExpressions:
        - ".webhooks[].matchConditions"

Review the live ValidatingWebhookConfiguration

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  annotations:
    argocd.argoproj.io/tracking-id: >-
      gatekeeper-4d785f5e:admissionregistration.k8s.io/ValidatingWebhookConfiguration:gatekeeper/gatekeeper-validating-webhook-configuration
  labels:
    app: gatekeeper
    app.kubernetes.io/instance: gatekeeper-4d785f5e
    chart: gatekeeper
    gatekeeper.sh/system: 'yes'
    heritage: Helm
    release: gatekeeper
  name: gatekeeper-validating-webhook-configuration
webhooks:
  - admissionReviewVersions:
      - v1
      - v1beta1
    clientConfig:
      service:
        name: gatekeeper-webhook-service
        namespace: gatekeeper
        path: /v1/admit
    failurePolicy: Ignore
    matchConditions: []
    matchPolicy: Exact
    name: validation.gatekeeper.sh
    namespaceSelector:
      matchExpressions:
        - key: admission.gatekeeper.sh/ignore
          operator: DoesNotExist
        - key: kubernetes.io/metadata.name
          operator: NotIn
          values:
            - gatekeeper
    objectSelector: {}
    rules:
      - apiGroups:
          - '*'
        apiVersions:
          - '*'
        operations:
          - CREATE
          - UPDATE
        resources:
          - '*'
          - pods/ephemeralcontainers
          - pods/exec
          - pods/log
          - pods/eviction
          - pods/portforward
          - pods/proxy
          - pods/attach
          - pods/binding
          - deployments/scale
          - replicasets/scale
          - statefulsets/scale
          - replicationcontrollers/scale
          - services/proxy
          - nodes/proxy
          - services/status
    sideEffects: None
    timeoutSeconds: 3
  - admissionReviewVersions:
      - v1
      - v1beta1
    clientConfig:
      service:
        name: gatekeeper-webhook-service
        namespace: gatekeeper
        path: /v1/admitlabel
    failurePolicy: Fail
    matchPolicy: Exact
    name: check-ignore-label.gatekeeper.sh
    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: NotIn
          values:
            - gatekeeper
    rules:
      - apiGroups:
          - ''
        apiVersions:
          - '*'
        operations:
          - CREATE
          - UPDATE
        resources:
          - namespaces
    sideEffects: None
    timeoutSeconds: 3

Create the ConstraintTemplate based on this manifest with kubectl apply -f

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  creationTimestamp: null
  name: userpriorityclassboundstest
spec:
  crd:
    spec:
      names:
        kind: UserPriorityClassBoundsTest
  targets:
    - libs:
        - |-
          package lib.core

          default is_gatekeeper = false

          is_gatekeeper {
            has_field(input, "review")
            has_field(input.review, "object")
          }

          resource = input.review.object {
            is_gatekeeper
          }

          resource = input {
            not is_gatekeeper
          }

          format(msg) = {"msg": msg}

          format_with_id(msg, id) = msg_fmt {
            msg_fmt := {
              "msg": sprintf("%s: %s", [id, msg]),
              "details": {"policyID": id},
            }
          }

          apiVersion = resource.apiVersion

          name = resource.metadata.name

          kind = resource.kind

          labels = resource.metadata.labels

          annotations = resource.metadata.annotations

          gv := split(apiVersion, "/")

          group = gv[0] {
            contains(apiVersion, "/")
          }

          group = "core" {
            not contains(apiVersion, "/")
          }

          version := gv[count(gv) - 1]

          has_field(obj, field) {
            not object.get(obj, field, "N_DEFINED") == "N_DEFINED"
          }

          missing_field(obj, field) {
            obj[field] == ""
          }

          missing_field(obj, field) {
            not has_field(obj, field)
          }
      rego: |-
        package userPriorityClassBounds

        import data.lib.core

        default kind := "Fake"

        violation[{"msg": msg}] {
          kind == "PriorityClass"
          kind == "Fake"

          msg := sprintf("%v is forbidden which is outside of the allowed bounds", [kind])
        }
      target: admission.k8s.gatekeeper.sh
status: {}

Create the Constraint with kubectl apply -f

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: UserPriorityClassBoundsTest
metadata:
  name: userpriorityclassboundstest
spec:
  enforcementAction: deny
  match:
    kinds:
    - apiGroups:
      - ""
      kinds:
      - PriorityClass

also tried with the following match condition in the Constraint

  match:
    kinds:
    - apiGroups:
      - "scheduling"
      kinds:
      - PriorityClass

Apply the below PriorityClass object with kubectl apply -f

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: bad-boys 
value: 200000000
preemptionPolicy: PreemptLowerPriority 
globalDefault: false
description: "Bad boys. Whatcha want, whatcha want? Whatcha gonna do When Sheriff John Brown come for you? Tell me, whatcha wanna do? Whatcha gonna do? Yeah." 
---

Observe that the object is successfully created

└─[$] <git:(INFRACOMP-1216/brgarcia/bk*)> k apply -f forbidden_PriorityClass.yaml
priorityclass.scheduling.k8s.io/bad-boys created

Describe the constraint

└─[$] <git:(brgarcia/bk*)> k describe userpriorityclassboundstest
Name:         userpriorityclassboundstest
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  constraints.gatekeeper.sh/v1beta1
Kind:         UserPriorityClassBoundsTest
Metadata:
Creation Timestamp:  2024-08-07T16:27:22Z
Generation:          1
Resource Version:    758700401
UID:                 4dfd6340-b171-44c7-b60b-83cc134203d5
Spec:
Enforcement Action:  deny
Match:
Kinds:
  API Groups:

  Kinds:
    PriorityClass
Status:
By Pod:
Constraint UID:       4dfd6340-b171-44c7-b60b-83cc134203d5
Enforced:             true
Id:                   gatekeeper-audit-56c75775d9-qbgfb
Observed Generation:  1
Operations:
  audit
  mutation-status
  status
Constraint UID:       4dfd6340-b171-44c7-b60b-83cc134203d5
Enforced:             true
Id:                   gatekeeper-controller-manager-79fc97bc84-4cgn9
Observed Generation:  1
Operations:
  mutation-webhook
  webhook
Constraint UID:       4dfd6340-b171-44c7-b60b-83cc134203d5
Enforced:             true
Id:                   gatekeeper-controller-manager-79fc97bc84-hwnk8
Observed Generation:  1
Operations:
  mutation-webhook
  webhook
Constraint UID:       4dfd6340-b171-44c7-b60b-83cc134203d5
Enforced:             true
Id:                   gatekeeper-controller-manager-79fc97bc84-x76gl
Observed Generation:  1
Operations:
  mutation-webhook
  webhook
Events:  <none>

What did you expect to happen: I expect the violation to be caught, AdmissionReview denied, and the msg to be logged back to the me when I run kubectl apply -f

What am I doing wrong? Why is Gatekeeper not evaluating PriorityClass objects? My policies for namespaced objects like Deployment, StatefulSet, Pod, etc work fine.

Environment:

Gatekeeper version: v3.16.3
Kubernetes version: (use kubectl version):

└─[$] <git:(/brgarcia/bk*)> kubectl version
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.6-eks-db838b0

msarfaty commented 1 month ago

any traction here?

JaydipGabani commented 1 month ago

@imbgar Can you try the same with below constaint to see if it works?

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: UserPriorityClassBoundsTest
metadata:
  name: userpriorityclassboundstest
spec:
  enforcementAction: deny
  match:
    kinds:
    - apiGroups:
      - "scheduling.k8s.io"
      kinds:
      - PriorityClass

In both the provided examples of Constraints, apiGroups for PriorityClass is not used correctly.

JaydipGabani commented 1 month ago

In my testing I was able to use below constraint for required_label policy to deny PriorityClass that does not have the required label.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: pod-must-have-gk
spec:
  match:
    kinds:
    - apiGroups:
      - "scheduling.k8s.io"
      kinds:
      - PriorityClass
  parameters:
    message: "All namespaces must have an `owner` label that points to your company username"
    labels:
      - key: owner
        allowedRegex: "^[a-zA-Z]+.agilebank.demo$"

based on the investigation, issue might be something else then "Gatekeeper constraints not correctly evaluating PriorityClass objects". Most likey the issue is misconfiguration of constraint. If not there might be bug in the rego code. I am removing the bug label for now.

@imbgar let me know if the constraint suggested in above comment does not work, we can investigate further. cc: @msarfaty

prasadkatti commented 1 month ago

In both the provided examples of Constraints, apiGroups for PriorityClass is not used correctly.

Hi @JaydipGabani. Thanks for taking the time to debug this. I verified that the constraint is misconfigured.

Specifically, the value of apiGroups in spec.match.kinds is incorrect. Upon setting this to scheduling.k8s.io, the constraint starts getting applied to all PriorityClass resources.

I believe we can close this issue.

open-policy-agent / gatekeeper

Gatekeeper constraints not correctly evaluating `PriorityClass` objects #3475