search

kyverno / kyverno

Cloud Native Policy Management
https://kyverno.io
Apache License 2.0
5.61k stars 858 forks source link

[Bug] PolicyException causes Kyverno to skip entire workload, not only the excepted parts #10581

Closed KRASSUSS closed 2 months ago

KRASSUSS commented 3 months ago

Kyverno Version

1.12.4

Description

There are 2 generic PolicyExceptions deployed on the cluster that except istio containers:

Polex 1:

spec:
  exceptions:
  - policyName: psp-baseline
    ruleNames:
    - baseline
  match:
    any:
    - resources:
        kinds:
        - Pod
  podSecurity:
  - controlName: Capabilities
    images:
    - '*/istio/proxyv2*'
    restrictedField: spec.initContainers[*].securityContext.capabilities.add
    values:
    - NET_ADMIN
    - NET_RAW

Polex 2:

spec:
  exceptions:
  - policyName: psp-restricted-limited
    ruleNames:
    - restricted
  match:
    any:
    - resources:
        kinds:
        - Pod
        selector:
          matchLabels:
            service.istio.io/canonical-revision: latest
  podSecurity:
  - controlName: Running as Non-root
    images:
    - '*/istio/proxyv2*'
    restrictedField: spec.initContainers[*].securityContext.runAsNonRoot
    values:
    - "false"
  - controlName: Running as Non-root user
    images:
    - '*/istio/proxyv2*'
    restrictedField: spec.initContainers[*].securityContext.runAsUser
    values:
    - "0"

However when a non-compliant workload with istio containers is deployed the 2 Polex'es above cause the entire pod to be skipped, not only the istio related config.

Below is an example of a workload Pod that is skipped, while container app-1 is not compliant:

apiVersion: v1
items:
- apiVersion: v1
  kind: Pod
  metadata:
    annotations:
      istio.io/rev: default
    labels:
      app: app-1
    name: app-1-67c8567d6d-xsxqb
    namespace: app-1-namespace
    ownerReferences:
    - apiVersion: apps/v1
      blockOwnerDeletion: true
      controller: true
      kind: ReplicaSet
      name: app-1-67c8567d6d
  spec:
    containers:
    - env:
      - name: CLOUD_OPTION
        value: csp
      image: container.registry/app-1:v.2.0.0
      imagePullPolicy: IfNotPresent
      name: app-1
      ports:
      - containerPort: 80
        protocol: TCP
      resources:
        limits:
          cpu: "2"
          memory: 4Gi
        requests:
          cpu: 50m
          memory: 256Mi
      securityContext:
        allowPrivilegeEscalation: false
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
    - args:
      - proxy
      - sidecar
      - --domain
      - $(POD_NAMESPACE).svc.cluster.local
      - --proxyLogLevel=warning
      - --proxyComponentLogLevel=misc:error
      - --log_output_level=default:info
      env:
      - name: JWT_POLICY
        value: Some envs
      image: us-east4-docker.pkg.dev/shared-gcr-c99a/mct-stack/istio/proxyv2:1.18.7
      imagePullPolicy: IfNotPresent
      name: istio-proxy
      ports:
      - containerPort: 15090
        name: http-envoy-prom
        protocol: TCP
      readinessProbe:
        failureThreshold: 30
        httpGet:
          path: /healthz/ready
          port: 15021
          scheme: HTTP
        initialDelaySeconds: 1
        periodSeconds: 2
        successThreshold: 1
        timeoutSeconds: 3
      resources:
        limits:
          cpu: "2"
          memory: 1Gi
        requests:
          cpu: 10m
          memory: 40Mi
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
          - ALL
        privileged: false
        readOnlyRootFilesystem: true
        runAsGroup: 1337
        runAsNonRoot: true
        runAsUser: 1337
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
    dnsPolicy: ClusterFirst
    enableServiceLinks: true
    initContainers:
    - args:
      - istio-iptables
      env:
      - name: TERMINATION_DRAIN_DURATION_SECONDS
        value: "30"
      image: some.registry/istio/proxyv2:1.18.7
      imagePullPolicy: IfNotPresent
      name: istio-init
      resources:
        limits:
          cpu: "2"
          memory: 1Gi
        requests:
          cpu: 10m
          memory: 40Mi
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          add:
          - NET_ADMIN
          - NET_RAW
          drop:
          - ALL
        privileged: false
        readOnlyRootFilesystem: false
        runAsGroup: 0
        runAsNonRoot: false
        runAsUser: 0

This is how it shows in the UI:

image

And policyreports on the cluster show the same result.

Slack discussion

No response

Troubleshooting

MariamFahmy98 commented 3 months ago

Could you please share with us the two policies which named psp-baseline and psp-restricted-limited?

MariamFahmy98 commented 3 months ago

There is a related issue: https://github.com/kyverno/kyverno/issues/10580 If it is a duplicate, could you please close one of them?

KRASSUSS commented 3 months ago

Sure, here are the policies:

psp-baseline:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: psp-baseline
  annotations:
    pod-policies.kyverno.io/autogen-controllers: none
spec:
  failurePolicy: Ignore
  background: true
  validationFailureAction: Audit
  rules:
  - name: baseline
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      podSecurity:
        level: baseline
        version: v1.29

psp-restricted-limited

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: psp-restricted-limited
  annotations:
    pod-policies.kyverno.io/autogen-controllers: none
spec:
  failurePolicy: Ignore
  background: true
  validationFailureAction: Audit
  rules:
  - name: restricted
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      podSecurity:
        level: restricted
        version: v1.29
        exclude:
        - controlName: Volume Types
        - controlName: Seccomp
        - controlName: Seccomp
          images:
          - '*'
        - controlName: Capabilities
          images:
          - "*"

Regarding the other issue I have submitted, they're both related to Polexes but the behaviour in both is different - this one excepts the resources too much, while the other one doesn't except resources at all while it should. If you think the root cause for both might be the same I can close one of them!

MariamFahmy98 commented 3 months ago

Why do you think that container app-1 violates the PSP policies?

Without deploying exceptions, the creation of the pod will be blocked because of the init containers not the app-1 container:

Error from server: error when creating "pod.yaml": admission webhook "validate.kyverno.svc-ignore" denied the request: 

resource Pod/default/test-pod was blocked due to the following policies 

psp-baseline:
  baseline: 'Validation rule ''baseline'' failed. It violates PodSecurity "baseline:v1.29":
    (Forbidden reason: non-default capabilities, field error list: [spec.initContainers[0].securityContext.capabilities.add
    is forbidden, forbidden values found: [NET_ADMIN NET_RAW]])'
psp-restricted-limited:
  restricted: 'Validation rule ''restricted'' failed. It violates PodSecurity "restricted:v1.29":
    (Forbidden reason: runAsNonRoot != true, field error list: [spec.initContainers[0].securityContext.runAsNonRoot
    is forbidden, forbidden values found: false])(Forbidden reason: runAsUser=0, field
    error list: [spec.initContainers[0].securityContext.runAsUser is forbidden, forbidden
    values found: 0])'

With deploying exceptions that exempt the init containers, the pod is expected to be created because only the init containers do violate the two policies.

KRASSUSS commented 2 months ago

Isn't the container app-1 violating the Running as Non-root security standard?

MariamFahmy98 commented 2 months ago

The container app-1 sets the field runAsNonRoot to true so it doesn't violate the restricted profile. https://kubernetes.io/docs/concepts/security/pod-security-standards/#restricted

KRASSUSS commented 2 months ago

Doesn't istio-proxy container set it? The only securityContext that is set by app-1 is allowPrivilegeEscalation: false. Or is Privilage Escalation false config enough then?

MariamFahmy98 commented 2 months ago

My bad! Yes, the container istio-proxy is the one which sets the field runAsNonRoot to true. For the container app-1, I see that the allowPrivilegeEscalation is set to false which doesn't violate the restricted profile.

From the docs:

Privilege escalation (such as via set-user-ID or set-group-ID file mode) should not be allowed. This is Linux only policy in v1.25+ (spec.os.name != windows)

Restricted Fields

spec.containers[].securityContext.allowPrivilegeEscalation spec.initContainers[].securityContext.allowPrivilegeEscalation spec.ephemeralContainers[*].securityContext.allowPrivilegeEscalation Allowed Values

false

akardaspg commented 2 months ago

My bad! Yes, the container istio-proxy is the one which sets the field runAsNonRoot to true. For the container app-1, I see that the allowPrivilegeEscalation is set to false which doesn't violate the restricted profile.

I think the point is that this container from above example from @KRASSUSS 

spec:
    containers:
    - env:
      - name: CLOUD_OPTION
        value: csp
      image: container.registry/app-1:v.2.0.0
      imagePullPolicy: IfNotPresent
      name: app-1
      ports:
      - containerPort: 80
        protocol: TCP
      resources:
        limits:
          cpu: "2"
          memory: 4Gi
        requests:
          cpu: 50m
          memory: 256Mi
      securityContext:
        allowPrivilegeEscalation: false
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File

it has no defined runAsNonRoot: false and this is not even blocked/reported by Kyverno itself, as you pasted above:

 Error from server: error when creating "pod.yaml": admission webhook "validate.kyverno.svc-ignore" denied the request: 

 resource Pod/default/test-pod was blocked due to the following policies 

 psp-baseline:
  baseline: 'Validation rule ''baseline'' failed. It violates PodSecurity "baseline:v1.29":
     (Forbidden reason: non-default capabilities, field error list: [spec.initContainers[0].securityContext.capabilities.add
    is forbidden, forbidden values found: [NET_ADMIN NET_RAW]])'
 psp-restricted-limited:
   restricted: 'Validation rule ''restricted'' failed. It violates PodSecurity "restricted:v1.29":
     (Forbidden reason: runAsNonRoot != true, field error list: [spec.initContainers[0].securityContext.runAsNonRoot
     is forbidden, forbidden values found: false])(Forbidden reason: runAsUser=0, field
     error list: [spec.initContainers[0].securityContext.runAsUser is forbidden, forbidden
     values found: 0])'

kyverno do not reports it, there is no Forbidden reason: runAsNonRoot != true for spec.containers[0] at all, only spec.initContainers[0] and I think it shoud be because there is no spec.securityContext.runAsNonRoot at POD level, only istio in init-container is setting this. So if there is PolicyException just for istio-init only, the container anyway can be deployed running as root because it is not blocked/reported by kyverno and I think that is the problem not allowPrivilegeEscalation - @KRASSUSS correct me if I'm wrong:

KRASSUSS commented 2 months ago

Correct, in short, it's passing Kyverno violation while it shouldn't! :)

akardaspg commented 2 months ago

I have played a bit with this, and I think problem is deeper, related to PSS checks.

As you can see below, check_runAsNonRoot that is used by kyverno to check PSS comppliance returns different result depending if any container within Pod has explicitly set up runAsNonRoot=false or not:

https://github.com/kubernetes/pod-security-admission/blob/b4ce4eb4b4290905b28da64c7f6ecebe5143dbc4/policy/check_runAsNonRoot.go#L110-L130

    // pod or containers explicitly set runAsNonRoot=false
    if len(badSetters) > 0 {
        return CheckResult{
            Allowed:         false,
            ForbiddenReason: "runAsNonRoot != true",
            ForbiddenDetail: fmt.Sprintf("%s must not set securityContext.runAsNonRoot=false", strings.Join(badSetters, " and ")),
        }
    }

    // pod didn't set runAsNonRoot and not all containers opted into runAsNonRoot
    if len(implicitlyBadContainers) > 0 {
        return CheckResult{
            Allowed:         false,
            ForbiddenReason: "runAsNonRoot != true",
            ForbiddenDetail: fmt.Sprintf(
                "pod or %s %s must set securityContext.runAsNonRoot=true",
                pluralize("container", "containers", len(implicitlyBadContainers)),
                joinQuote(implicitlyBadContainers),
            ),
        }
    }

So if you have Pod definition like in this example, that istio-init explicity set up runAsNonRoot=false and other containers does not set up it, so those are implicitlyBadContainers (running as root, and not providing runAsNonRoot=false explicitly in config) CheckResult is not returning info about them to kyverno, have a look here, there is no info that CLOUD_OPTION container from the example is going to run as root (implicitly), there is only information about istio-init:

Error from server: error when creating "pod.yaml": admission webhook "validate.kyverno.svc-ignore" denied the request: 

 resource Pod/default/test-pod was blocked due to the following policies 

 psp-baseline:
  baseline: 'Validation rule ''baseline'' failed. It violates PodSecurity "baseline:v1.29":
     (Forbidden reason: non-default capabilities, field error list: [spec.initContainers[0].securityContext.capabilities.add
    is forbidden, forbidden values found: [NET_ADMIN NET_RAW]])'
 psp-restricted-limited:
   restricted: 'Validation rule ''restricted'' failed. It violates PodSecurity "restricted:v1.29":
     (Forbidden reason: runAsNonRoot != true, field error list: [spec.initContainers[0].securityContext.runAsNonRoot
     is forbidden, forbidden values found: false])(Forbidden reason: runAsUser=0, field
     error list: [spec.initContainers[0].securityContext.runAsUser is forbidden, forbidden
     values found: 0])'

And then we have that situation, when we provide PolicyException only for istio-init, all implicitlyBadContainers are also allowed to run as root when they should not be allowed, but can't be verified by kyverno as PSS CheckResoult does not returns info abut them, because badSetters within POD definition and the first "if".

When you remove from POD all containers which explicitly set runAsNonRoot=false then PSS CheckResult returns information about implicitlyBadContainers, so in my opinion that results should be merged as single CheckResoult within PSS, to provide correct output data from PPS check to kverno.

@realshuting @MariamFahmy98 cloud you look at above if our assumptions is correct?

MariamFahmy98 commented 2 months ago

I run the following test cases to understand the current behavior:

  1. policy.yaml:

    apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: psp-restricted-limited
annotations:
pod-policies.kyverno.io/autogen-controllers: none
spec:
failurePolicy: Ignore
background: true
validationFailureAction: Enforce
rules:
- name: restricted
match:
  any:
  - resources:
      kinds:
      - Pod
validate:
  podSecurity:
    level: restricted
    version: v1.29
    exclude:
    - controlName: Volume Types
    - controlName: Seccomp
    - controlName: Seccomp
      images:
      - '*'
    - controlName: Capabilities
      images:
      - "*"

  2. Run a pod that has one container without setting the runAsNonRoot field:

    apiVersion: v1
kind: Pod
metadata:
labels:
run: test-pod
name: test-pod
spec:
containers:
- image: nginx
name: test-pod
resources:
  limits:
    cpu: "2"
    memory: 4Gi
  requests:
    cpu: 50m
    memory: 256Mi
securityContext:
  allowPrivilegeEscalation: false

    The resource is blocked because runAsNonRoot field is required since this field isn't set under the spec level (expected behavior):

    
resource Pod/default/test-pod was blocked due to the following policies

psp-restricted-limited: restricted: 'Validation rule ''restricted'' failed. It violates PodSecurity "restricted:v1.29": (Forbidden reason: runAsNonRoot != true, field error list: [spec.containers[0].securityContext.runAsNonRoot: Required value])'

3. Run a pod that has a container that doesn't set the `runAsNonRoot` field and an init container that sets the field to `true`:

apiVersion: v1 kind: Pod metadata: labels: run: test-pod name: test-pod spec: containers:

psp-restricted-limited: restricted: 'Validation rule ''restricted'' failed. It violates PodSecurity "restricted:v1.29": (Forbidden reason: runAsNonRoot != true, field error list: [spec.initContainers[0].securityContext.runAsNonRoot is forbidden, forbidden values found: false])'


The pod is blocked but it complains about setting the field to `false` only! It should also state that this field is required for `containers[0]`. 

I think this is the real bug we want to solve and if it is solved, it will work as expected when deploying exceptions.
MariamFahmy98 commented 2 months ago

@akardaspg - Yes, you are correct.

If either pod or container sets the field to false, It goes to the 1st condition and returns the check result directly without even checking the implicit bad containers that don't set the field at all.

KRASSUSS commented 2 months ago

All in all, what we want to achieve is exactly what's mentioned here: https://github.com/kyverno/kyverno/issues/8570

MariamFahmy98 commented 2 months ago

All in all, what we want to achieve is exactly what's mentioned here: #8570

Exceptions can be applied at container level for validate.podSecurity rules. This enhancement is for other rule types.

For the sake of this issue, it is a bug as mentioned above and I pushed a fix in the upstream repo. Once it is merged, I will test it against Kyverno with/without exceptions.