Sample policies for new image data functionality

[chipzoller]

Several new policies can be created based on PR https://github.com/kyverno/kyverno/pull/2946, some of which come from the Proof Manifests.

Resolve image to its digest

apiVersion : kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: resolve-images
spec:
  background: false
  rules:
  - name: resolve-tag-to-digest
    match:
      resources:
        kinds:
        - Pod
    preconditions:
      all:
      - key: "{{request.operation}}"
        operator: NotEquals
        value: DELETE
    mutate:
      foreach:
      - list: "request.object.spec.containers"
        context:
          - name: resolvedRef
            imageRegistry:
              reference: "{{ element.image }}"
              jmesPath: "resolvedImage"
        patchStrategicMerge:
          spec:
            containers:
            - name: "{{ element.name }}"           
              image: "{{ resolvedRef }}"

Any Pod can be used, ex.,

apiVersion: v1
kind: Pod
metadata:
  name: busybox
spec:
  containers:
  - name: busybox
    image: busybox:1.28

Block images with volumes

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: images-with-volumes
spec:
  validationFailureAction: enforce
  rules:
  - name: block-images-with-vols
    match:
      resources:
        kinds:
        - Pod
    preconditions:
      all:
      - key: "{{request.operation}}"
        operator: NotEquals
        value: DELETE
    validate:
      message: "Images containing built-in volumes are prohibited."
      foreach:
      - list: "request.object.spec.containers"
        context: 
        - name: imageData
          imageRegistry: 
            reference: "{{ element.image }}"
        deny:
          conditions:
            all:
              - key: "{{ imageData.configData.config.Volumes || '' | length(@) }}"
                operator: GreaterThan
                value: 0

Bad

apiVersion: v1
kind: Pod
metadata:
  name: image-vol
spec:
  containers:
  - name: image-vol
    image: clover/volume:passbolt

Good

apiVersion: v1
kind: Pod
metadata:
  name: busybox
spec:
  containers:
  - name: busybox
    image: busybox:1.28

Block large images

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: images
spec:
  validationFailureAction: enforce
  rules:
  - name: only-allow-small-images
    match:
      resources:
        kinds:
        - Pod
    preconditions:
      all:
      - key: "{{request.operation}}"
        operator: NotEquals
        value: DELETE
    validate:
      message: "images with size greater than 2Gi not allowed"  
      foreach:
      - list: "request.object.spec.containers"
        context: 
        - name: imageSize
          imageRegistry: 
            reference: "{{ element.image }}"
            # Note that we need to use `to_string` here to allow kyverno to treat it like a resource quantity of type memory
            # the total size of an image as calculated by docker is the total sum of its layer sizes
            jmesPath: "to_string(sum(manifest.layers[*].size))"
        deny:
          conditions:
            all:
            - key: "2Gi"
              operator: LessThan
              value: "{{imageSize}}"

Bad

apiVersion: v1
kind: Pod
metadata:
  name: large-image
spec:
  containers:
  - name: large-image
    image: nvidia/cuda:11.6.0-devel-ubi8

Good

apiVersion: v1
kind: Pod
metadata:
  name: small-image
spec:
  containers:
  - name: small-image
    image: busybox:1.28

Only trustworthy registries with root users

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: check-trustable-images
spec:
  validationFailureAction: enforce
  rules:
  - name: only-allow-trusted-images
    match:
      resources:
        kinds:
        - Pod
    preconditions:
      all:
      - key: "{{request.operation}}"
        operator: NotEquals
        value: DELETE
    validate:
      message: "images with root user are not allowed"  
      foreach:
      - list: "request.object.spec.containers"
        context: 
        - name: imageData
          imageRegistry: 
            reference: "{{ element.image }}"
        deny:
          conditions:
            all:
              - key: "{{ imageData.configData.config.User || ''}}"
                operator: Equals
                value: ""
              - key: "{{ imageData.registry }}"
                operator: NotEquals
                value: "ghcr.io"

Bad

# fails
apiVersion: v1
kind: Pod
metadata:
  name: pod-with-root-user
spec:
  containers:
  - name: busybox
    image: busybox:1.28

Good

---
# passes
apiVersion: v1
kind: Pod
metadata:
  name: pod-with-trusted-registry
spec:
  containers:
  - name: kyverno
    image: ghcr.io/kyverno/kyverno:latest

Check NVIDIA GPU is built in image

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: images-using-gpus
spec:
  validationFailureAction: enforce
  rules:
  - name: check-nvidia-gpus
    match:
      resources:
        kinds:
        - Pod
    preconditions:
      all:
      - key: "{{request.operation}}"
        operator: NotEquals
        value: DELETE
    validate:
      message: "Images which reserve NVIDIA GPUs must be built to use them."
      foreach:
      - list: "request.object.spec.containers"
        context: 
        - name: imageData
          imageRegistry: 
            reference: "{{ element.image }}"
        deny:
          conditions:
            all:
              # If a container image calls for an NVIDIA GPU in its resources.limits, it must also
              # have been built with the CUDA environment variable `NVIDIA_VISIBLE_DEVICES`.
              - key: "NVIDIA_VISIBLE_DEVICES=*?"
                operator: AnyNotIn
                value: "{{ imageData.configData.config.Env || '' }}"
              - key: "{{ element.resources.limits.\"nvidia.com/gpu\" || '' }}"
                operator: GreaterThan
                value: 0

Bad (blocked)

apiVersion: v1
kind: Pod
metadata:
  name: badpod01
spec:
  containers:
    - name: no-gpu
      image: busybox:1.28
      command: ["sleep","9999"]
      resources:
        limits:
          nvidia.com/gpu: 1

Good (allowed)

apiVersion: v1
kind: Pod
metadata:
  name: goodpod01
spec:
  containers:
    - name: gpu-example
      image: nvidia/cuda:11.6.0-devel-ubi8
      command: ["nvidia-smi"]
      resources:
        limits:
          nvidia.com/gpu: 1

apiVersion: v1
kind: Pod
metadata:
  name: goodpod02
spec:
  containers:
    - name: busybox
      image: busybox:1.28
      command: ["sleep","9999"]

apiVersion: v1
kind: Pod
metadata:
  name: goodpod03
spec:
  containers:
    - name: gpu-example-nolimits
      image: nvidia/cuda:11.6.0-devel-ubi8
      command: ["nvidia-smi"]

Require image source

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: image-source
spec:
  validationFailureAction: enforce
  rules:
  - name: check-source
    match:
      resources:
        kinds:
        - Pod
    preconditions:
      all:
      - key: "{{request.operation}}"
        operator: NotEquals
        value: DELETE
    validate:
      message: "The image source must be specified in a label or annotation."
      foreach:
      - list: "request.object.spec.containers"
        context: 
        - name: imageData
          imageRegistry: 
            reference: "{{ element.image }}"
            jmesPath: "{labels: configData.config.Labels, annotations: manifest.annotations}"
        deny:
          conditions:
            all:
              - key: "{{ imageData.labels.\"org.opencontainers.image.source\" || '' }}"
                operator: NotEquals
                value: "*?"
              - key: "{{ imageData.annotations.\"org.opencontainers.image.source\" || '' }}"
                operator: NotEquals
                value: "*?"

Good (allowed)

apiVersion: v1
kind: Pod
metadata:
  name: goodpod01
spec:
  containers:
    - name: test
      image: ghcr.io/kyverno/kyverno-annotations-example:latest

Bad (blocked)

apiVersion: v1
kind: Pod
metadata:
  name: badpod01
spec:
  containers:
    - name: test
      image: busybox:1.28

[JimBugwadia]

@chipzoller - would it be possible to detect if a container layer tries to replace something in a base image?

[chipzoller]

In the manifest, there's a layers object which contains size and digest info, so you could conceivably compare a reference image's specific layer to another version and see if there's divergence.

$ crane manifest ghcr.io/kyverno/kyverno-annotations-example:latest | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 945,
    "digest": "sha256:8cbb1e98c14b1aadac527cc6e1cbe116e18d2710babdd2ceae3d84ad9affc7bd"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 2479,
      "digest": "sha256:2db29710123e3e53a794f2694094b9b4338aa9ee5c40b930cb8063a1be392c54"
    }
  ],
  "annotations": {
    "org.opencontainers.image.base.name": "index.docker.io/library/hello-world:latest",
    "org.opencontainers.image.source": "https://github.com/kyverno/kyverno-examples"
  }
}