kyverno / kyverno

Cloud Native Policy Management
https://kyverno.io
Apache License 2.0
5.67k stars 862 forks source link

[Bug] No the error message/note field in the Kyverno logs when using Validate Rules (version 1.12) #10170

Open TheKnowledgeKeeper opened 5 months ago

TheKnowledgeKeeper commented 5 months ago

Kyverno Version

1.12

Kubernetes Version

1.24

Kubernetes Platform

GKE

Description

In Kyverno version 1.12, I don't see any msg or note fields in Kyverno logs that need to be displayed when using the validate rule type. But in version 1.11 it is fully available.

My policy

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: slsa-verify-all-images
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  background: true
  rules:
  - name: slsa-restrict-third-party-images
    match:
      any:
      - resources:
          kinds:
          - Pod
          operations:
          - CREATE
          - UPDATE
    exclude:
      any:
      - resources:
          kinds:
            - ReplicaSet
      - resources:
          namespaces:
            - kyverno
            - kube-system
    skipBackgroundRequests: true
    validate:
      message: "Your third-party images have not been whitelisted {{ request.object.spec.[ephemeralContainers, initContainers, containers][].image }}"
      foreach:
      - list: "request.object.spec.[ephemeralContainers, initContainers, containers][]"
        context: 
        - name: imageData
          imageRegistry: 
            reference: "{{ element.image }}"
        deny:
          conditions:
            all:
              - key: '{{ imageData.registry || "" }}'
                operator: NotEquals
                value: gcr.io
              - key: '{{ imageData.resolvedImage || "" }}'
                operator: AnyNotIn
                value:
                - index.docker.io/apache/airflow@sha256:*
                - index.docker.io/apache/superset@sha256:*
                - index.docker.io/bitnami/git@sha256:*
                - index.docker.io/bitnami/postgresql@sha256:*
                - index.docker.io/bitnami/redis@sha256:*
                - index.docker.io/curlimages/curl@sha256:*

Manifest of my pod applied

apiVersion: v1
kind: Pod
metadata:
  name: taile-test-kyverno-pod
  namespace: default
  labels:
    app: taile-test-kyverno-pod
spec:
  containers:
  - name: taile
    image: python:alpine3.19
    tty: true

The error message I get from the console screen

Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request:

resource Pod/default/taile-test-kyverno-pod was blocked due to the following policies

slsa-verify-all-images:
  slsa-restrict-third-party-images: 'validation failure: Your third-party images have
    not been whitelisted ["python:alpine3.19"]'

Below are the types of logs I get for each version

Version 1.11 - This log has the note field describing the error message

{
    "@timestamp": 1714365065.994449,
    "k8s_container_hash": "ghcr.io/kyverno/kyverno@sha256:76d0252892bab3b7682444f283660274977a6243450720a3893221b510eed94e",
    "container_name": "/k8s_kyverno_kyverno-admission-controller-8694857c99-zfw9t_kyverno",
    "k8s_container_name": "kyverno",
    "k8s_namespace_name": "kyverno",
    "k8s_docker_id": "8100b91c1586326d7180edae8ef298d3c2050dea67d5131cc2247e42ce53d905",
    "k8s_workload_path": "staging.kyverno.kyverno-admission-controller",
    "_p": "F",
    "k8s_container_image": "ghcr.io/kyverno/kyverno:v1.11.4",
    "stream": "stderr",
    "time": "2024-04-29T04:31:05.994449375Z",
    "k8s_pod_name": "kyverno-admission-controller-8694857c99-zfw9t",
    "fluentd_event_source": "flb-kafka-26mzb",
    "log":
    {
        "level": "info",
        "ts": 1714365065.9941533,
        "logger": "klog",
        "caller": "events/event_broadcaster.go:338",
        "msg": "Event occurred",
        "object":
        {
            "name": "imageref-demo",
            "namespace": "watchtower-v"
        },
        "kind": "Policy",
        "apiVersion": "kyverno.io/v1",
        "type": "Warning",
        "reason": "PolicyViolation",
        "action": "Resource Blocked",
        "note": "Pod watchtower-v/taile-test-kyverno-pod: [no-root-images] fail (blocked); validation failure: Your third-party images have not been whitelisted [\"python:alpine3.19\"]"
    },
    "k8s_pod_id": "a869eead-58c6-417b-be2f-6fa9611b1407",
    "k8s_cluster_name": "staging",
    "k8s_host": "gke-k8s-prod-staging-np-e2-4-8-2d44b41e-snjw",
    "k8s_labels":
    {
        "app.kubernetes.io/instance": "kyverno",
        "app.kubernetes.io/managed-by": "Helm",
        "app.kubernetes.io/part-of": "kyverno",
        "app.kubernetes.io/version": "3.1.4",
        "pod-template-hash": "8694857c99",
        "app.kubernetes.io/component": "admission-controller",
        "helm.sh/chart": "kyverno-3.1.4"
    },
    "container_img": "ghcr.io/kyverno/kyverno@sha256:76d0252892bab3b7682444f283660274977a6243450720a3893221b510eed94e"
}

Version 1.12 - I get 4 logs but no field describing the error message: Your third-party images have not been whitelisted ...

{
        "level": "Level(-2)",
        "ts": "2024-05-03T14:32:35Z",
        "logger": "webhooks.resource.validate",
        "caller": "validation/validation.go:116",
        "msg": "validation failed",
        "gvk": "/v1, Kind=Pod",
        "gvr":
        {
            "group": "",
            "version": "v1",
            "resource": "pods"
        },
        "namespace": "default",
        "name": "taile-test-kyverno-pod",
        "operation": "CREATE",
        "uid": "4a68c2d5-3efc-4a67-9045-73dfa7c726cd",
        "user":
        {
            "username": "tai.le@xyz.com",
            "groups":
            [
                "system:authenticated"
            ],
            "extra":
            {
                "iam.gke.io/user-assertion":
                [
                    "<snipped>"
                ],
                "user-assertion.cloud.google.com":
                [
                    "<snipped>"
                ]
            }
        },
        "roles":
        [],
        "clusterroles":
        [
            "system:basic-user",
            "system:discovery",
            "system:public-info-viewer"
        ],
        "resource.gvk": "/v1, Kind=Pod",
        "kind": "Pod",
        "URLParams": "",
        "action": "Enforce",
        "resource": "default/Pod/taile-test-kyverno-pod",
        "policy": "slsa-verify-all-images",
        "failed rules":
        [
            "slsa-restrict-third-party-images"
        ]
    }
{
        "level": "Level(-2)",
        "ts": "2024-05-03T14:32:36Z",
        "logger": "webhooks.resource.validate",
        "caller": "validation/validation.go:116",
        "msg": "validation failed",
        "gvk": "/v1, Kind=Pod",
        "gvr":
        {
            "group": "",
            "version": "v1",
            "resource": "pods"
        },
        "namespace": "default",
        "name": "taile-test-kyverno-pod",
        "operation": "CREATE",
        "uid": "4a68c2d5-3efc-4a67-9045-73dfa7c726cd",
        "user":
        {
            "username": "tai.le@xyz.com",
            "groups":
            [
                "system:authenticated"
            ],
            "extra":
            {
                "iam.gke.io/user-assertion":
                [
                    "<snipped>"
                ],
                "user-assertion.cloud.google.com":
                [
                    "<snipped>"
                ]
            }
        },
        "roles":
        [],
        "clusterroles":
        [
            "system:basic-user",
            "system:discovery",
            "system:public-info-viewer"
        ],
        "resource.gvk": "/v1, Kind=Pod",
        "kind": "Pod",
        "URLParams": "",
        "action": "Enforce",
        "resource": "default/Pod/taile-test-kyverno-pod",
        "policy": "slsa-verify-all-images",
        "failed rules":
        [
            "slsa-restrict-third-party-images"
        ]
    }
{
        "level": "Level(-2)",
        "ts": "2024-05-03T14:32:36Z",
        "logger": "webhooks.resource.validate",
        "caller": "utils/block.go:29",
        "msg": "blocking admission request",
        "gvk": "/v1, Kind=Pod",
        "gvr":
        {
            "group": "",
            "version": "v1",
            "resource": "pods"
        },
        "namespace": "default",
        "name": "taile-test-kyverno-pod",
        "operation": "CREATE",
        "uid": "4a68c2d5-3efc-4a67-9045-73dfa7c726cd",
        "user":
        {
            "username": "tai.le@xyz.com",
            "groups":
            [
                "system:authenticated"
            ],
            "extra":
            {
                "iam.gke.io/user-assertion":
                [
                    "<snipped>"
                ],
                "user-assertion.cloud.google.com":
                [
                    "<snipped>"
                ]
            }
        },
        "roles":
        [],
        "clusterroles":
        [
            "system:basic-user",
            "system:discovery",
            "system:public-info-viewer"
        ],
        "resource.gvk": "/v1, Kind=Pod",
        "kind": "Pod",
        "URLParams": "",
        "action": "validate",
        "resource": "default/Pod/taile-test-kyverno-pod",
        "policy": "slsa-verify-all-images"
    }
{
        "level": "info",
        "ts": "2024-05-03T14:32:36Z",
        "logger": "webhooks.resource.validate",
        "caller": "resource/handlers.go:146",
        "msg": "admission request denied",
        "gvk": "/v1, Kind=Pod",
        "gvr":
        {
            "group": "",
            "version": "v1",
            "resource": "pods"
        },
        "namespace": "default",
        "name": "taile-test-kyverno-pod",
        "operation": "CREATE",
        "uid": "4a68c2d5-3efc-4a67-9045-73dfa7c726cd",
        "user":
        {
            "username": "tai.le@xyz.com",
            "groups":
            [
                "system:authenticated"
            ],
            "extra":
            {
                "iam.gke.io/user-assertion":
                [
                    "<snipped>"
                ],
                "user-assertion.cloud.google.com":
                [
                    "<snipped>"
                ]
            }
        },
        "roles":
        [],
        "clusterroles":
        [
            "system:basic-user",
            "system:discovery",
            "system:public-info-viewer"
        ],
        "resource.gvk": "/v1, Kind=Pod",
        "kind": "Pod",
        "URLParams": ""
    }

Steps to reproduce

  1. Upgrade kyverno version to v1.12 .
  2. Apply My policy in above description .
  3. Use kubectl to capture logs kubectl logs -f -l app.kubernetes.io/part-of=kyverno -n kyverno | grep "taile-test-kyverno-pod" (taile-test-kyverno-pod is my pod name) .
  4. Apply Manifest of my pod .
  5. Check logs get from step 3 .

Expected behavior

I would like the log information to be fully described as in v1.11, which I showed above.

"log":
    {
        "level": "info",
        "ts": 1714365065.9941533,
        "logger": "klog",
        "caller": "events/event_broadcaster.go:338",
        "msg": "Event occurred",
        "object":
        {
            "name": "imageref-demo",
            "namespace": "watchtower-v"
        },
        "kind": "Policy",
        "apiVersion": "kyverno.io/v1",
        "type": "Warning",
        "reason": "PolicyViolation",
        "action": "Resource Blocked",
        "note": "Pod watchtower-v/taile-test-kyverno-pod: [no-root-images] fail (blocked); validation failure: Your third-party images have not been whitelisted [\"python:alpine3.19\"]"
    },

Screenshots

No response

Kyverno logs

No response

Slack discussion

No response

Troubleshooting

chipzoller commented 5 months ago

Not a policy issue but code issue. Transferred to kyverno/kyverno.

Manoramsharma commented 5 months ago

/assign

praddy26 commented 1 week ago

/assign