Bug Report

Kyverno incorrectly blocks the canary deployment of the image produced by the sample-pipeline

Current Behavior

The pod for the picalc deployment is never created in the prod namespace.

Inspecting the kyverno logs yields the following warning message:

I0407 14:48:47.004013       1 event.go:294] "Event occurred" object="attest-code-review" fieldPath="" kind="ClusterPolicy" apiVersion="kyverno.io/v1" type="Warning" reason="PolicyViolation" message=<
    Pod prod/: [attest-code-review] fail (blocked); failed to verify image ttl.sh/0198a0cce14eb56330031c3d351561fc/example-sample@sha256:31caf278e83a77d1bab5255e9fca3ebec4aea1f2fe03624561504d7fa224272d: no matching attestations:
    no certificate found on attestation
     no certificate found on attestation
 >

But the following checks are both successful:

cosign verify --insecure-ignore-tlog --key k8s://tekton-chains/signing-secrets "${IMAGE_URL}"
cosign verify-attestation --insecure-ignore-tlog --type slsaprovenance --key k8s://tekton-chains/signing-secrets "${IMAGE_URL}"

Expected Behavior

The canary deploy should yield a running pod, e.g.:

$ kubectl get all -n prod
NAME                          READY   STATUS    RESTARTS   AGE
pod/picalc-576dd6b788-sszmh   1/1     Running   0          32s

NAME             TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/picalc   NodePort   10.107.77.128   <none>        8080:30907/TCP   37s

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/picalc   1/1     1            1           38s

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/picalc-576dd6b788   1         1         1       38s

Steps to Reproduce

make setup-frsca
make example-sample-pipeline

Context

This issue is not visible on the main CI build because it uses an internal registry instead of ttl.sh. The current attest-code-review policy does not have entry for the internal registry.

k get clusterpolicy attest-code-review -n kyverno -o yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: attest-code-review
spec:
  background: false
  failurePolicy: Fail
  rules:
  - context:
    - configMap:
        name: keys
        namespace: default
      name: keys
    match:
      resources:
        kinds:
        - Pod
        namespaces:
        - tekton-pipelines
        - tekton-chains
        - prod
    name: attest-code-review
    verifyImages:
    - attestations:
      - conditions:
        - all:
          - key: '{{ builder.id }}'
            operator: Equals
            value: https://tekton.dev/chains/v2
          - key: '{{ buildType }}'
            operator: Equals
            value: tekton.dev/v1beta1/TaskRun
        predicateType: https://slsa.dev/provenance/v0.2
      image: ttl.sh/*
      key: '{{ keys.data.ttlsh }}'
      mutateDigest: true
      required: true
      verifyDigest: true
    - attestations:
      - conditions:
        - all:
          - key: '{{ builder.id }}'
            operator: Equals
            value: https://tekton.dev/chains/v2
          - key: '{{ buildType }}'
            operator: Equals
            value: https://tekton.dev/attestations/chains@v2
        predicateType: https://slsa.dev/provenance/v0.2
      image: gcr.io/tekton-releases/github.com/tektoncd/*
      key: '{{ keys.data.tektoncd }}'
      mutateDigest: true
      required: true
      verifyDigest: true
  validationFailureAction: enforce
  webhookTimeoutSeconds: 30

Your Environment

Last commit:
  a264f57 fixed broken link (#429)  (HEAD -> main, origin/main, origin/HEAD)
Kubernetes:
  Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.3", GitCommit:"9e644106593f3f4aa98f8a84b23db5fa378900bd", GitTreeState:"clean", BuildDate:"2023-03-15T13:33:11Z", GoVersion:"go1.19.7", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.4+k3s1", GitCommit:"0dc63334c0db3e7b99244427615e091909fc486e", GitTreeState:"clean", BuildDate:"2022-11-18T18:14:41Z", GoVersion:"go1.19.3", Compiler:"gc", Platform:"linux/arm64"}
Tekton:
  Client version: 0.30.0
Chains version: v0.15.0
Pipeline version: v0.41.0
Triggers version: v0.22.0

buildsec / frsca

Kyverno incorrectly blocks sample-pipeline canary deploy #430