New signatures can't be verified

[raulcabello]

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

New signatures can't be verified. Something has changed with the new sigstore releases and we can no longer verify new signatures. Old signatures can still be verified.

2022-06-15T13:25:10.270978Z  INFO sigstore::cosign::signature_layers: Skipping OCI layer because of error error=CertificateValidityError("Not issued by a  trusted root")           

2022-06-15T13:25:10.271005Z ERROR validation{host="policy-server-default-5b45bcf784-79stq" policy_id="clusterwide-verify-image-signatures-policy" kind="Pod" kind_group="" kind_version="v1" name="privileged-pod" namespace="default" operation="CREATE" request_uid="39a73e4b-4869-49bd-8302-258a5791b3b8" resource="pods" resource_group="" resource_version="v1" subresource=""}:policy_eval:validate{self=PolicyEvaluator { id: "clusterwide-verify-image-signatures-poli cy", settings: {"signatures": Array([Object({"image": String("*"), "keyless": Array([Object({"issuer": String("https://token.actions.githubusercontent.com "), "subject": String("https://github.com/raulcabello/app-example/.github/workflows/ci.yml@refs/tags/v0.4.0")})])})])} }}: policy_evaluator::runtimes::wap c: callback evaluation failed policy_id=7 binding="kubewarden" operation="v1/verify" error="No Signature Layer passed verification"                       

Expected Behavior

Signatures can be verified

Steps To Reproduce

Create this policy:

apiVersion: policies.kubewarden.io/v1alpha2
kind: ClusterAdmissionPolicy
metadata:
  name: verify-image-signatures-policy
spec:
  module: registry://ghcr.io/kubewarden/policies/verify-image-signatures:v0.1.4
  rules:
  - apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
    operations:
    - CREATE
    - UPDATE
  mutating: true
  settings:
    signatures:
        - image: "*"
          keyless: 
            - issuer: "https://token.actions.githubusercontent.com"
              subject: "https://github.com/raulcabello/app-example/.github/workflows/ci.yml@refs/tags/v0.4.0"

and try to create the following pod:

kubectl apply -f - <<EOF   
apiVersion: v1
kind: Pod
metadata:
  name: privileged-pod
spec:
  containers:
    - name: nginx
      image: ghcr.io/raulcabello/app-example:v0.3.0
EOF

Environment

- OS:
- Architecture:

Anything else?

No response

[raulcabello]

This issue was fixed in https://github.com/sigstore/sigstore-rs/issues/70 I'll bump sigstore-rs to use the latest version

[zosocanuck]

Getting a similar issue with just public key verification (no keyless):

2022-09-23T18:59:40.265706Z  INFO sigstore::cosign::signature_layers: Skipping OCI layer because of error error=SigstoreRekorBundleNotFoundError
2022-09-23T18:59:40.265821Z ERROR validation{host="policy-server-default-ff596885c-xxsss" policy_id="clusterwide-verify-image-signatures-policy" kind="Pod" kind_group="" kind_version="v1" name="signed" namespace="default" operation="CREATE" request_uid="095016b8-c9e2-4969-9884-42c5f57fa4a8" resource="pods" resource_group="" resource_version="v1" subresource=""}:policy_eval:validate{self=PolicyEvaluator { id: "clusterwide-verify-image-signatures-policy", settings: {"signatures": Array([Object({"image": String("ghcr.io/zosocanuck/*"), "pubKeys": Array([String("-----BEGIN PUBLIC KEY-----MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEyyWAk9WaA9OkeZfGUnvW4Nb8Phf45p3r1IjdBLIkbYxq7ZSyj9YDihsmDvrDc4bAijgAx/RAyl0u3ElBurXLfQ==-----END PUBLIC KEY-----")])})])} }}: policy_evaluator::runtimes::wapc: callback evaluation failed policy_id=5 binding="kubewarden" operation="v2/verify" error="No Signature Layer passed verification"

relevant policy is as follows:

settings:
    signatures:
      - image: "ghcr.io/zosocanuck/*" # match all tags 
        pubKeys:
          - "-----BEGIN PUBLIC KEY-----MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEyyWAk9WaA9OkeZfGUnvW4Nb8Phf45p3r1IjdBLIkbYxq7ZSyj9YDihsmDvrDc4bAijgAx/RAyl0u3ElBurXLfQ==-----END PUBLIC KEY-----"

[raulcabello]

@zosocanuck I can't reproduce the issue with the latest version. Can you please provide more information about your environment? which version of Kubewarden and verify-image-signatures are you using? Provide the image you are trying to verify if possible (from what I can see it is in ghcr, if it is public I can try to replicate the issue with the same image)

[zosocanuck]

@raulcabello Testing with minikube v1.27, kubewarden/policy-server:v1.2, kubewarden/kubewarden-controler:v1.1.1 and verify-image-signatures:v0.1.7. I'm following the latest quick start guide.

image is ghcr.io/zosocanuck/cert-manager-dashboard:0.1

cosign signature looks good via cosign tree as well as cosign verify

[raulcabello]

Thanks @zosocanuck ! I can reproduce the issue with ghcr.io/zosocanuck/cert-manager-dashboard:0.1 However I can't reproduce it with images I sign with cosign. I can verify it with cosign, so there is probably something wrong in our end.

Can you please tell me how you signed the image and which version of cosign you used? Did you use a keypair generated with cosign? If you look at the annotations for the layers with crane manifest $(cosign triangulate ghcr.io/zosocanuck/cert-manager-dashboard:0.1) | jq there is an annotation dev.sigstore.cosign/certificate, I would like to understand how this annotation was created. Did you use the --certificate flag when you signed the image?

[zosocanuck]

@raulcabello Yes, there is a dev.sigstore.cosign/certificate annotation. I used cosign sign -key "pkcs11:..." ghcr.io/zosocanuck/cert-manager-dashboard:0.1 without the --certificate flag. Signing private key is based out of a PKCS#11 provider.

cosign version is v1.12.1

[flavio]

This is a sigstore-rs issue. @raulcabello: can you open an issue against it please?

[raulcabello]

done https://github.com/sigstore/sigstore-rs/issues/135