Sigstore verification does not work

[RobusTetus]

I have followed carefully the README.md to set up verification. Just for the record, here is what I did.

1. Created /etc/pki/containers
2. used curl to get .pub (The url in README is incorrect, there is an extra 'o' in toolbx) githubusercontent.com/toolbox-images/images/main/quay.io-toolbx-images.pub
3. moved the correct .pub containing the signature into /etc/pki/containers/
4. Ran sudo restorecon -RFv /etc/pki/containers
5. Used nano to make /etc/containers/registries.d/quay.io-toolbx-images.yaml
6. Ran sudo restorecon -RFv /etc/containers/registries.d/quay.io-toolbx-images.yaml
7. Used nano to edit /etc/containers/policy.json which now contains the following:

   {
   "default": [
       {
           "type": "reject"
       }
   ],
   "transports": {
       "docker": {
           "quay.io/toolbx-images": [
               {
                   "type": "sigstoreSigned",
                   "keyPath": "/etc/pki/containers/quay.io-toolbx-images.pub",
                   "signedIdentity": {
                       "type": "matchRepository"
                   }
               }
           ],
       "registry.access.redhat.com": [
       {
           "type": "signedBy",
           "keyType": "GPGKeys",
           "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
       }
       ],
       "registry.redhat.io": [
       {
           "type": "signedBy",
           "keyType": "GPGKeys",
           "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
       }
       ],
           "": [
               {
                   "type": "insecureAcceptAnything"
               }
           ]
   },
       "docker-daemon": {
       "": [
       {
           "type": "insecureAcceptAnything"
       }
       ]
   }
   }
   }

   After this. Trying to download a fresh image with toolbox --verbose create --image quay.io/toolbx-images/ubuntu-toolbox:23.04 reports following:

   Download quay.io/toolbx-images/ubuntu-toolbox:23.04 (500MB)? [y/N]: y
   DEBU Pulling image quay.io/toolbx-images/ubuntu-toolbox:23.04 
   Trying to pull quay.io/toolbx-images/ubuntu-toolbox:23.04...
   Error: copying system image from manifest list: Source image rejected: A signature was required, but no signature exists
   Error: failed to pull image quay.io/toolbx-images/ubuntu-toolbox:23.04

Just to be sure I checked /etc/pki/containers/quay.io-toolbx-images.pub and it does contain signature matching with this repo.

-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEQr63Nsc66mA3oGMArrQPm8/AkuTO
K+ZrK3WCWzx00LW5K1qu+BS3U4eyMmXaFKIsX69PEFZWzXKy9psum8wj9Q==
-----END PUBLIC KEY-----

[travier]

Hum, this is weird. I will have to look at this.

[travier]

Hum, I indeed have the same error with podman but cosign tells me the images signatures are valid. 🤔

[RobusTetus]

Hum, I indeed have the same error with podman but cosign tells me the images signatures are valid. 🤔

Yes, I have noticed on quay.io that the cosign signatures are indeed there. I have read redhat docs on sigstore verification but there is nothing different from what you have in README.md. I am genuinely confused on what is breaking.

[travier]

podman pull  --log-level=debug quay.io/toolbx-images/centos-toolbox:stream9
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level=debug quay.io/toolbx-images/centos-toolbox:stream9) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using boltdb as database backend             
DEBU[0000] Initializing boltdb state at /var/home/tim/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/tim/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/tim/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/tim/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 25             
DEBU[0000] Pulling image quay.io/toolbx-images/centos-toolbox:stream9 (policy: always) 
DEBU[0000] Looking up image "quay.io/toolbx-images/centos-toolbox:stream9" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "quay.io/toolbx-images/centos-toolbox:stream9" ... 
DEBU[0000] parsed reference into "[overlay@/var/home/tim/.local/share/containers/storage+/run/user/1000/containers]@8b5cb3b4ad68ab1d5014c4009bfc60f845f0f41b1c89022d04033a21d490b399" 
DEBU[0000] Found image "quay.io/toolbx-images/centos-toolbox:stream9" as "quay.io/toolbx-images/centos-toolbox:stream9" in local containers storage 
DEBU[0000] Found image "quay.io/toolbx-images/centos-toolbox:stream9" as "quay.io/toolbx-images/centos-toolbox:stream9" in local containers storage ([overlay@/var/home/tim/.local/share/containers/storage+/run/user/1000/containers]@8b5cb3b4ad68ab1d5014c4009bfc60f845f0f41b1c89022d04033a21d490b399) 
DEBU[0000] exporting opaque data as blob "sha256:8b5cb3b4ad68ab1d5014c4009bfc60f845f0f41b1c89022d04033a21d490b399" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Attempting to pull candidate quay.io/toolbx-images/centos-toolbox:stream9 for quay.io/toolbx-images/centos-toolbox:stream9 
DEBU[0000] parsed reference into "[overlay@/var/home/tim/.local/share/containers/storage+/run/user/1000/containers]quay.io/toolbx-images/centos-toolbox:stream9" 
Trying to pull quay.io/toolbx-images/centos-toolbox:stream9...
DEBU[0000] Copying source image //quay.io/toolbx-images/centos-toolbox:stream9 to destination image [overlay@/var/home/tim/.local/share/containers/storage+/run/user/1000/containers]quay.io/toolbx-images/centos-toolbox:stream9 
DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Trying to access "quay.io/toolbx-images/centos-toolbox:stream9" 
DEBU[0000] No credentials matching quay.io/toolbx-images/centos-toolbox found in /run/user/1000/containers/auth.json 
DEBU[0000] No credentials matching quay.io/toolbx-images/centos-toolbox found in /var/home/tim/.config/containers/auth.json 
DEBU[0000] Found credentials for quay.io/toolbx-images/centos-toolbox in credential helper containers-auth.json in file /var/home/tim/.docker/config.json 
DEBU[0000]  Lookaside configuration: using "docker" namespace quay.io/toolbx-images 
DEBU[0000]  No signature storage configuration found for quay.io/toolbx-images/centos-toolbox:stream9, using built-in default file:///var/home/tim/.local/share/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/quay.io 
DEBU[0000]  Sigstore attachments: using "docker" namespace quay.io/toolbx-images 
DEBU[0000] GET https://quay.io/v2/                      
DEBU[0000] Ping https://quay.io/v2/ status 401          
DEBU[0000] GET https://quay.io/v2/auth?account=openshift-release-dev%2Bocm_access_c273414d3a374a04b57d071678c1f310&scope=repository%3Atoolbx-images%2Fcentos-toolbox%3Apull&service=quay.io 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] GET https://quay.io/v2/toolbx-images/centos-toolbox/manifests/stream9 
DEBU[0000] Content-Type from manifest GET is "application/vnd.oci.image.index.v1+json" 
DEBU[0000] Using SQLite blob info cache at /var/home/tim/.local/share/containers/cache/blob-info-cache-v1.sqlite 
DEBU[0000] Source is a manifest list; copying (only) instance sha256:d95326eb8f36e881f449e106efa70cea95304fb5ac9e8add538d1ee4f57b2246 for current system 
DEBU[0000] GET https://quay.io/v2/toolbx-images/centos-toolbox/manifests/sha256:d95326eb8f36e881f449e106efa70cea95304fb5ac9e8add538d1ee4f57b2246 
DEBU[0001] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json" 
DEBU[0001] IsRunningImageAllowed for image docker:quay.io/toolbx-images/centos-toolbox:stream9 
DEBU[0001]  Using transport "docker" specific policy section quay.io/toolbx-images 
DEBU[0001] Reading /var/home/tim/.local/share/containers/sigstore/toolbx-images/centos-toolbox@sha256=d95326eb8f36e881f449e106efa70cea95304fb5ac9e8add538d1ee4f57b2246/signature-1 
DEBU[0001] Looking for sigstore attachments in quay.io/toolbx-images/centos-toolbox:sha256-d95326eb8f36e881f449e106efa70cea95304fb5ac9e8add538d1ee4f57b2246.sig 
DEBU[0001] GET https://quay.io/v2/toolbx-images/centos-toolbox/manifests/sha256-d95326eb8f36e881f449e106efa70cea95304fb5ac9e8add538d1ee4f57b2246.sig 
DEBU[0001] Content-Type from manifest GET is "application/json" 
DEBU[0001] Fetching sigstore attachment manifest failed, assuming it does not exist: reading manifest sha256-d95326eb8f36e881f449e106efa70cea95304fb5ac9e8add538d1ee4f57b2246.sig in quay.io/toolbx-images/centos-toolbox: manifest unknown 
DEBU[0001] Requirement 0: denied, done                  
DEBU[0001] Error pulling candidate quay.io/toolbx-images/centos-toolbox:stream9: copying system image from manifest list: Source image rejected: A signature was required, but no signature exists 
Error: copying system image from manifest list: Source image rejected: A signature was required, but no signature exists
DEBU[0001] Shutting down engines

[travier]

Looks like this is related to multi-arch images. I have the following other repositories that work but they are x86_64 only: https://github.com/travier/quay-containerfiles

In my test repo (https://github.com/travier/cosign-test), the cosign example does not "work" but the podman one does:

# Validates using cosign but podman pull fails
$ cosign verify --key quay.io-travier-containers.pub quay.io/travier/cosign-example:latest-cosign
$ podman pull --log-level=debug quay.io/travier/cosign-example:latest-cosign
# Does not validate using cosign but podman pull works
$ cosign verify --key quay.io-travier-containers.pub quay.io/travier/cosign-example:latest-podman
$ podman pull --log-level=debug quay.io/travier/cosign-example:latest-podman

[RobusTetus]

Looks like this is related to multi-arch images.

My device is x86_64 too, so podman should follow my architecture when pulling an image, right?

However when searching I have stumbled upon this blog. Where when pushing the image, they are using --remove-signatures flag.

By default the Podman ecosystem treats signatures as an integral part of the image. So “–remove-signatures” is necessary to make a copy that doesn’t include the signatures, including copies that don’t preserve the images bit-for-bit-exactly.

Maybe sigstore/cosign-installer@v3.3.0 you are using in the workflow is not signing/pushing images this way :thinking:. At this point I am honestly just guessing. :laughing:

[travier]

Confirmed with:

$ podman pull --log-level=debug quay.io/travier/cosign-example:latest-cosign-x86_64

which works when the multi-arch image does not. So this is likely an issue in how podman verifies the signature for multi-arch containers. Or we need to build and sign our multi-arch containers differently using cosign.

[travier]

I've filed: https://github.com/containers/podman/issues/21209

[travier]

Looks like this is working now! The Fedora & Ubuntu images are still building but I just pulled the CentOS one and it worked for me. Thanks a lot for the report!