search

toolbx-images / images

Community maintained container images to use with toolbx and distrobox
https://containertoolbx.org/
Apache License 2.0
323 stars 29 forks source link

Sigstore verification does not work #113

Closed RobusTetus closed 8 months ago

RobusTetus commented 8 months ago

I have followed carefully the README.md to set up verification. Just for the record, here is what I did.

  1. Created /etc/pki/containers
  2. used curl to get .pub (The url in README is incorrect, there is an extra 'o' in toolbx) githubusercontent.com/toolbox-images/images/main/quay.io-toolbx-images.pub
  3. moved the correct .pub containing the signature into /etc/pki/containers/
  4. Ran sudo restorecon -RFv /etc/pki/containers
  5. Used nano to make /etc/containers/registries.d/quay.io-toolbx-images.yaml
  6. Ran sudo restorecon -RFv /etc/containers/registries.d/quay.io-toolbx-images.yaml
  7. Used nano to edit /etc/containers/policy.json which now contains the following: 
    {
"default": [
    {
        "type": "reject"
    }
],
"transports": {
    "docker": {
        "quay.io/toolbx-images": [
            {
                "type": "sigstoreSigned",
                "keyPath": "/etc/pki/containers/quay.io-toolbx-images.pub",
                "signedIdentity": {
                    "type": "matchRepository"
                }
            }
        ],
    "registry.access.redhat.com": [
    {
        "type": "signedBy",
        "keyType": "GPGKeys",
        "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
    }
    ],
    "registry.redhat.io": [
    {
        "type": "signedBy",
        "keyType": "GPGKeys",
        "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
    }
    ],
        "": [
            {
                "type": "insecureAcceptAnything"
            }
        ]
},
    "docker-daemon": {
    "": [
    {
        "type": "insecureAcceptAnything"
    }
    ]
}
}
}

    After this. Trying to download a fresh image with toolbox --verbose create --image quay.io/toolbx-images/ubuntu-toolbox:23.04 reports following:

    Download quay.io/toolbx-images/ubuntu-toolbox:23.04 (500MB)? [y/N]: y
DEBU Pulling image quay.io/toolbx-images/ubuntu-toolbox:23.04 
Trying to pull quay.io/toolbx-images/ubuntu-toolbox:23.04...
Error: copying system image from manifest list: Source image rejected: A signature was required, but no signature exists
Error: failed to pull image quay.io/toolbx-images/ubuntu-toolbox:23.04

Just to be sure I checked /etc/pki/containers/quay.io-toolbx-images.pub and it does contain signature matching with this repo.

-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEQr63Nsc66mA3oGMArrQPm8/AkuTO
K+ZrK3WCWzx00LW5K1qu+BS3U4eyMmXaFKIsX69PEFZWzXKy9psum8wj9Q==
-----END PUBLIC KEY-----
travier commented 8 months ago

Hum, this is weird. I will have to look at this.

travier commented 8 months ago

Hum, I indeed have the same error with podman but cosign tells me the images signatures are valid. 🤔

RobusTetus commented 8 months ago

Hum, I indeed have the same error with podman but cosign tells me the images signatures are valid. 🤔

Yes, I have noticed on quay.io that the cosign signatures are indeed there. I have read redhat docs on sigstore verification but there is nothing different from what you have in README.md. I am genuinely confused on what is breaking.

travier commented 8 months ago 
podman pull  --log-level=debug quay.io/toolbx-images/centos-toolbox:stream9
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level=debug quay.io/toolbx-images/centos-toolbox:stream9) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using boltdb as database backend             
DEBU[0000] Initializing boltdb state at /var/home/tim/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/tim/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/tim/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/tim/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 25             
DEBU[0000] Pulling image quay.io/toolbx-images/centos-toolbox:stream9 (policy: always) 
DEBU[0000] Looking up image "quay.io/toolbx-images/centos-toolbox:stream9" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "quay.io/toolbx-images/centos-toolbox:stream9" ... 
DEBU[0000] parsed reference into "[overlay@/var/home/tim/.local/share/containers/storage+/run/user/1000/containers]@8b5cb3b4ad68ab1d5014c4009bfc60f845f0f41b1c89022d04033a21d490b399" 
DEBU[0000] Found image "quay.io/toolbx-images/centos-toolbox:stream9" as "quay.io/toolbx-images/centos-toolbox:stream9" in local containers storage 
DEBU[0000] Found image "quay.io/toolbx-images/centos-toolbox:stream9" as "quay.io/toolbx-images/centos-toolbox:stream9" in local containers storage ([overlay@/var/home/tim/.local/share/containers/storage+/run/user/1000/containers]@8b5cb3b4ad68ab1d5014c4009bfc60f845f0f41b1c89022d04033a21d490b399) 
DEBU[0000] exporting opaque data as blob "sha256:8b5cb3b4ad68ab1d5014c4009bfc60f845f0f41b1c89022d04033a21d490b399" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Attempting to pull candidate quay.io/toolbx-images/centos-toolbox:stream9 for quay.io/toolbx-images/centos-toolbox:stream9 
DEBU[0000] parsed reference into "[overlay@/var/home/tim/.local/share/containers/storage+/run/user/1000/containers]quay.io/toolbx-images/centos-toolbox:stream9" 
Trying to pull quay.io/toolbx-images/centos-toolbox:stream9...
DEBU[0000] Copying source image //quay.io/toolbx-images/centos-toolbox:stream9 to destination image [overlay@/var/home/tim/.local/share/containers/storage+/run/user/1000/containers]quay.io/toolbx-images/centos-toolbox:stream9 
DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Trying to access "quay.io/toolbx-images/centos-toolbox:stream9" 
DEBU[0000] No credentials matching quay.io/toolbx-images/centos-toolbox found in /run/user/1000/containers/auth.json 
DEBU[0000] No credentials matching quay.io/toolbx-images/centos-toolbox found in /var/home/tim/.config/containers/auth.json 
DEBU[0000] Found credentials for quay.io/toolbx-images/centos-toolbox in credential helper containers-auth.json in file /var/home/tim/.docker/config.json 
DEBU[0000]  Lookaside configuration: using "docker" namespace quay.io/toolbx-images 
DEBU[0000]  No signature storage configuration found for quay.io/toolbx-images/centos-toolbox:stream9, using built-in default file:///var/home/tim/.local/share/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/quay.io 
DEBU[0000]  Sigstore attachments: using "docker" namespace quay.io/toolbx-images 
DEBU[0000] GET https://quay.io/v2/                      
DEBU[0000] Ping https://quay.io/v2/ status 401          
DEBU[0000] GET https://quay.io/v2/auth?account=openshift-release-dev%2Bocm_access_c273414d3a374a04b57d071678c1f310&scope=repository%3Atoolbx-images%2Fcentos-toolbox%3Apull&service=quay.io 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] GET https://quay.io/v2/toolbx-images/centos-toolbox/manifests/stream9 
DEBU[0000] Content-Type from manifest GET is "application/vnd.oci.image.index.v1+json" 
DEBU[0000] Using SQLite blob info cache at /var/home/tim/.local/share/containers/cache/blob-info-cache-v1.sqlite 
DEBU[0000] Source is a manifest list; copying (only) instance sha256:d95326eb8f36e881f449e106efa70cea95304fb5ac9e8add538d1ee4f57b2246 for current system 
DEBU[0000] GET https://quay.io/v2/toolbx-images/centos-toolbox/manifests/sha256:d95326eb8f36e881f449e106efa70cea95304fb5ac9e8add538d1ee4f57b2246 
DEBU[0001] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json" 
DEBU[0001] IsRunningImageAllowed for image docker:quay.io/toolbx-images/centos-toolbox:stream9 
DEBU[0001]  Using transport "docker" specific policy section quay.io/toolbx-images 
DEBU[0001] Reading /var/home/tim/.local/share/containers/sigstore/toolbx-images/centos-toolbox@sha256=d95326eb8f36e881f449e106efa70cea95304fb5ac9e8add538d1ee4f57b2246/signature-1 
DEBU[0001] Looking for sigstore attachments in quay.io/toolbx-images/centos-toolbox:sha256-d95326eb8f36e881f449e106efa70cea95304fb5ac9e8add538d1ee4f57b2246.sig 
DEBU[0001] GET https://quay.io/v2/toolbx-images/centos-toolbox/manifests/sha256-d95326eb8f36e881f449e106efa70cea95304fb5ac9e8add538d1ee4f57b2246.sig 
DEBU[0001] Content-Type from manifest GET is "application/json" 
DEBU[0001] Fetching sigstore attachment manifest failed, assuming it does not exist: reading manifest sha256-d95326eb8f36e881f449e106efa70cea95304fb5ac9e8add538d1ee4f57b2246.sig in quay.io/toolbx-images/centos-toolbox: manifest unknown 
DEBU[0001] Requirement 0: denied, done                  
DEBU[0001] Error pulling candidate quay.io/toolbx-images/centos-toolbox:stream9: copying system image from manifest list: Source image rejected: A signature was required, but no signature exists 
Error: copying system image from manifest list: Source image rejected: A signature was required, but no signature exists
DEBU[0001] Shutting down engines
travier commented 8 months ago

Looks like this is related to multi-arch images. I have the following other repositories that work but they are x86_64 only: https://github.com/travier/quay-containerfiles

In my test repo (https://github.com/travier/cosign-test), the cosign example does not "work" but the podman one does:

# Validates using cosign but podman pull fails
$ cosign verify --key quay.io-travier-containers.pub quay.io/travier/cosign-example:latest-cosign
$ podman pull --log-level=debug quay.io/travier/cosign-example:latest-cosign
# Does not validate using cosign but podman pull works
$ cosign verify --key quay.io-travier-containers.pub quay.io/travier/cosign-example:latest-podman
$ podman pull --log-level=debug quay.io/travier/cosign-example:latest-podman
RobusTetus commented 8 months ago

Looks like this is related to multi-arch images.

My device is x86_64 too, so podman should follow my architecture when pulling an image, right?

However when searching I have stumbled upon this blog. Where when pushing the image, they are using --remove-signatures flag.

By default the Podman ecosystem treats signatures as an integral part of the image. So “–remove-signatures” is necessary to make a copy that doesn’t include the signatures, including copies that don’t preserve the images bit-for-bit-exactly.

Maybe sigstore/cosign-installer@v3.3.0 you are using in the workflow is not signing/pushing images this way :thinking:. At this point I am honestly just guessing. :laughing:

travier commented 8 months ago

Confirmed with:

$ podman pull --log-level=debug quay.io/travier/cosign-example:latest-cosign-x86_64

which works when the multi-arch image does not. So this is likely an issue in how podman verifies the signature for multi-arch containers. Or we need to build and sign our multi-arch containers differently using cosign.

travier commented 8 months ago

I've filed: https://github.com/containers/podman/issues/21209

travier commented 8 months ago

Looks like this is working now! The Fedora & Ubuntu images are still building but I just pulled the CentOS one and it worked for me. Thanks a lot for the report!