Multi-arch container images signed with cosign can not be pulled/verified by podman

Issue Description

Multi-arch images created by podman and signed by cosign can not be pulled and the signature verified by podman, but they are reported valid by cosign verify.

Pushing and signing the image via podman works.

Steps to reproduce the issue

Exemple test repo to reproduce the issue: https://github.com/travier/cosign-test Container images: https://quay.io/repository/travier/cosign-example?tab=tags

# Valid signatures according to cosign
$ cosign verify --key quay.io-travier-containers.pub quay.io/travier/cosign-example:latest-cosign
$ cosign verify --key quay.io-travier-containers.pub quay.io/travier/cosign-example:latest-cosign-x86_64

# Works
$ podman pull --log-level=debug quay.io/travier/cosign-example:latest-cosign-x86_64

# Fails
$ podman pull --log-level=debug quay.io/travier/cosign-example:latest-cosign
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called pull.PersistentPreRunE(podman pull --log-level=debug quay.io/travier/cosign-example:latest-cosign) 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
INFO[0000] Using boltdb as database backend             
DEBU[0000] Initializing boltdb state at /var/home/tim/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/tim/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /var/home/tim/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/tim/.local/share/containers/storage/volumes 
DEBU[0000] Using transient store: false                 
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that overlay is supported 
DEBU[0000] Cached value indicated that metacopy is not being used 
DEBU[0000] Cached value indicated that native-diff is usable 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=true, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] Configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument 
DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument 
DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument 
DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument 
DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument 
DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument 
DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument 
DEBU[0000] Using OCI runtime "/usr/bin/crun"            
INFO[0000] Setting parallel job count to 25             
DEBU[0000] Pulling image quay.io/travier/cosign-example:latest-cosign (policy: always) 
DEBU[0000] Looking up image "quay.io/travier/cosign-example:latest-cosign" in local containers storage 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Trying "quay.io/travier/cosign-example:latest-cosign" ... 
DEBU[0000] reference "[overlay@/var/home/tim/.local/share/containers/storage+/run/user/1000/containers]quay.io/travier/cosign-example:latest-cosign" does not resolve to an image ID 
DEBU[0000] Trying "quay.io/travier/cosign-example:latest-cosign" ... 
DEBU[0000] reference "[overlay@/var/home/tim/.local/share/containers/storage+/run/user/1000/containers]quay.io/travier/cosign-example:latest-cosign" does not resolve to an image ID 
DEBU[0000] Trying "quay.io/travier/cosign-example:latest-cosign" ... 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" 
DEBU[0000] Normalized platform linux/amd64 to {amd64 linux  [] } 
DEBU[0000] Attempting to pull candidate quay.io/travier/cosign-example:latest-cosign for quay.io/travier/cosign-example:latest-cosign 
DEBU[0000] parsed reference into "[overlay@/var/home/tim/.local/share/containers/storage+/run/user/1000/containers]quay.io/travier/cosign-example:latest-cosign" 
Trying to pull quay.io/travier/cosign-example:latest-cosign...
DEBU[0000] Copying source image //quay.io/travier/cosign-example:latest-cosign to destination image [overlay@/var/home/tim/.local/share/containers/storage+/run/user/1000/containers]quay.io/travier/cosign-example:latest-cosign 
DEBU[0000] Using registries.d directory /etc/containers/registries.d 
DEBU[0000] Trying to access "quay.io/travier/cosign-example:latest-cosign" 
DEBU[0000] No credentials matching quay.io/travier/cosign-example found in /run/user/1000/containers/auth.json 
DEBU[0000] No credentials matching quay.io/travier/cosign-example found in /var/home/tim/.config/containers/auth.json 
DEBU[0000] Found credentials for quay.io/travier/cosign-example in credential helper containers-auth.json in file /var/home/tim/.docker/config.json 
DEBU[0000]  Lookaside configuration: using "docker" namespace quay.io/travier 
DEBU[0000]  No signature storage configuration found for quay.io/travier/cosign-example:latest-cosign, using built-in default file:///var/home/tim/.local/share/containers/sigstore 
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/quay.io 
DEBU[0000]  Sigstore attachments: using "docker" namespace quay.io/travier 
DEBU[0000] GET https://quay.io/v2/                      
DEBU[0000] Ping https://quay.io/v2/ status 401          
DEBU[0000] GET https://quay.io/v2/auth?account=openshift-release-dev%2Bocm_access_c273414d3a374a04b57d071678c1f310&scope=repository%3Atravier%2Fcosign-example%3Apull&service=quay.io 
DEBU[0000] Increasing token expiration to: 60 seconds   
DEBU[0000] GET https://quay.io/v2/travier/cosign-example/manifests/latest-cosign 
DEBU[0001] Content-Type from manifest GET is "application/vnd.oci.image.index.v1+json" 
DEBU[0001] Using SQLite blob info cache at /var/home/tim/.local/share/containers/cache/blob-info-cache-v1.sqlite 
DEBU[0001] Source is a manifest list; copying (only) instance sha256:3c3d35273a04751de96ba2e1bd2e57b06f69329cd7f42c0d80454e90dfa0e44d for current system 
DEBU[0001] GET https://quay.io/v2/travier/cosign-example/manifests/sha256:3c3d35273a04751de96ba2e1bd2e57b06f69329cd7f42c0d80454e90dfa0e44d 
DEBU[0001] Content-Type from manifest GET is "application/vnd.oci.image.manifest.v1+json" 
DEBU[0001] IsRunningImageAllowed for image docker:quay.io/travier/cosign-example:latest-cosign 
DEBU[0001]  Using transport "docker" specific policy section quay.io/travier 
DEBU[0001] Reading /var/home/tim/.local/share/containers/sigstore/travier/cosign-example@sha256=3c3d35273a04751de96ba2e1bd2e57b06f69329cd7f42c0d80454e90dfa0e44d/signature-1 
DEBU[0001] Looking for sigstore attachments in quay.io/travier/cosign-example:sha256-3c3d35273a04751de96ba2e1bd2e57b06f69329cd7f42c0d80454e90dfa0e44d.sig 
DEBU[0001] GET https://quay.io/v2/travier/cosign-example/manifests/sha256-3c3d35273a04751de96ba2e1bd2e57b06f69329cd7f42c0d80454e90dfa0e44d.sig 
DEBU[0001] Content-Type from manifest GET is "application/json" 
DEBU[0001] Fetching sigstore attachment manifest failed, assuming it does not exist: reading manifest sha256-3c3d35273a04751de96ba2e1bd2e57b06f69329cd7f42c0d80454e90dfa0e44d.sig in quay.io/travier/cosign-example: manifest unknown 
DEBU[0001] Requirement 0: denied, done                  
DEBU[0001] Error pulling candidate quay.io/travier/cosign-example:latest-cosign: copying system image from manifest list: Source image rejected: A signature was required, but no signature exists 
Error: copying system image from manifest list: Source image rejected: A signature was required, but no signature exists
DEBU[0001] Shutting down engines

Describe the results you received

Pulling multi-arch container images signed with cosign fails with podman.

Describe the results you expected

Pulling multi-arch container images signed with cosign succeeds with podman.

podman info output

host:
  arch: amd64
  buildahVersion: 1.33.2
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.8-2.fc39.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: '
  cpuUtilization:
    idlePercent: 80.96
    systemPercent: 3.92
    userPercent: 15.12
  cpus: 8
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: kinoite
    version: "39"
  eventLogger: journald
  freeLocks: 2041
  hostname: phoenix
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.6.9-200.fc39.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 1837248512
  memTotal: 33423486976
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.9.0-1.fc39.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.9.0
    package: netavark-1.9.0-1.fc39.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.9.0
  ociRuntime:
    name: crun
    package: crun-1.12-1.fc39.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.12
      commit: ce429cb2e277d001c2179df1ac66a470f00802ae
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20231204.gb86afe3-1.fc39.x86_64
    version: |
      pasta 0^20231204.gb86afe3-1.fc39.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8588881920
  swapTotal: 8589930496
  uptime: 33h 2m 29.00s (Approximately 1.38 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
store:
  configFile: /var/home/tim/.config/containers/storage.conf
  containerStore:
    number: 7
    paused: 0
    running: 4
    stopped: 3
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/tim/.local/share/containers/storage
  graphRootAllocated: 254917292032
  graphRootUsed: 184687009792
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 12
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /var/home/tim/.local/share/containers/storage/volumes
version:
  APIVersion: 4.8.2
  Built: 1702300984
  BuiltTime: Mon Dec 11 14:23:04 2023
  GitCommit: ""
  GoVersion: go1.21.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.8.2

Podman in a container

Privileged Or Rootless

Rootless

Upstream Latest Release

Additional environment details

Fedora Kinoite 39

$ podman version Client: Podman Engine Version: 4.8.2 API Version: 4.8.2 Go Version: go1.21.4 Built: Mon Dec 11 14:23:04 2023 OS/Arch: linux/amd64

Additional information

It looks like there is a difference in how podman and cosign sign and push signatures for multi-arch images.

Initially reported in https://github.com/toolbx-images/images/issues/113.

containers / podman