search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.83k stars 2.42k forks source link

Podman on MacOS (and CentOS 9) still can't access images on sites like registry.redhat.io/redhat/community-operator-index #20727

Closed bentito closed 11 months ago

bentito commented 12 months ago

Issue Description

https://github.com/containers/podman/issues/17391 is still a valid problem.

podman pull --tls-verify=false registry.redhat.io/redhat/community-operator-index:v4.14
Trying to pull registry.redhat.io/redhat/community-operator-index:v4.14...
Error: copying system image from manifest list: Source image rejected: Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"1AC4971355A34A82", Status:gpgme.Error{err:0x9}, Timestamp:time.Date(2023, time.November, 20, 5, 30, 12, 0, time.Local), ExpTimestamp:time.Date(1969, time.December, 31, 19, 0, 0, 0, time.Local), WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}

Doesn't matter if tls-verify is on/off.

And actually adding that this is a problem on CentOS 9 too. Adding an extra podman info and error message below.

Steps to reproduce the issue

Steps to reproduce the issue

  1. Try to pull image from registry.redhat.io/redhat/community-operator-index
  2. Ends in gpg related error.

Describe the results you received

MacOS: Error: copying system image from manifest list: Source image rejected: Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"1AC4971355A34A82", Status:gpgme.Error{err:0x9}, Timestamp:time.Date(2023, time.November, 20, 5, 30, 12, 0, time.Local), ExpTimestamp:time.Date(1969, time.December, 31, 19, 0, 0, 0, time.Local), WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}

CentOS: podman pull registry.redhat.io/redhat/community-operator-index:v4.14 Error: docker pull registry.redhat.io/redhat/community-operator-index:v4.14 failed with error: (exit status 125) Trying to pull registry.redhat.io/redhat/community-operator-index:v4.14... Error: copying system image from manifest list: Source image rejected: Invalid GPG signature: gpgme.Signature{Summary:128, Fingerprint:"1AC4971355A34A82", Status:gpgme.Error{err:0x9}, Timestamp:time.Date(2023, time.November, 20, 17, 35, 46, 0, time.Local), ExpTimestamp:time.Date(1970, time.January, 1, 0, 0, 0, 0, time.Local), WrongKeyUsage:false, PKATrust:0x0, ChainModel:false, Validity:0, ValidityReason:error(nil), PubkeyAlgo:1, HashAlgo:8}

Describe the results you expected

The image to pull

podman info output (MacOS)

host:
  arch: arm64
  buildahVersion: 1.32.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-3.fc39.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: '
  cpuUtilization:
    idlePercent: 99.13
    systemPercent: 0.38
    userPercent: 0.49
  cpus: 1
  databaseBackend: boltdb
  distribution:
    distribution: fedora
    variant: coreos
    version: "39"
  eventLogger: journald
  freeLocks: 2043
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.5.9-300.fc39.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 1305497600
  memTotal: 1980256256
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.8.0-1.fc39.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: netavark-1.8.0-2.fc39.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: crun-1.11-1.fc39.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.11
      commit: 11f8d3dc9fc4bb8a0adcff5ba8bd340f24612701
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20231004.gf851084-1.fc39.aarch64
    version: |
      pasta 0^20231004.gf851084-1.fc39.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.fc39.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 80h 24m 8.00s (Approximately 3.33 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 0
    stopped: 4
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 24557383680
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 61
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 4.7.0
  Built: 1695838660
  BuiltTime: Wed Sep 27 14:17:40 2023
  GitCommit: ""
  GoVersion: go1.21.1
  Os: linux
  OsArch: linux/arm64
  Version: 4.7.0

podman info output (CentOS)

host:
  arch: amd64
  buildahVersion: 1.31.3
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.8-1.el9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: 65953271fc1e506ac4eb890c645f3f75976973b4'
  cpuUtilization:
    idlePercent: 97.47
    systemPercent: 0.86
    userPercent: 1.67
  cpus: 2
  databaseBackend: boltdb
  distribution:
    distribution: '"centos"'
    version: "9"
  eventLogger: file
  freeLocks: 2048
  hostname: my-centos-dev-vm.northamerica-northeast1-a.c.tofeltech.internal
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 603295573
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 603295573
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
  kernel: 5.14.0-373.el9.x86_64
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 15894261760
  memTotal: 16486543360
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.8.0-1.el9.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.8.0
    package: netavark-1.8.0-3.el9.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.8.0
  ociRuntime:
    name: crun
    package: crun-1.11.1-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.11.1
      commit: 1084f9527c143699b593b44c23555fb3cc4ff2f3
      rundir: /run/user/603295573/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    path: /run/user/603295573/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-1.el9.x86_64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 0
  swapTotal: 0
  uptime: 0h 8m 56.00s
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/mail_reknew_gmail_com/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/mail_reknew_gmail_com/.local/share/containers/storage
  graphRootAllocated: 214469492736
  graphRootUsed: 15523348480
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/user/603295573/containers
  transientStore: false
  volumePath: /home/mail_reknew_gmail_com/.local/share/containers/storage/volumes
version:
  APIVersion: 4.6.1
  Built: 1692961071
  BuiltTime: Fri Aug 25 10:57:51 2023
  GitCommit: ""
  GoVersion: go1.20.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.6.1

Podman in a container

Yes and No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

MacOS and CentOS both fail similarly

Additional information

No response

flouthoc commented 11 months ago

Hi,

Afaik this needs correct signing key for pulling images from private repos and /etc/containers/policy.json must be modified to refect path and entry for the signing key and /etc/containers/policy.json must be on the VM and not on your macOS client.

Could you confirm if policy.json is correctly populated and on the VM ?

Use podman machine ssh to ssh inside the machine and verify it.

Overall I think @mtrmac can help here.

mtrmac commented 11 months ago

https://github.com/containers/podman/issues/17391 is still a valid problem.

17391 is decidedly an user error, using an incorrect way to specify trusted keys for verifying signatures.

My best guess here is that Podman is correctly enforcing the signature policy as configured (in particular, as shipped by default), and that policy does not trust the involved key. Compare https://access.redhat.com/solutions/6542281 for a rough outline of how the policy might need to be adapted.

There’s probably a discussion to be had about whether these signing keys should (or should not) be trusted by default; that debate needs to happen with the relevant Red Hat product teams, not here in the upstream Podman repository.

bentito commented 11 months ago

Yes, policy.json is there:

core@localhost:~$ cat /etc/containers/policy.json
{
    "default": [
        {
            "type": "insecureAcceptAnything"
        }
    ],
    "transports": {
        "docker": {
        "registry.access.redhat.com": [
        {
            "type": "signedBy",
            "keyType": "GPGKeys",
            "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
        }
        ],
        "registry.redhat.io": [
        {
            "type": "signedBy",
            "keyType": "GPGKeys",
            "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
        }
        ]
    },
        "docker-daemon": {
        "": [
        {
            "type": "insecureAcceptAnything"
        }
        ]
    }
    }
}
bentito commented 11 months ago

And of course, this insecure workaround does allow the pull to work:

cat /etc/containers/policy.json
{
    "default": [
        {
            "type": "insecureAcceptAnything"
        }
    ],
    "transports": {
        "docker": {
        "registry.access.redhat.com": [
        {
            "type": "signedBy",
            "keyType": "GPGKeys",
            "keyPath": "/etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release"
        }
        ],
            "registry.redhat.io": [
                {
                    "type": "insecureAcceptAnything"
                }
            ]
    },
        "docker-daemon": {
        "": [
        {
            "type": "insecureAcceptAnything"
        }
        ]
    }
    }
}
mtrmac commented 11 months ago

Thanks. The above applies then: either locally update your policy.json, or discuss, using official Red Hat product support channels, what the policy should be.