search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.07k stars 2.35k forks source link

[MacOS 14.6.1]podman failed to pull image #23642

Open jianzhangbjz opened 1 month ago

jianzhangbjz commented 1 month ago

Issue Description

Failed to pull image.

jiazha-mac:acto jiazha$ podman pull alpine:latest
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
Error: copying system image from manifest list: parsing image configuration: Get "https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/0b/0b4426ad4bf25e13fb09112b9dcb5d5b09b3c5684599654583913b2714a705a2/data?verify=1723800131-9hSOHjemZZcjXnQadjnslY2zO6Q%3D": EOF

Steps to reproduce the issue

Steps to reproduce the issue

  1. run podman pull alpine:latest

Describe the results you received

Error: copying system image from manifest list: parsing image configuration: Get "https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/0b/0b4426ad4bf25e13fb09112b9dcb5d5b09b3c5684599654583913b2714a705a2/data?verify=1723800131-9hSOHjemZZcjXnQadjnslY2zO6Q%3D": EOF

I also tried to pull other images, got the same error, like below

jiazha-mac:acto jiazha$ podman pull docker.io/k8ssandra/system-logger:v1.10.3
Trying to pull docker.io/k8ssandra/system-logger:v1.10.3...
Error: parsing image configuration: Get "https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/0d/0d5e40b5ad31dfd452b9ee00116447872791dd1a0e2588514ccc5e520e8b3945/data?verify=1723800682-Xphj0qDQru8bY1gV%2FiyRTBRDLGI%3D": EOF

Describe the results you expected

Pull image successfully.

podman info output

If you are unable to run podman info for any reason, please provide the podman version, operating system and its version and the architecture you are running.

jiazha-mac:acto jiazha$ podman info 
host:
  arch: arm64
  buildahVersion: 1.36.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 99.34
    systemPercent: 0.38
    userPercent: 0.28
  cpus: 5
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "40"
  eventLogger: journald
  freeLocks: 2048
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.8.11-300.fc40.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 1534312448
  memTotal: 2044608512
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.11.0-1.20240628130058229856.main.10.g5ad6420.fc40.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.0-dev
    package: netavark-1.11.0-1.20240702123536284903.main.32.g49fb0c2.fc40.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.0-dev
  ociRuntime:
    name: crun
    package: crun-1.15-1.20240708144150212138.main.51.g6c158dd.fc40.aarch64
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: 54f958d21c4e2299eae6b0f4d8b742304540dce6
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240624.g1ee2eca-1.fc40.aarch64
    version: |
      pasta 0^20240624.g1ee2eca-1.fc40.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 0h 3m 28.00s
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 4496355328
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 5.1.2
  Built: 1720569600
  BuiltTime: Wed Jul 10 08:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.5
  Os: linux
  OsArch: linux/arm64
  Version: 5.1.2

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

I remove the old machine and re-init a new one, but still get this error.

jiazha-mac:acto jiazha$ podman machine list 
NAME                     VM TYPE     CREATED        LAST UP            CPUS        MEMORY      DISK SIZE
podman-machine-default*  applehv     7 minutes ago  Currently running  5           2GiB        100GiB

jiazha-mac:acto jiazha$ podman images -a
REPOSITORY  TAG         IMAGE ID    CREATED     SIZE

Additional information

jiazha-mac:acto jiazha$ podman --log-level debug pull alpine:latest
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called pull.PersistentPreRunE(podman --log-level debug pull alpine:latest) 
...
ig.json 
DEBU[0000] DoRequest Method: POST URI: http://d/v5.1.1/libpod/images/pull 
Resolved "alpine" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/alpine:latest...
Error: copying system image from manifest list: parsing image configuration: Get "https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/0b/0b4426ad4bf25e13fb09112b9dcb5d5b09b3c5684599654583913b2714a705a2/data?verify=1723800505-iPwjyLDbWJg4kgCE3DvtJ9oDIEo%3D": EOF
DEBU[0016] Shutting down engines
baude commented 3 weeks ago

are the cloudflare references in the URLs something? if images -a works, which it apparently did, then podman and the machine are working nicely.

any chance podman system connection has other connections in it besides your machine ?

jianzhangbjz commented 3 weeks ago

Full log:

jiazha-mac:~ jiazha$ podman --log-level debug  pull docker.io/k8ssandra/system-logger:v1.10.3
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called pull.PersistentPreRunE(podman --log-level debug pull docker.io/k8ssandra/system-logger:v1.10.3) 
DEBU[0000] SSH Ident Key "/Users/jiazha/.local/share/containers/podman/machine/machine" SHA256:xxx ssh-ed25519 
DEBU[0000] DoRequest Method: GET URI: http://d/v5.1.1/libpod/_ping 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] No credentials matching localhost:5000 found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] No credentials matching localhost:5000 found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] Found credentials for localhost:5000 in credential helper containers-auth.json in file /Users/jiazha/.docker/config.json 
DEBU[0000] No credentials matching registry.build01.ci.openshift.org found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] No credentials matching registry.build01.ci.openshift.org found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] Found credentials for registry.build01.ci.openshift.org in credential helper containers-auth.json in file /Users/jiazha/.docker/config.json 
DEBU[0000] No credentials matching registry.build02.ci.openshift.org found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] No credentials matching registry.build02.ci.openshift.org found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] Found credentials for registry.build02.ci.openshift.org in credential helper containers-auth.json in file /Users/jiazha/.docker/config.json 
DEBU[0000] No credentials matching upshift-quay.mirror-registry.qe.devcluster.openshift.com:5001 found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] No credentials matching upshift-quay.mirror-registry.qe.devcluster.openshift.com:5001 found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] Found credentials for upshift-quay.mirror-registry.qe.devcluster.openshift.com:5001 in credential helper containers-auth.json in file /Users/jiazha/.docker/config.json 
DEBU[0000] No credentials matching cloud.openshift.com found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] No credentials matching cloud.openshift.com found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] Found credentials for cloud.openshift.com in credential helper containers-auth.json in file /Users/jiazha/.docker/config.json 
DEBU[0000] Found credentials for quay.io/openshift-qe-optional-operators in credential helper containers-auth.json in file /Users/jiazha/.config/containers/auth.json 
DEBU[0000] No credentials matching registry.connect.redhat.com found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] No credentials matching registry.connect.redhat.com found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] Found credentials for registry.connect.redhat.com in credential helper containers-auth.json in file /Users/jiazha/.docker/config.json 
DEBU[0000] No credentials matching brew.registry.redhat.io found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] No credentials matching brew.registry.redhat.io found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] Found credentials for brew.registry.redhat.io in credential helper containers-auth.json in file /Users/jiazha/.docker/config.json 
DEBU[0000] No credentials matching registry.svc.ci.openshift.org found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] No credentials matching registry.svc.ci.openshift.org found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] Found credentials for registry.svc.ci.openshift.org in credential helper containers-auth.json in file /Users/jiazha/.docker/config.json 
DEBU[0000] No credentials matching registry.ci.openshift.org found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] No credentials matching registry.ci.openshift.org found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] Found credentials for registry.ci.openshift.org in credential helper containers-auth.json in file /Users/jiazha/.docker/config.json 
DEBU[0000] No credentials matching registry.redhat.io found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] No credentials matching registry.redhat.io found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] Found credentials for registry.redhat.io in credential helper containers-auth.json in file /Users/jiazha/.docker/config.json 
DEBU[0000] No credentials matching registry.stage.redhat.io found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] No credentials matching registry.stage.redhat.io found in /Users/jiazha/.config/containers/auth.json 
DEBU[0000] Found credentials for registry.stage.redhat.io in credential helper containers-auth.json in file /Users/jiazha/.docker/config.json 
DEBU[0000] Found credentials for quay.io in credential helper containers-auth.json in file /Users/jiazha/.config/containers/auth.json 
DEBU[0000] DoRequest Method: POST URI: http://d/v5.1.1/libpod/images/pull 
Trying to pull docker.io/k8ssandra/system-logger:v1.10.3...
Error: parsing image configuration: Get "https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/0d/0d5e40b5ad31dfd452b9ee00116447872791dd1a0e2588514ccc5e520e8b3945/data?verify=1724122513-DxwbwnR3LX6Cri1mUnKFFs%2BKrQo%3D": EOF
DEBU[0017] Shutting down engines

podman system connection

jiazha-mac:~ jiazha$ podman system connection list 
Name                         URI                                                         Identity                                                      Default     ReadWrite
podman-machine-default       ssh://core@127.0.0.1:51642/run/user/501/podman/podman.sock  /Users/jiazha/.local/share/containers/podman/machine/machine  true        true
podman-machine-default-root  ssh://root@127.0.0.1:51642/run/podman/podman.sock           /Users/jiazha/.local/share/containers/podman/machine/machine  false       true
rhatdan commented 3 weeks ago

Can you pull other images? Could this be a memory issue with the VM running out of memory.

jianzhangbjz commented 3 weeks ago

Yes, some, but some not. 

jiazha-mac:acto jiazha$ podman pull gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0
Trying to pull gcr.io/kubebuilder/kube-rbac-proxy:v0.8.0...
Getting image source signatures
Copying blob sha256:9d443775309fe30cb1c2b25750352e70004cf26baaf32c81daff7f5a05375845
Copying blob sha256:5c7fe08dec514a5105d43dfcae1a5a45be002bcd3251dd178ced702aecfb7531
Copying config sha256:595970b55bfb6a38ec55ce14bae7be1ffd6ae94d2201725aa221e04391a47538
Writing manifest to image destination
WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64)
595970b55bfb6a38ec55ce14bae7be1ffd6ae94d2201725aa221e04391a47538

However, I can pull this image successfully in the RHEL host, like below. 

[cloud-user@preserve-olm-env2 ~]$ podman pull docker.io/k8ssandra/system-logger:v1.10.3 
ERRO[0000] Refreshing volume 411cf6eebf07587da24f5e71c1b31a7690cb7d2d0ac2262ba5b7aaf1fda4b480: acquiring lock 1 for volume 411cf6eebf07587da24f5e71c1b31a7690cb7d2d0ac2262ba5b7aaf1fda4b480: file exists 
ERRO[0000] Refreshing volume a05e9c5537a1970162e5a929e2b1efd08e355c57f9c8641e9f9405fe51d7cd87: acquiring lock 2 for volume a05e9c5537a1970162e5a929e2b1efd08e355c57f9c8641e9f9405fe51d7cd87: file exists 
Trying to pull docker.io/k8ssandra/system-logger:v1.10.3...
Getting image source signatures
Copying blob fa4aa5826ad0 skipped: already exists  
Copying blob c9a00ad3ec08 skipped: already exists  
Copying blob 5ddbc98b1a08 skipped: already exists  
Copying blob 5d3891f7e1b5 skipped: already exists  
Copying blob d8085c2f5a4a skipped: already exists  
Copying blob 4f4fb700ef54 skipped: already exists  
Copying config 0d5e40b5ad done   | 
Writing manifest to image destination
0d5e40b5ad31dfd452b9ee00116447872791dd1a0e2588514ccc5e520e8b3945

Could this be a memory issue with the VM running out of memory.

I don't think so since there are no logs for the OOM.

jiazha-mac:acto jiazha$ podman images -a 
REPOSITORY                                          TAG         IMAGE ID      CREATED        SIZE
registry.redhat.io/redhat/certified-operator-index  v4.15       0d4f4b6ea3f3  2 days ago     991 MB
quay.io/olmqe/nginxolm-operator-index               59380       b81981b9f010  18 months ago  65.2 MB
gcr.io/kubebuilder/kube-rbac-proxy                  v0.8.0      595970b55bfb  3 years ago    47.9 MB