search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.41k stars 2.38k forks source link

bug: podman forgets a pulled image when it encounters a lack of space when pulling an image #20963

Open unknowndevQwQ opened 10 months ago

unknowndevQwQ commented 10 months ago

Issue Description

When running cuda with podman, I need to download a very large image, and I ran out of space during the download. After podman exited, I found that the .local/share/containers folder contained the images I had just downloaded, but they were not listed in the podman system df.

Trying to pull quay.io/truecharts/local-ai-cublas-cuda11:latest...
Getting image source signatures
Copying blob 2ae7f73b0979 done   | 
Copying blob 2ae7f73b0979 done   | 
Copying blob 2ae7f73b0979 done   | 
Copying blob 2ae7f73b0979 done   | 
Copying blob 98dfe1ecd6ba done   | 
Copying blob 6da684c9ba3f done   | 
Copying blob 1dedceb9c21c done   | 
Copying blob 69b3efbf67c2 done   | 
Copying blob e816ce73d406 done   | 
Copying blob a46e08f63792 done   | 
Copying blob 4f4fb700ef54 done   | 
Copying blob 4f4fb700ef54 skipped: already exists  
Copying blob 28deb52f31db done   | 
Copying blob 5e889ffd8249 done   | 
Copying blob 9a3d5a5890ee done   | 
Copying blob 2425d318d5c6 done   | 
Copying blob 4f4fb700ef54 skipped: already exists  
Copying blob 22323b635ad6 done   | 
Copying blob 0e396fb044cd done   | 
Copying blob 82d5ac839ebf done   | 
Copying blob 4885221cb6e7 done   | 
Copying blob 9c96ac266172 [=============>------------------------] 2.0GiB / 5.3GiB | 525.1 KiB/s
Copying blob afa74ff792fe [==>-----------------------------------] 389.1MiB / 5.0GiB | 9.3 MiB/s
Copying blob dbf2c688dc8c done   | 
Copying blob 305873340f83 done   | 
Copying blob 4f4fb700ef54 skipped: already exists  
Copying blob 4f4fb700ef54 skipped: already exists  
Copying blob 4f664a4c2757 done   | 
Copying blob 4e5a617c1590 [=========>----------------------------] 256.8MiB / 944.3MiB | 108.1 KiB/s
Copying blob 7fceb3e8ac24 done   | 
Copying blob f2ac2f564948 done   | 
Copying blob 1e422c4b4b9e done   | 
Copying blob dcf7dd3923d8 done   | 
Error: copying system image from manifest list: writing blob: storing blob to file "/var/tmp/container_images_storage317867221/17": write /var/tmp/container_images_storage317867221/17: no space left on device

Steps to reproduce the issue

Steps to reproduce the issue

  1. podman pull quay.io/truecharts/local-ai-cublas-cuda11
  2. no space left on device
  3. run ncdu, A lot of space under /home is eaten up by image!
  4. run podman system df -v, I can't believe I only have a hello-world image that's less than 1 MBit?

Describe the results you received

Unrecorded images taking up my space.

Describe the results you expected

I can look at these images via podman system df and decide if I want to abort the pull and delete them.

podman info output

host:
  arch: amd64
  buildahVersion: 1.33.2
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: /usr/bin/conmon is owned by conmon 1:2.1.8-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.8, commit: 00e08f4a9ca5420de733bf542b930ad58e1a7e7d'
  cpuUtilization:
    idlePercent: 77.54
    systemPercent: 8.82
    userPercent: 13.64
  cpus: 4
  databaseBackend: boltdb
  distribution:
    distribution: arch
    version: unknown
  eventLogger: journald
  freeLocks: 2048
  hostname: arch
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.6.5-zen1-1-zen
  linkmode: dynamic
  logDriver: journald
  memFree: 762372096
  memTotal: 6588948480
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns: {}
    package: /usr/lib/cni/ is owned by cni-plugins 1.4.0-1
    path: /usr/lib/cni
  ociRuntime:
    name: crun
    package: /usr/bin/crun is owned by crun 1.12-1
    path: /usr/bin/crun
    version: |-
      crun version 1.12
      commit: ce429cb2e277d001c2179df1ac66a470f00802ae
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: /usr/bin/slirp4netns is owned by slirp4netns 1.2.2-1
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.4
  swapFree: 5384052736
  swapTotal: 9736540160
  uptime: 6h 20m 52.00s (Approximately 0.25 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - docker.io
  - ghcr.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/user/.local/share/containers/storage
  graphRootAllocated: 50540048384
  graphRootUsed: 44232503296
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 4.8.1
  Built: 1701796166
  BuiltTime: Wed Dec  6 01:09:26 2023
  GitCommit: ef6e5ac40302770027e7598ee09d177454adcd2f-dirty
  GoVersion: go1.21.4
  Os: linux
  OsArch: linux/amd64
  Version: 4.8.1

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

run on vbox

Additional information

How can I safely delete undocumented mirrors? will podman prompt in the future that they are undocumented?

baude commented 10 months ago

@mheon can you put this on your radar (out further of course). To others, if anyone wants to take a stab, also great ...

github-actions[bot] commented 9 months ago

A friendly reminder that this issue had no activity for 30 days.

unknowndevQwQ commented 9 months ago

A friendly reminder that this issue had no activity for 30 days.

/ping

unknowndevQwQ commented 9 months ago

@baude Any progress?