system prune not reclaiming disk space

kavishgr commented 2 months ago

Issue Description

Podman Version

core@coreos:~$ podman version
Client:       Podman Engine
Version:      5.1.2
API Version:  5.1.2
Go Version:   go1.22.5
Built:        Wed Jul 10 04:00:00 2024
OS/Arch:      linux/arm64

core@coreos:~$ rpm -q podman
podman-5.1.2-1.fc40.aarch64

When reclaiming disk space in Podman, it reported that 0B were reclaimed, and nothing was deleted.

Podman disk usage:

core@coreos:~$ podman system df
TYPE           TOTAL       ACTIVE      SIZE        RECLAIMABLE
Images         5           4           1.813GB     868.9MB (48%)
Containers     0           0           0B          0B (0%)
Local Volumes  0           0           0B          0B (0%)

A system prune says total space reclaimed is 0B:

core@coreos:~$ podman system prune -a
WARNING! This command removes:
  - all stopped containers
  - all networks not used by at least one container
  - all images without at least one container associated with them
  - all build cache

Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B

Disk usage stays the same:

core@coreos:~$ podman system df
TYPE           TOTAL       ACTIVE      SIZE        RECLAIMABLE
Images         5           4           1.813GB     868.9MB (48%)
Containers     0           0           0B          0B (0%)
Local Volumes  0           0           0B          0B (0%)

Steps to reproduce the issue

Pull or build an image
Show disk usage with podman system df
Run podman system prune -a to reclaim disk space
Verify if disk usage was reclaimed with podman system df

Describe the results you received

Nothing were reclaimed after executing a podman system prune

Describe the results you expected

Disk space should be reclaimed.

podman info output

host:
  arch: arm64
  buildahVersion: 1.36.0
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 99.13
    systemPercent: 0.62
    userPercent: 0.25
  cpus: 2
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "40"
  eventLogger: journald
  freeLocks: 2048
  hostname: coreos
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 524288
      size: 65536
  kernel: 6.9.12-200.fc40.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 1501282304
  memTotal: 2041966592
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.1-1.fc40.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.1
    package: netavark-1.12.1-1.fc40.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.1
  ociRuntime:
    name: crun
    package: crun-1.15-1.fc40.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.15
      commit: e6eacaf4034e84185fd8780ac9262bbf57082278
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240726.g57a21d2-1.fc40.aarch64
    version: |
      pasta 0^20240726.g57a21d2-1.fc40.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 0h 8m 50.00s
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 33754689536
  graphRootUsed: 6709608448
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 4
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 5.1.2
  Built: 1720569600
  BuiltTime: Wed Jul 10 04:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.5
  Os: linux
  OsArch: linux/arm64
  Version: 5.1.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

OS Info

core@coreos:~$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="40.20240808.3.0 (CoreOS)"
ID=fedora
VERSION_ID=40
VERSION_CODENAME=""
PLATFORM_ID="platform:f40"
PRETTY_NAME="Fedora CoreOS 40.20240808.3.0"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:40"
HOME_URL="https://getfedora.org/coreos/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/"
SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=40
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=40
SUPPORT_END=2025-05-13
VARIANT="CoreOS"
VARIANT_ID=coreos
OSTREE_VERSION='40.20240808.3.0'

Additional information

Executing podman system reset removes everything.

Luap99 commented 2 months ago

Please provide an exact reproducer, how are the images created? If you have 0 containers then prune should be able to delete all images and systemd df should not show them as active.

kavishgr commented 2 months ago

Please provide an exact reproducer, how are the images created? If you have 0 containers then prune should be able to delete all images and systemd df should not show them as active.

Images were pulled from quay.io. They were official fedora/mariadb images(different versions).

rhatdan commented 2 months ago

External containers, I would figure.

If you add --external, does it work?

rhatdan commented 2 months ago

buildah containers Should show the extra containers, or podman ps --all --external

kavishgr commented 1 month ago

After performing a podman system reset, I was unable to reproduce the issue. I will re-open it if the problem recurs. Thanks.

containers / podman