error storing layer to file: inconsistent layer size

WIPocket commented 7 months ago

Issue Description

Unable to build any container, COMMIT always fails.

Steps to reproduce the issue

$ cat > Dockerfile <<EOF
FROM docker.io/debian:bullseye
RUN touch updates
EOF
$ podman build -f Dockerfile

Describe the results you received

This error:

STEP 1/2: FROM docker.io/debian:bullseye
STEP 2/2: RUN touch updates
COMMIT
Error: error committing container for step {Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:run Args:[touch updates] Flags:[] Attrs:map[] Message:RUN touch updates Original:RUN touch updates}: error copying layers and metadata for container "06f4e7947607ed8502adfac2b3d8f49d1efa7c70e5d4fd2940236abc57c9aaba": initializing source containers-storage:debian-working-container: error storing layer "e6a20a572244916d8d4c4f61f1a4413124cdae502ee06335a7b3aee36eacfc51" to file: inconsistent layer size (copied 5632, wrote 2560)

Describe the results you expected

No error

podman info output

host:
  arch: amd64
  buildahVersion: 1.33.3
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon_2.1.6+ds1-1_amd64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.6, commit: unknown'
  cpuUtilization:
    idlePercent: 91.74
    systemPercent: 2.38
    userPercent: 5.88
  cpus: 12
  databaseBackend: boltdb
  distribution:
    codename: trixie
    distribution: debian
    version: unknown
  eventLogger: journald
  freeLocks: 2048
  hostname: enbian
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.6.13-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 970321920
  memTotal: 16425033728
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns_1.4.0-5_amd64
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.4.0
    package: netavark_1.4.0-4_amd64
    path: /usr/lib/podman/netavark
    version: netavark 1.4.0
  ociRuntime:
    name: crun
    package: crun_1.14-1_amd64
    path: /usr/bin/crun
    version: |-
      crun version 1.14
      commit: 667e6ebd4e2442d39512e63215e79d693d0780aa
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt_0.0~git20231230.f091893-1_amd64
    version: |
      pasta unknown version
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: false
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.1-1_amd64
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 30411059200
  swapTotal: 34359734272
  uptime: 30h 28m 20.00s (Approximately 1.25 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/wipocket/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/wipocket/.local/share/containers/storage
  graphRootAllocated: 1967335682048
  graphRootUsed: 977349177344
  graphStatus: {}
  imageCopyTmpDir: /tmp/user/1000
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  transientStore: false
  volumePath: /home/wipocket/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.0
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.21.6
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.0

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

Yes

Additional environment details

No response

Additional information

I have managed to bisect this to eb28a1c08469d56494006d0f2c64933ab7078d01. Resetting the storage (either using system reset or by removing /home/wipocket/.local/share/containers did not help.

mheon commented 7 months ago

@nalind PTAL

github-actions[bot] commented 6 months ago

A friendly reminder that this issue had no activity for 30 days.

WIPocket commented 5 months ago

I can no longer replicate this on master. My system podman version 4.9.3 still fails to build anything but a fresh build of podman version 5.1.0-dev (at 447d3e2a4) is able to commit.

containers / podman