search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.83k stars 2.42k forks source link

Builds fail with "no such device or address" #5430

Closed ctron closed 4 years ago

ctron commented 4 years ago

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

All of a sudden podman stopped building images, when it tries to commit the first build step, it always fails with no such device or address:

Steps to reproduce the issue:

  1. I don't how to get podman into this state

Describe the results you received:

DEBU[0072] COMMIT                                       
DEBU[0072] parsed reference into "[overlay@/home/jreimann/.local/share/containers/storage+/run/user/XXX/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/library/808b6b1ad1d9de6f7d21a816efa7d62de5b50a6fe15690813aa168d8dde6de77-tmp:latest" 
DEBU[0072] committing image with reference "containers-storage:[overlay@/home/jreimann/.local/share/containers/storage+/run/user/XXX/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/library/808b6b1ad1d9de6f7d21a816efa7d62de5b50a6fe15690813aa168d8dde6de77-tmp:latest" is allowed by policy 
DEBU[0072] parsed reference into "[overlay@/home/jreimann/.local/share/containers/storage+/run/user/XXX/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@9382431041983a6d9bc305bddbe238b262d8d5a888667c8b0075e0261c44077d" 
DEBU[0072] base image "9382431041983a6d9bc305bddbe238b262d8d5a888667c8b0075e0261c44077d" is already present in local storage, no need to copy its layers 
DEBU[0072] layer list: ["fc86ad79496a0932dd56764ae4e00954f6a760d84a27967bd61a41c6ce0bc59c" "91ede8b45c8090bf46f464486216e90182dcbaa6fdb1060ea3020da416156d3d"] 
DEBU[0072] using "/var/tmp/buildah869835859" to hold temporary data 
DEBU[0072] compressing layer "91ede8b45c8090bf46f464486216e90182dcbaa6fdb1060ea3020da416156d3d" with gzip 
DEBU[0072] overlay: mount_data=ro,lowerdir=/home/jreimann/.local/share/containers/storage/overlay/fc86ad79496a0932dd56764ae4e00954f6a760d84a27967bd61a41c6ce0bc59c/empty,upperdir=/home/jreimann/.local/share/containers/storage/overlay/fc86ad79496a0932dd56764ae4e00954f6a760d84a27967bd61a41c6ce0bc59c/diff,workdir=/home/jreimann/.local/share/containers/storage/overlay/fc86ad79496a0932dd56764ae4e00954f6a760d84a27967bd61a41c6ce0bc59c/work,context="system_u:object_r:container_file_t:s0:c338,c515" 
Error: error committing container for step {Env:[DISTTAG=f31container FGC=f31 container=oci  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] Command:run Args:[microdnf -y install java-11-openjdk-headless] Flags:[] Attrs:map[] Message:RUN microdnf -y install java-11-openjdk-headless Original:RUN microdnf -y install java-11-openjdk-headless}: error copying layers and metadata for container "015c297dc28b5c6d29e323a257660177a3576cbc1444e10819afd294e0dbf8e7": Error initializing source containers-storage:fedora-minimal-working-container: error extracting layer "91ede8b45c8090bf46f464486216e90182dcbaa6fdb1060ea3020da416156d3d": no such device or address

Describe the results you expected:

A working build, as before.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman version
Version:            1.8.0
RemoteAPI Version:  1
Go Version:         go1.13.6
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.13.6
  podman version: 1.8.0
host:
  BuildahVersion: 1.13.1
  CgroupVersion: v2
  Conmon:
    package: conmon-2.0.10-2.fc31.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.10, commit: 6b526d9888abb86b9e7de7dfdeec0da98ad32ee0'
  Distribution:
    distribution: fedora
    version: "31"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 110147
      size: 1
    - container_id: 1
      host_id: 120000
      size: 165536
    uidmap:
    - container_id: 0
      host_id: 110147
      size: 1
    - container_id: 1
      host_id: 120000
      size: 165536
  MemFree: 5785677824
  MemTotal: 33441271808
  OCIRuntime:
    name: crun
    package: crun-0.12.2.1-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.12.2.1
      commit: cd7cea7114db5f6aa35fbb69fa307c19c2728a31
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 16793989120
  SwapTotal: 16793989120
  arch: amd64
  cpus: 8
  eventlogger: journald
  hostname: flunder
  kernel: 5.5.7-200.fc31.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.0-20.1.dev.gitbbd6f25.fc31.x86_64
    Version: |-
      slirp4netns version 0.4.0-beta.3+dev
      commit: bbd6f25c70d5db2a1cd3bfb0416a8db99a75ed7e
  uptime: 101h 0m 55.65s (Approximately 4.21 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - quay.io
store:
  ConfigFile: /home/jreimann/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7.5-2.fc31.x86_64
      Version: |-
        fusermount3 version: 3.6.2
        fuse-overlayfs: version 0.7.5
        FUSE library version 3.6.2
        using FUSE kernel interface version 7.29
  GraphRoot: /home/jreimann/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 1
  RunRoot: /run/user/110147/containers
  VolumePath: /home/jreimann/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.8.0-2.fc31.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

physical

rhatdan commented 4 years ago

Could you try the podman release that is in updates-testing?

rhatdan commented 4 years ago

Also make sure you have the latest crun and fuse-overlayfs

ctron commented 4 years ago 
$ rpm -q crun 
crun-0.12.2.1-1.fc31.x86_64
$ rpm -q fuse-overlayfs
fuse-overlayfs-0.7.5-2.fc31.x86_64

It is the latest from Fedora 31:

# dnf update
Last metadata expiration check: 2:12:53 ago on Mon Mar  9 14:30:52 2020.
Dependencies resolved.
Nothing to do.
Complete!
ctron commented 4 years ago

I just figured out, that it doesn't seem to be an issue with my machine, but with the base image:

It fails with FROM registry.fedoraproject.org/fedora-minimal:31, but works when I change back to registry.access.redhat.com/ubi8/ubi-minimal.

ctron commented 4 years ago

Could you try the podman release that is in updates-testing?

If you can tell me "how", sure.

rhatdan commented 4 years ago

dnf -y update podman --enablerepo=updates-testing

ctron commented 4 years ago

So here is a bit more information:

rhatdan commented 4 years ago

@giuseppe Any ideas?

adelton commented 4 years ago

Is this the same issue as https://github.com/containers/buildah/issues/2161?

adelton commented 4 years ago

For the record, I see this problem when building from registry.fedoraproject.org/fedora:32 or registry.fedoraproject.org/fedora:rawhide images but not with registry.fedoraproject.org/fedora:31, registry.fedoraproject.org/fedora:30, or registry.fedoraproject.org/fedora:29.

TomSweeneyRedHat commented 4 years ago

@adelton it may be the same, but I'd like to keep this open until we fully determine that. Sure looks like it though.

github-actions[bot] commented 4 years ago

A friendly reminder that this issue had no activity for 30 days.

rhatdan commented 4 years ago

@adelton @TomSweeneyRedHat Still looking at this or should we close?

adelton commented 4 years ago

I do not experience any issues with rootless podman-1.8.2-2.fc31.x86_64.

ctron commented 4 years ago

I can no longer reproduce this issue. Maybe it was my update to podman 1.8.2, or maybe it was due to some update in the fedora images, I can't tell. Now it works for me.