search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.83k stars 2.42k forks source link

podman first run: ERRO[0000] overlay test mount with multiple lowers failed, but succeeded with a single lower #10153

Closed asottile closed 3 years ago

asottile commented 3 years ago

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

The first run of podman after installation spews this error:

ERRO[0000] overlay test mount with multiple lowers failed, but succeeded with a single lower

Steps to reproduce the issue:

  1. clean install podman
  2. I have this storage configuration: 
    [storage]
driver = "overlay"
[storage.options]
mount_program = "/usr/bin/fuse-overlayfs"
  3. run any podman command (only the first one shows this error)

Describe the results you received:

$ podman info
ERRO[0000] overlay test mount with multiple lowers failed, but succeeded with a single lower 
host:
  arch: amd64
  buildahVersion: 1.20.1
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 2
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: journald
  hostname: asottile-VirtualBox
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.8.0-50-generic
  linkmode: dynamic
  memFree: 197468160
  memTotal: 4126838784
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.19.1.3-9b83-dirty
      commit: 33851ada2cc9bf3945915565bf3c2df97facb92c
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 1943183360
  swapTotal: 1964396544
  uptime: 25h 48m 30.79s (Approximately 1.04 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/asottile/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.9.0
        fuse-overlayfs: version 1.4
        FUSE library version 3.9.0
        using FUSE kernel interface version 7.31
  graphRoot: /home/asottile/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 6
  runRoot: /run/user/1000/containers
  volumePath: /home/asottile/.local/share/containers/storage/volumes
version:
  APIVersion: 3.1.2
  Built: 0
  BuiltTime: Wed Dec 31 16:00:00 1969
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 3.1.2

Describe the results you expected:

not an error

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

$ podman version
Version:      3.1.2
API Version:  3.1.2
Go Version:   go1.15.2
Built:        Wed Dec 31 16:00:00 1969
OS/Arch:      linux/amd64

Output of podman info --debug:

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.20.1
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.27, commit: '
  cpus: 2
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: journald
  hostname: asottile-VirtualBox
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.8.0-50-generic
  linkmode: dynamic
  memFree: 296677376
  memTotal: 4126838784
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.19.1.3-9b83-dirty
      commit: 33851ada2cc9bf3945915565bf3c2df97facb92c
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 1943183360
  swapTotal: 1964396544
  uptime: 25h 51m 45.21s (Approximately 1.04 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/asottile/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.9.0
        fuse-overlayfs: version 1.4
        FUSE library version 3.9.0
        using FUSE kernel interface version 7.31
  graphRoot: /home/asottile/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 6
  runRoot: /run/user/1000/containers
  volumePath: /home/asottile/.local/share/containers/storage/volumes
version:
  APIVersion: 3.1.2
  Built: 0
  BuiltTime: Wed Dec 31 16:00:00 1969
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 3.1.2

Package info (e.g. output of rpm -q podman or apt list podman):

$ apt list podman
Listing... Done
podman/unknown 100:3.1.2-1 amd64
podman/unknown 100:3.1.2-1 arm64
podman/unknown 100:3.1.2-1 armhf
podman/unknown 100:3.1.2-1 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes I believe this is the latest version

Additional environment details (AWS, VirtualBox, physical, etc.):

both AWS + VirtualBox

mheon commented 3 years ago

@giuseppe PTAL - Could this be fuse-overlay itself, given the test mount from c/storage is failing?

giuseppe commented 3 years ago

that is a test failing in containers/storage. I think it is the issue we had with not testing userxattr on Ubuntu kernels where unprivileged overlay exists but it works differently than upstream.

giuseppe commented 3 years ago

it should be fixed upstream now

rhatdan commented 3 years ago

I am not sure about this.

My reading of: https://github.com/containers/storage/blob/master/drivers/overlay/overlay.go#L604-L624 Says that if these mounts will both fail throwing this error, if the kernel does not support the userxattr option for rootless mode.

tobwen commented 3 years ago

Whoops... this is still an issue on Debian Bullseye RC1 with podman v3.2.0-rc2 (b060a7726e8e97f7752b40f18e69236cb330d22d) and kernel v5.10.0-6-amd64.

complains by podman

Error: kernel does not support overlay fs: kernel too old to provide multiple lowers feature for overlay: driver not supported

permit overlay for userns

# /etc/modprobe.d/10-overlay-userns.conf
options overlay permit_mounts_in_userns=1

kernel messages

[    3.600287] overlayfs: overlayfs: Allowing overlay mounts in user namespaces bears security risks
[...]
[ 1709.190690] overlayfs: unrecognized mount option "userxattr" or missing value
[ 1709.190944] overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.

See also: https://github.com/containers/podman/issues/10248

rhatdan commented 3 years ago

@giuseppe PTAL

giuseppe commented 3 years ago

is the Debian kernel using the same patches for enabling unprivileged overlay as Ubuntu?

That could be the reason why these tests fail, does it fallback to fuse-overlayfs?

tobwen commented 3 years ago

From the very beginning the Debian Overlay module has got a switch in config to allow overlay in usersns. It worked in previous versions of Podman. This is not a backported patch or anything like that.

Yes, the fallback works. But I wanted to work with the module again.

giuseppe commented 3 years ago

that was never supported. In some older kernel versions it was not possible to create whiteout files

tobwen commented 3 years ago

I've just checked it: It's the same kernel patch as in Ubuntu, but there it's activated by default. In Debian, it needs a switch. Sad to hear it's not supported. Debian won't get 5.11 in the near future.

tobwen commented 3 years ago

That's the patch Debian uses (even for Bullseye / v11): https://salsa.debian.org/kernel-team/linux/blob/master/debian/patches/debian/overlayfs-permit-mounts-in-userns.patch

Since it acts like root, it normally should work. Even with whiteout files. Can't you just add an override to not set userxattr?