Unprivileged overlay attempts to mknod

[flx42]

I don't believe #1949 truly got fixed. I'm running podman devel version from d652c8656d7425c34234071eddb55cbba9ac070f, I have shadow-utils binaries with setuid and UID/GID ranges configured, and fuse-overlay version 0.3

When pulling an image with file whiteouts, I get "permission denied":

$ podman --storage-driver=overlay pull docker://ubuntu:18.04
Trying to pull docker://ubuntu:18.04...Getting image source signatures
Copying blob 898c46f3b1a1 skipped: already exists
Copying blob 63366dfa0a50 skipped: already exists
Copying blob 6e1bee0f8701 done
Copying blob 041d4cd74a92 done
Copying config 94e814e2ef done
Writing manifest to image destination
Storing signatures
ERRO[0003] Error pulling image ref //ubuntu:18.04: Error committing the finished image: error adding layer with blob "sha256:041d4cd74a929bc4b66ee955ab5b229de098fa389d1a1fb9565e536d8878e15f": Error processing tar file(exit status 1): operation not permitted
Failed
ERRO[0003] error pulling image "docker://ubuntu:18.04"

This is because the following code unconditionally attempts to mknod for file whiteouts: https://github.com/containers/storage/blob/83889dc0b1fb641897591af51318cd8ee3a2b5a1/pkg/archive/archive_linux.go#L131-L132

$ strace -f -e 'mknodat' podman --storage-driver=overlay pull docker://ubuntu:18.04
[...]
mknodat(AT_FDCWD, "/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_bionic-updates_InRelease", S_IFCHR|000, makedev(0, 0)) = -1 EPERM (Operation not permitted)

[mheon]

@giuseppe PTAL

[rhatdan]

@flx42 Could you look at ~/.config/containers/storage.conf Does this include fuse-overlayfs in the file? And the overlay driver.

[flx42]

@rhatdan good catch, it wasn't, I installed fuse-overlayfs after running a vfs container, then nuked my ~/.local/share/containers/ when podman complains about a database configuration mismatch, then ran the overlay pull, but with the vfs configuration file, sigh...

So, it looked like this:

$ podman --storage-driver=vfs run docker://ubuntu:18.04

$ grep -E 'driver|mount_program'  ~/.config/containers/storage.conf
  driver = "vfs"
    mount_program = ""

$ sudo apt-get install -y fuse-overlayfs

$ podman --storage-driver=overlay run -ti docker://ubuntu:18.04
Error: could not get runtime: database storage graph driver vfs does not match our storage graph driver overlay: database configuration mismatch

$ rm -rf ~/.local/share/containers/
$ podman --storage-driver=overlay run -ti docker://ubuntu:18.04
[...]
ERRO[0005] Error pulling image ref //ubuntu:18.04: Error committing the finished image: error adding layer with blob "sha256:041d4cd74a929bc4b66ee955ab5b229de098fa389d1a1fb9565e536d8878e15f": Error processing tar file(exit status 1): operation not permitted
Failed
Error: unable to pull docker://ubuntu:18.04: unable to pull image: Error committing the finished image: error adding layer with blob "sha256:041d4cd74a929bc4b66ee955ab5b229de098fa389d1a1fb9565e536d8878e15f": Error processing tar file(exit status 1): operation not permitted

$ rm -rf ~/.config/containers/storage.conf
$ podman --storage-driver=overlay run -ti docker://ubuntu:18.04
[now works fine]
$ grep -E 'driver|mount_program'  ~/.config/containers/storage.conf
  driver = "overlay"
    mount_program = "/usr/bin/fuse-overlayfs"

Thanks for the help, but it would helpful to detect this situation if possible.

[mheon]

Some OSes do allow unprivileged kernel overlayfs (Ubuntu by default, debian by installing a kernel module, I believe), so I don't know if we want to add a warning. Would be nice if we could detect that the kernel supports unprivileged overlay and warn if we don't have a mount option if it doesn't, though.

[rhatdan]

Container storage should report that ENOSUP for overlay. So we could easily catch it. And then look to see if there is a mount options for mount program, and report it then.

[flx42]

Some OSes do allow unprivileged kernel overlayfs (Ubuntu by default, debian by installing a kernel module, I believe)

But how is that expected to work with podman? You won't be able to mknod or set xattrs anyway to convert the whiteouts to overlay format.

[rhatdan]

Good point, unless Ubuntu has also relaxed the rules and allow users to create the devices that Overlay needs without mknod privs (Perhaps inside of a User Namespace.

@rhvgoyal do you know?

[rhvgoyal]

@rhatdan I don't know. My guess is probably they might be carrying a patch which is not upstream.

[flx42]

I don't think they do, https://lkml.org/lkml/2018/7/5/607 is a revert to upstream.

[rhatdan]

Well is that the upstream kernel? If Ubuntu/Debian does not apply that patch it still might work.

[flx42]

It was reverted by a Canonical engineer, so I doubt it :)

Edit: and I confirm the revert commit is indeed in the kernel sources for 19.04

[rhatdan]

Well in that case podman on Kernel Overlay will not work in rootless mode. You need to use fuse-overlay. Which seems to work pretty well.

[flx42]

Right, so back to

Some OSes do allow unprivileged kernel overlayfs (Ubuntu by default, debian by installing a kernel module, I believe), so I don't know if we want to add a warning.

It means we can add a warning :)

[rhatdan]

So you want to container/storage to warn if overlay is used without a mount_program in rootless mode?

[flx42]

I'll let you decide what you deem is best, but that would have helped troubleshooting this particular case.

[rhatdan]

@nalind What do you think of adding a test of trying to create a whiteout devicenode on the overlay to verify that the overlayfs is working in rootless mode, if the caller does not specify a mount-program

[baude]

where do we stand on this one ?

[rhatdan]

@giuseppe Could you open a PR against container-storage to verify that the whiteout can be created in non-mount program mode.

[giuseppe]

@giuseppe Could you open a PR against container-storage to verify that the whiteout can be created in non-mount program mode.

I've tried on Ubuntu and mknod still doesn't work for unprivileged users (also from a user namespace). I think it is allowed only from overlay itself, in facts it gets created in the upper dir whenever I delete something from the overlay mount.

[giuseppe]

I'll close this issue, as checking for mknod is not enough on Ubuntu/Debian as well, since an unprivileged user still has no permissions for that.