[WARN] fusermount: mount failed: Operation not permitted

[kcgthb]

Hi!

I'm trying to track down a permission issue with enroot start on RHEL/CentOS 7.6.

All the listed requirements seem to be satisfied:

# rpm -q glibc
glibc-2.17-260.el7_6.6.x86_64

# ./enroot-check_2.0.0.run --verify
Kernel version:

Linux version 3.10.0-957.27.2.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Mon Jul 29 17:46:05 UTC 2019

Kernel configuration:

CONFIG_NAMESPACES                 : OK
CONFIG_USER_NS                    : OK
CONFIG_SECCOMP_FILTER             : OK
CONFIG_OVERLAY_FS                 : OK (module)
CONFIG_X86_VSYSCALL_EMULATION     : KO (required if glibc <= 2.13)
CONFIG_VSYSCALL_EMULATE           : KO (required if glibc <= 2.13)
CONFIG_VSYSCALL_NATIVE            : KO (required if glibc <= 2.13)

Kernel command line:

namespace.unpriv_enable=1         : OK
user_namespace.enable=1           : OK
vsyscall=native                   : KO (required if glibc <= 2.13)
vsyscall=emulate                  : KO (required if glibc <= 2.13)

Kernel parameters:

user.max_user_namespaces          : OK
user.max_mnt_namespaces           : OK

Extra packages:

nvidia-container-cli              : OK
pv                                : OK

and

# rpm -q squashfuse fuse-overlayfs
squashfuse-0.1.102-1.el7.x86_64
fuse-overlayfs-0.5.2-1.el7.x86_64

I can import a container just fine:

# ENROOT_SQUASH_OPTIONS='-comp xz -noD' enroot import -o /tmp/alpine.sqsh docker://alpine
[INFO] Querying registry for permission grant
[INFO] Authenticating with user: <anonymous>
[INFO] Authentication succeeded
[...]
Parallel mksquashfs: Using 40 processors
Creating 4.0 filesystem on /tmp/alpine.sqsh, block size 131072.
[==============================================================================================================================================================================================================================================================|] 105/105 100%

Exportable Squashfs 4.0 filesystem, xz compressed, data block size 131072
        uncompressed data, compressed metadata, compressed fragments, compressed xattrs
        duplicates are removed
Filesystem size 5030.47 Kbytes (4.91 Mbytes)
        91.86% of uncompressed filesystem size (5476.08 Kbytes)
Inode table size 2374 bytes (2.32 Kbytes)
        13.65% of uncompressed inode table size (17387 bytes)
Directory table size 4207 bytes (4.11 Kbytes)
        50.96% of uncompressed directory table size (8255 bytes)
Number of duplicate files found 7
Number of inodes 486
Number of files 73
Number of fragments 6
Number of symbolic links  324
Number of device nodes 0
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 89
Number of ids (unique uids + gids) 1
Number of uids 1
        root (0)
Number of gids 1
        root (0)

But then (as root):

# enroot start /tmp/alpine.sqsh
[WARN] fusermount: mount failed: Operation not permitted
[ERROR] Failed to mount: /tmp/alpine.sqsh

The command that fails is in runtime::_do_mount_rootfs()

[WARN] + squashfuse -f -o uid=0,gid=0 /tmp/alpine.sqsh /run/enroot/alpine/lower
[WARN] fusermount: mount failed: Operation not permitted

Fuse and fusermount work fine on that host (especially as root), and I can run that very same command outside of enroot:

# squashfuse -o uid=0,gid=0 /tmp/alpine.sqsh /run/enroot/alpine/lower
# df /run/enroot/alpine/lower
Filesystem     1K-blocks  Used Available Use% Mounted on
squashfuse             0     0         0    - /run/enroot/alpine/lower

But I can't figure out where the EPERM in enroot start is coming from. Any hint to diagnose the issue and hopefully fix it would be much appreciated!

Thanks!

[3XX0]

Yeah, unfortunately you can only start images directly on Linux >= 4.18. This is because you need fuse support in user namespaces (https://github.com/torvalds/linux/commit/4ad769f3c346ec3d458e255548dec26ca5284cf6).

AFAIK RHEL 7.X doesn't have this patch, RHEL 8 might though.

In the meantime, you will have to unpack the image, and create a statefull container out of it: enroot create alpine.sqsh && enroot start alpine

[3XX0]

Note that you could also mount the image outside of enroot, either

1. As root on the loopback with a tmpfs overlay
2. Replicating runtime::_mount_rootfs() with fuse

Then you could use ENROOT_DATA_PATH=... enroot start ... to start it.

The difference is that you will have to cleanup/unmount after you're done with the rootfs since there are no namespaces in this case, and the above would require some privileges.

[kcgthb]

Aaah, I see, sorry I missed that. I'd ideally prefer the stateless, transparent mount option, but thanks for pointing out the missing kernel patch. And thanks for suggesting workarounds, I'll give them a try.

[kcgthb]

For the sake of completeness, and future reference, in case that helps anyone, I confirmed that:

• RHEL/CentOS 7.6 (kernel 3.10.0-957) does not contain the patch
• RHEL/CentOS 7.7 (kernel 3.10.0-1062) does not contain the patch
• RHEL/CentOS 8.0 (kernel 4.18.0-80) does contain the patch

[davidmlw]

Does enroot need a special argument to enable fuse? I met similar issue that enroot did not have /dev/fuse. I remember docker needs "--device /dev/fuse". I didn't see enroot manual mentions similar arguments.