ERROR while doing enroot import : enroot-aufs2ovlfs: failed to create ovlfs whiteout

[KapilS25]

Hi, I am facing issue while doing enroot import . ---------------------------------------------------------------------------------- Description : Enroot Version : 3.2.0

$ echo $TMPDIR /var/tmp/pbs.1512380.pbshpc

Inside enroot.conf ENROOT_TEMP_PATH ${TMPDIR:-/tmp}

-------------------------------------------------------------------------

enroot import docker://dispel4py/docker.openmpi

[INFO] Querying registry for permission grant [INFO] Authenticating with user: [INFO] Authentication succeeded [INFO] Fetching image manifest list [INFO] Fetching image manifest [INFO] Found all layers in cache [INFO] Extracting image layers...

100% 41:0=0s 064f9af025390d8da3dfab763fac261dd67f8807343613239d66304cda8f5d16

[INFO] Converting whiteouts...

90% 37:4=0s 064f9af025390d8da3dfab763fac261dd67f8807343613239d66304cda8f5d16 enroot-aufs2ovlfs: failed to create ovlfs whiteout: /var/tmp/pbs.1512380.pbshpc/enroot.dvToIaBvG7/39/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty_restricted_binary-amd64_Packages: Operation not permitted 95% 39:2=0s 064f9af025390d8da3dfab763fac261dd67f8807343613239d66304cda8f5d16 enroot-aufs2ovlfs: failed to create ovlfs whiteout: /var/tmp/pbs.1512380.pbshpc/enroot.dvToIaBvG7/37/var/cache/apt/pkgcache.bin: Operation not permitted 100% 41:0=0s 064f9af025390d8da3dfab763fac261dd67f8807343613239d66304cda8f5d16

enroot import docker://sysmso/docker-openmpi

[INFO] Querying registry for permission grant [INFO] Authenticating with user: [INFO] Authentication succeeded [INFO] Fetching image manifest list [INFO] Fetching image manifest [INFO] Found all layers in cache [INFO] Extracting image layers...

100% 9:0=0s a48c500ed24e62926cb079df35f964c57d8bb08159b1d29c6a3b0a58dc365dc1

[INFO] Converting whiteouts...

44% 4:5=0s 1e1de00ff7e1fea0858b6a4b5ca208eeca970607ea9a6eb5fc972494e7a0cdde enroot-aufs2ovlfs: failed to create ovlfs whiteout: /var/tmp/pbs.1512380.pbshpc/enroot.iNvNjgQYdQ/7/var/lib/apt/lists/auxfiles: Operation not permitted 100% 9:0=0s a48c500ed24e62926cb079df35f964c57d8bb08159b1d29c6a3b0a58dc365dc1

--------------------------------------------------------- WORKING IN CASE

enroot import docker://ubuntu [INFO] Querying registry for permission grant [INFO] Authenticating with user: [INFO] Authentication succeeded [INFO] Fetching image manifest list [INFO] Fetching image manifest [INFO] Found all layers in cache [INFO] Extracting image layers...

100% 3:0=0s 5d3b2c2d21bba59850dac063bcbb574fddcb6aefb444ffcc63843355d878d54f

[INFO] Converting whiteouts...

100% 3:0=0s 5d3b2c2d21bba59850dac063bcbb574fddcb6aefb444ffcc63843355d878d54f

[INFO] Creating squashfs filesystem...

Parallel mksquashfs: Using 24 processors Creating 4.0 filesystem on /scratch/cc/vfaculty/skapil.vfaculty/ubuntu.sqsh, block size 131072. [============================================================================================================-] 2761/2761 100%

Exportable Squashfs 4.0 filesystem, lzo compressed, data block size 131072 uncompressed data, compressed metadata, compressed fragments, compressed xattrs, compressed ids duplicates are removed Filesystem size 51815.18 Kbytes (50.60 Mbytes) 72.58% of uncompressed filesystem size (71388.69 Kbytes) Inode table size 40140 bytes (39.20 Kbytes) 37.35% of uncompressed inode table size (107466 bytes) Directory table size 33061 bytes (32.29 Kbytes) 51.07% of uncompressed directory table size (64734 bytes) Number of duplicate files found 110 Number of inodes 3263 Number of files 2501 Number of fragments 286 Number of symbolic links 184 Number of device nodes 0 Number of fifo nodes 0 Number of socket nodes 0 Number of directories 578 Number of ids (unique uids + gids) 1 Number of uids 1 root (0) Number of gids 1 root (0) -------------------------------------------------------------------------

[3XX0]

Do you have the +caps package installed and is SeLinux enabled?

[KapilS25]

Yes, we installed libcap-2.46 from source as a dependency for enroot. OS : CentOS 7.6 Can you please tell me how to enable SeLinux ? Makefile.txt Make.Rules.txt

Please find the attached Make.Rules & Makefile used for installation, suggest changes required if any.

[3XX0]

Not the libcap package, the enroot cap package listed here: https://github.com/NVIDIA/enroot/blob/master/doc/installation.md

If you are using SELinux you can try disabling it with sudo setenforce 0

[KapilS25]

As in my case enroot installation is also done from source, how to install enroot+cap package from source, i am unable to find out the source code. On my system output of setenforce 0 is $ setenforce 0 setenforce: SELinux is disabled

i. e. already disabled

[3XX0]

make setcap

https://github.com/NVIDIA/enroot/blob/master/doc/installation.md

[KapilS25]

make setcap is already done with sudo.

make setcap output: setcap cap_sys_admin+pe /home/apps/centos7/enroot/3.2.0/bin/enroot-mksquashovlfs setcap cap_sys_admin,cap_mknod+pe /home/apps/centos7/enroot/3.2.0/bin/enroot-aufs2ovlfs.

Still there is a issue.

[3XX0]

What type of filesystem is your ENROOT_TEMP_PATH (i.e. /var/tmp/pbs.1512380.pbshpc)?

[KapilS25]

File System : lustre /var/tmp/pbs.1512380.pbshpc TMPDIR set by PBS.

[KapilS25]

I checked, without PBS its working fine. What is a significance of ENROOT_TEMP_PATH ? What is the default value for that ?

[3XX0]

It is used as a temporary directory for image conversion. It shouldn't be a network storage, use a tmpfs or a temporary local directory

[KapilS25]

Hi , i am still facing this issue

[KapilS25]

Function do_mknod(whiteout) is returning -1 in my case i think so can you please help me to resolve this . https://github.com/NVIDIA/enroot/blob/master/bin/enroot-aufs2ovlfs.c

[3XX0]

Did you change ENROOT_TEMP_PATH to a local directory?

[KapilS25]

Yes

[3XX0]

Did you make sure that you can write to ENROOT_TEMP_PATH (i.e. check permission, selinux, etc), and that the binary has the correct capabilities (getcap /usr/bin/enroot-aufs2ovlfs)?

Also to be sure, check that you're not already in a user namespace and that ENROOT_TEMP_PATH isn't under a mount with nodev

You should be able to create a whiteout manually if everything checks out:

sudo capsh --caps="cap_mknod+eip cap_setpcap,cap_setuid,cap_setgid+ep" \
                   --user=nobody --addamb=cap_mknod \
                   -- -c "mknod ${ENROOT_TEMP_PATH}/foo c 0 0"

[KapilS25]

getcap enroot-aufs2ovlfs enroot-aufs2ovlfs cap_sys_admin,cap_mknod=ep

[3XX0]

Have you checked the above too?

Also check dmesg maybe it's being reported there, and try with different directories see if it changes anything