apt-get update failed on overlay storage driver

[gao-feng]

Step 6 : RUN apt-get update ---> Running in 33968afa2cb0 Ign http://archive.ubuntu.com trusty InRelease Get:1 http://archive.ubuntu.com trusty-updates InRelease [65.9 kB] E: Unable to determine file size for fd 7 - fstat (2: No such file or directory)

[bergwolf]

Looks like a limitation due to 9p?

[lear@tests]$cat open_unlink_fstat.c
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>

void main(void)
{
        char *filename = "foobar";
        struct stat buf;
        int fd;

        fd = open(filename, O_CREAT|O_RDWR, 0744);
        if (fd < 0)
                perror("open failed");
        unlink(filename);

        if (fstat(fd, &buf) < 0)
                perror("fstat failed");
        close(fd);
}

On a running container:

root@6lqxvbik88:/data# ./a.out
fstat failed: No such file or directory

[bergwolf]

Above was distracted by strace apt-get update

[bergwolf]

I confirm that it fixes the problem by adding back 9p "cache=loose" mount option.

root@zg9ediym6q:/# ./data/a.out
root@zg9ediym6q:/# apt-get update
Ign http://archive.ubuntu.com trusty InRelease
Get:1 http://archive.ubuntu.com trusty-updates InRelease [65.9 kB]
Get:2 http://archive.ubuntu.com trusty-security InRelease [65.9 kB]
Hit http://archive.ubuntu.com trusty Release.gpg
Get:3 http://archive.ubuntu.com trusty-updates/main Sources [344 kB]
Get:4 http://archive.ubuntu.com trusty-updates/restricted Sources [5217 B]
Get:5 http://archive.ubuntu.com trusty-updates/universe Sources [193 kB]
Get:6 http://archive.ubuntu.com trusty-updates/main amd64 Packages [948 kB]
Get:7 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [23.5 kB]
Get:8 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [463 kB]
Get:9 http://archive.ubuntu.com trusty-security/main Sources [141 kB]
Get:10 http://archive.ubuntu.com trusty-security/restricted Sources [3920 B]
Get:11 http://archive.ubuntu.com trusty-security/universe Sources [40.9 kB]
Get:12 http://archive.ubuntu.com trusty-security/main amd64 Packages [570 kB]
Get:13 http://archive.ubuntu.com trusty-security/restricted amd64 Packages [20.2 kB]
Get:14 http://archive.ubuntu.com trusty-security/universe amd64 Packages [165 kB]
Hit http://archive.ubuntu.com trusty Release
Get:15 http://archive.ubuntu.com trusty/main Sources [1335 kB]
Get:16 http://archive.ubuntu.com trusty/restricted Sources [5335 B]
<snip>

[bergwolf]

So this is a classic unlink-after-open use case. Applications use this pattern to create an anonymous file. 9p has special handling for write-after-open-unlink case but it seems it does not handle fstat-after-open-unlink.

[gnawux]

some related info

• https://github.com/docker/docker/issues/10180
• https://bugzilla.redhat.com/show_bug.cgi?id=1213602

problem of overlay?

[bergwolf]

It is a different issue than https://bugzilla.redhat.com/show_bug.cgi?id=1213602, which is caused by overlayfs when a file lives in the down layer, opening it RO and RW can result in data inconsistency. And its workaround is to do touch /var/lib/rpm/* that copies everything up to the upper layer. Yum added a a plugin to do it.

However, in our case, the operation pattern is:

1. create a new file
2. unlink it
3. write to it several times
4. fstat

So the file is in the upper layer in the first place. And above test program (and apt-get update) passes in a docker container backed by overlay.

[gnawux]

I observed the same error with https://bugzilla.redhat.com/show_bug.cgi?id=1213602

[bergwolf]

Did you see it from apt-get update or did you write a program to do the same as https://bugzilla.redhat.com/show_bug.cgi?id=1213602 ? The same error exists if you use overlay. But it is different than as shown in strace apt-get update AFAICT.

[gnawux]

I saw it from yum install, just the same output like the bugzilla.

[bergwolf]

That should be fixed by newer yum plugin, namingly yum-plugin-ovl according to https://github.com/docker/docker/issues/10180#issuecomment-190551066

[gnawux]

no, the ovl plugin is present

[bergwolf]

With the hack patch on 9p client (https://github.com/bergwolf/linux/commit/084b4a5aa4b90975f7ac6b4714a6b449430ef4c4), I no longer see errors in both multiline cat and apt-get update:

[hyperpublic@hyperhq]$sudo hyperctl exec -t pod-iiWpsaJqIP bash
root@ubuntu-fio-4059277877:/# cat << EOF
> foo
> EOF
foo
root@ubuntu-fio-4059277877:/# apt-get update
Ign http://archive.ubuntu.com trusty InRelease
Get:1 http://archive.ubuntu.com trusty-updates InRelease [65.9 kB]
Get:2 http://archive.ubuntu.com trusty-security InRelease [65.9 kB]
Hit http://archive.ubuntu.com trusty Release.gpg
Get:3 http://archive.ubuntu.com trusty-updates/main Sources [344 kB]
Get:4 http://archive.ubuntu.com trusty-updates/restricted Sources [5217 B]
Get:5 http://archive.ubuntu.com trusty-updates/universe Sources [194 kB]
Get:6 http://archive.ubuntu.com trusty-updates/main amd64 Packages [949 kB]
Get:7 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [23.5 kB]
Get:8 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [464 kB]
Get:9 http://archive.ubuntu.com trusty-security/main Sources [143 kB]
Get:10 http://archive.ubuntu.com trusty-security/restricted Sources [3920 B]
Get:11 http://archive.ubuntu.com trusty-security/universe Sources [41.8 kB]
Get:12 http://archive.ubuntu.com trusty-security/main amd64 Packages [574 kB]
Get:13 http://archive.ubuntu.com trusty-security/restricted amd64 Packages [20.2 kB]
Get:14 http://archive.ubuntu.com trusty-security/universe amd64 Packages [165 kB]
Hit http://archive.ubuntu.com trusty Release
Hit http://archive.ubuntu.com trusty/main Sources
Hit http://archive.ubuntu.com trusty/restricted Sources
Hit http://archive.ubuntu.com trusty/universe Sources
Hit http://archive.ubuntu.com trusty/main amd64 Packages
Hit http://archive.ubuntu.com trusty/restricted amd64 Packages
Hit http://archive.ubuntu.com trusty/universe amd64 Packages
Fetched 3061 kB in 2min 25s (21.0 kB/s)
Reading package lists... Done

[bergwolf]

Well, spoke too soon... Another run gave me a different error:

root@ubuntu-8303749161:/# apt-get update
Hit:1 http://archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:3 http://archive.ubuntu.com/ubuntu xenial-security InRelease
Reading package lists... Done
W: chown to _apt:root of directory /var/lib/apt/lists/partial failed - SetupAPTPartialDirectory (1: Operation not permitted)
W: Can't drop privileges for downloading as file '/var/lib/apt/lists/partial/archive.ubuntu.com_ubuntu_dists_xenial_InRelease' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)

Although multiline cat still works:

root@ubuntu-8303749161:/# cat << EOF
foo
EOF
foo
root@ubuntu-8303749161:/#

[bergwolf]

The warning occurs because currently chown would fail in hyper container:

root@zwx2985t46:/# mkdir foo
root@zwx2985t46:/# chown nobody foo
chown: changing ownership of 'foo': Operation not permitted

[bergwolf]

yum install still fails with:

Error unpacking rpm package strace-4.8-11.el7.x86_64
error: unpacking of archive failed on file /usr/bin/strace;5729f30c: cpio: open
  Verifying  : strace-4.8-11.el7.x86_64                                     1/1

And that's because

open("/usr/bin/strace;5729e6a5", O_WRONLY|O_CREAT|O_TRUNC, 0666) = -1 EACCES (Permission denied)

And it seems that 9p's permission check does not allow root to bypass it. May be related to qemu squashing user mapping?

bash-4.2# ls -l /usr/ |grep bin
dr-xr-xr-x  2 root root 12288 Apr 17 20:08 bin
dr-xr-xr-x  2 root root  4096 Apr 17 20:08 sbin
bash-4.2# touch /usr/bin/foo
touch: cannot touch '/usr/bin/foo': Permission denied

[bergwolf]

The open() EACCESS failure boils down to qemu failing to create new files under a directory that it does not have write access to. However, since qemu is running as root, it is supposed to be able to create new files there.

local_open2 uid 0 euid 0
local_open2 open /var/run/hyper/vm-mneaNuKouV/share_dir///890d7482501827b4ea6d846373dded0ad925ce2e0efb7c1c1b8df736a34e4be1/rootfs/usr/bin/foo flags 133185 mode 420 returns -1 errno 13

The directory does not allow to be written by root.

[hyperpublic@hyperd]$ls -l /var/run/hyper/vm-hykxBPqylV/share_dir///b8d4b840ba1b1659a753dd264c08d03872d389a5e6182b8a79c5d71b59941522/rootfs/usr/
total 52
dr-xr-xr-x.  2 root root 12288 Apr 18 04:08 bin

Possibly the qemu process has lost CAP_DAC_OVERRIDE and CAP_CHOWN at some point.

[bergwolf]

All qemu process capabilities are dropped by libvirt according to https://libvirt.org/drvqemu.html#securitycap

As a result:

the QEMU process will only be able to access files owned by root, and not files owned by any other user.

And it cannot bypass 9p directory permission check due to missing CAP_DAC_OVERRIDE.

chown 9p files in a hyper container always fails as well due to QEMU process lacking CAP_CHOWN.

[bergwolf]

The capabilities issue is solved by setting clear_emulator_capabilities = 0 in /etc/libvirt/qemu.conf, as stated in https://github.com/hyperhq/hyperstart/issues/77

Then yum install -y strace goes through to the end till hitting https://github.com/hyperhq/hyperstart/issues/88

[bergwolf]

update:

The qemu 9p server does not support open()->unlink()->getattr() operation (fix available but not merged, http://patchwork.ozlabs.org/patch/626194/). kernel 9pfs client also needs fixing (patch available but not merged, https://github.com/ericvh/linux/commit/eaf70223eac094291169f5a6de580351890162a2).

So the hack (https://github.com/hyperhq/hyperstart/blob/master/build/kernel_patch/0001-HACK-9P-always-use-cached-inode-to-fill-in-v9fs_vfs_.patch) is necessary until both above fixes are merged, which seems unlikely in near future.

ref: https://bugs.launchpad.net/qemu/+bug/1336794