Container storage check plugin fails when /etc/mtab is a symlink

[JCallicoat]

The current container storage check does a chroot into /proc/<container init process pid>/root and calls psutil.disk_partitions(). This fails when /etc/mtab is a symlink to ../proc/self/mounts, which is the default configuration in current linux distributions.

OSError: [Errno 2] No such file or directory: '/proc/self'

This happens because the container is running in a separate pid and mount namespace and /proc/self is not present in the chroot.

Using nsenter with pid and mount namespace works correctly and shows the correct mounts for the container (in this example init_pid is 5169):

nsenter -m -p -t 5169 -- /bin/bash -c 'cat /etc/mtab'

It may be possible to achieve the same thing from python with something like python-nsenter. Another option would be to use the lxc.attach_wait method to run a command like df.

[JCallicoat]

It looks like python-nsenter will not work without calling out to another process since the python process itself is still in the global pid and mount namespace.

Note that this works if I change the os.path.realpath call to something like os.system('cat /etc/mtab') because the new cat process inherits the pid/mnt namespace of the container we entered.

We could however use something like this and the subprocess module to call out to df for example.

import os
import lxc
import nsenter
try:
    from contextlib import ExitStack
except ImportError:
    from contextlib2 import ExitStack

c = lxc.Container('test_container')
print(c.init_pid)

with ExitStack() as stack:
    mnt_ns = nsenter.Namespace(c.init_pid, 'mnt')
    pid_ns = nsenter.Namespace(c.init_pid, 'pid')
    stack.enter_context(pid_ns)
    stack.enter_context(mnt_ns)
    print('checking mtab')
    p = os.path.realpath('/etc/mtab')
    print(open(p).read())

# python /tmp/nsenter/test.py
5169
Entering pid namespace 5169
Entering mnt namespace 5169
checking mtab
Leaving mnt namespace 5169
Leaving pid namespace 5169
Traceback (most recent call last):
  File "/tmp/nsenter/test.py", line 53, in <module>
    p = os.path.realpath('/etc/mtab')
  File "/usr/lib/python2.7/posixpath.py", line 375, in realpath
    path, ok = _joinrealpath('', filename, {})
  File "/usr/lib/python2.7/posixpath.py", line 414, in _joinrealpath
    path, ok = _joinrealpath(path, os.readlink(newpath), seen)
  File "/usr/lib/python2.7/posixpath.py", line 414, in _joinrealpath
    path, ok = _joinrealpath(path, os.readlink(newpath), seen)
OSError: [Errno 2] No such file or directory: '/proc/self'

[JCallicoat]

It's also possible to call out to another process in the container pid/mnt namepace with the lxc module itself rather than using the nsenter module, with the attach_wait method on Container objects. It's a bit ugly because you have to use a pipe to get the output from attach_wait.

import os
import lxc

c = lxc.Container('test_container')
print(c.init_pid)

r, w = os.pipe()

with os.fdopen(w, 'w') as fh:
  c.attach_wait(lxc.attach_run_command, ["df", "-h"], stdout=fh)

with os.fdopen(r) as fh:
  for l in fh.read().splitlines()[1:]:
    print(l)

# python /tmp/nsenter/test.py
5169
/dev/lxc/test_container      4.8G  835M  3.8G  18% /
none                         492K     0  492K   0% /dev
/dev/mapper/lxc-openstack00  235G   86G  137G  39% /var/log
tmpfs                        126G     0  126G   0% /dev/shm
tmpfs                        126G  8.2M  126G   1% /run
tmpfs                        5.0M     0  5.0M   0% /run/lock
tmpfs                        126G     0  126G   0% /sys/fs/cgroup