chrooting a cpython process causes weird errors

[ssssam]

For example:

[0/1/128] [base-system-x86_64-generic] Running install-commands
[0/1/128] [base-system-x86_64-generic] Running command:
if which depmod; then (cd /lib/modules && for version in *; do depmod -a "$version"; done) fi 
Traceback (most recent call last):
  File "/src/ybd/ybd/sandbox.py", line 173, in run_sandboxed
    env=env, **config)
  File "/usr/local/lib/python2.7/dist-packages/sandboxlib/chroot.py", line 227, in run_sandbox_with_redirection
    exit, out, err = run_sandbox(command, **sandbox_config)
  File "/usr/local/lib/python2.7/dist-packages/sandboxlib/chroot.py", line 223, in run_sandbox
    raise exception
LookupError: unknown encoding: string-escape
[0/1/128] [SANDBOX] ERROR: in run_sandbox_with_redirection 99

As pointed out here, the cpython process has some of its functionality in external modules. It expects to be able to load these at any time.

I looked at reimplementing the chroot backend by calling /bin/chroot, but this has a couple of downsides:

• /bin/chroot program is required on the host
• /bin/sh is required in the chroot, if we want be able to set current directory to anything other than /

Not sure what is the best way forward right now.

[richardmaw-codethink]

On Tue, May 10, 2016 at 08:04:30AM -0700, Sam Thursfield wrote:

For example:

[0/1/128] [base-system-x86_64-generic] Running install-commands
[0/1/128] [base-system-x86_64-generic] Running command:
if which depmod; then (cd /lib/modules && for version in *; do depmod -a "$version"; done) fi 
Traceback (most recent call last):
  File "/src/ybd/ybd/sandbox.py", line 173, in run_sandboxed
    env=env, **config)
  File "/usr/local/lib/python2.7/dist-packages/sandboxlib/chroot.py", line 227, in run_sandbox_with_redirection
    exit, out, err = run_sandbox(command, **sandbox_config)
  File "/usr/local/lib/python2.7/dist-packages/sandboxlib/chroot.py", line 223, in run_sandbox
    raise exception
LookupError: unknown encoding: string-escape
[0/1/128] [SANDBOX] ERROR: in run_sandbox_with_redirection 99

As pointed out here, the cpython process has some of its functionality in external modules. It expects to be able to load these at any time.

I looked at reimplementing the chroot backend by calling /bin/chroot, but this has a couple of downsides:

• /bin/chroot program is required on the host
• /bin/sh is required in the chroot, if we want be able to set current directory to anything other than /

Not sure what is the best way forward right now.

It's not generally solvable with python's lazily loading of modules.

If you can guarantee /proc is mounted in the chroot, you can open file descriptors to all the old search path roots, and replace the search path with '/proc/self/fd/%d' % dirfd.

Something like the following may work.

with contextlib2.ExitStack() as dirfds_context:
    old_searchpath = tuple(sys.path)
    new_searchpath = []
    for dir in sys.path:
        fd = os.open(dir if dir != '' else '.', os.O_DIRECTORY)
        @dirfds_context.callback
        def closefd():
            os.close(fd)
        new_searchpath.append('/proc/self/fd/%d' % fd)
    @dirfds_context.callback
    def resetsearchpath():
        sys.path[:] = old_searchpath
    sys.path[:] = new_searchpath

    os.chroot(chroot_path)
    …

[ssssam]

That assumes a specific layout for /proc, can we be sure that it'll have /proc/self/fd on all platforms? The main point of this chroot backend (and of this library existing at all) is to support some level of containment on non-Linux platforms, after all

[ssssam]

It seems only one thing uses "string-escape" encoding: pickle. So I think the problem here is something tried to use pickle, which should really not be necessary when all we are doing is executing a subprocess.

[ssssam]

The subprocess module seems to call pickle to transfer exceptions, so maybe that's what triggers this error. It may in fact be hiding a different exception.

[richardmaw-codethink]

On Tue, May 10, 2016 at 08:38:04AM -0700, Sam Thursfield wrote:

That assumes a specific layout for /proc, can we be sure that it'll have /proc/self/fd on all platforms?

On all Linux platforms, yes.

On GNU/Hurd, also yes, since there are Linux procfs emulation daemons.

BSDs, probably not. I know that the only reason Debian got to use sysvinit on the BSD port as well as the Linux one is that they factored the /proc access out into a library

The main point of this chroot backend (and of this library existing at all) is to support some level of containment on non-Linux platforms, after all.

Also other Linuxes that don't have linux-user-chroot packaged, which would benefit from having the package search path fixed.

Otherwise, your best approach would be to trace which libraries are needed and import them ahead of time, so they are in-memory and don't need to be loaded after chrooting.

[ssssam]

Hopefully https://github.com/CodethinkLabs/sandboxlib/pull/20 has worked around this issue.