sandstorm does not start

[antihec]

Using https://docs.sandstorm.io/en/latest/install/#option-4-installing-from-source

make worked, make install:

Your server is coming online. Waiting up to 90 seconds.............................................................................................*** INSTALLATION FAILED ***

Your server never started listening.

You can report bugs at: http://github.com/sandstorm-io/sandstorm
Makefile:135: recipe for target 'install' failed
make: *** [install] Error 1

---

strace -f /opt/sandstorm/latest/sandstorm start

[...]
[pid 15289] stat(".", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
[pid 15289] access("sandstorm", F_OK)   = 0
[pid 15289] stat("sandstorm", {st_mode=S_IFREG|0755, st_size=5074688, ...}) = 0
[pid 15289] access("../sandstorm.conf", F_OK) = 0
[pid 15289] stat("../sandstorm.conf", {st_mode=S_IFREG|0644, st_size=182, ...}) = 0
[pid 15289] unshare(CLONE_NEWNS)        = 0
[pid 15289] mount("none", "/", NULL, MS_REC|MS_PRIVATE, NULL) = -1 EACCES (Permission denied)
[pid 15289] brk(0x24f1000)              = 0x24f1000
[pid 15289] futex(0xad70c0, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[pid 15289] close(4)                    = 0
[pid 15289] close(3)                    = 0
[pid 15289] writev(2, [{"*** Uncaught exception ***\nsands"..., 212}, {"\n", 1}], 2) = 213
[pid 15289] exit_group(1)               = ?
[pid 15289] +++ exited with 1 +++
[pid 15287] <... read resumed> "\21\0\0\0\0\0\0\0\1\0\0\0\3\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 128) = 128
[pid 15287] wait4(-1, [{WIFEXITED(s) && WEXITSTATUS(s) == 1}], WNOHANG, NULL) = 3
[pid 15287] wait4(-1, 0x7ffef8e98e8c, WNOHANG, NULL) = 0
[pid 15287] writev(2, [{"** Server monitor died. Aborting"..., 33}, {"\n", 1}], 2) = 34
[pid 15287] exit_group(1)               = ?
[pid 15288] +++ killed by SIGKILL +++
+++ exited with 1 +++

``

[dwrensha]

Thanks for the report!

Just as a bit of preliminary diagnosis: it looks like the problem is that the mount() call here is failing with EACESS.

[kentonv]

By any chance are you running inside a Docker container?

[antihec]

No docker but yes, I tried to install and start sandstorm inside a container for testing.

[paulproteus]

It seems to me that the install script should test for this condition so people know what to change before Sandstorm sort-of-starts.

(Note that I maintain the install script!)

[kentonv]

@antihec Interesting. What kind of container?

[antihec]

Of the LXC kind, I set it up in userspace using lxd/lxc stack https://linuxcontainers.org/lxd/ like so:

Install lxd/lxc and add image resource:

  add-apt-repository ppa:ubuntu-lxc/lxd-stable
  apt-get update
  apt-get dist-upgrade
  apt-get install lxd
  lxc remote add images images.linuxcontainers.org

Launch a container to try out sandstorm, and install requirements:

  lxc launch images:ubuntu/wily/amd64 sandstorm
  lxc exec sandstorm -- apt-get update
  lxc exec sandstorm -- apt-get upgrade
  lxc exec sandstorm -- apt-get install -y build-essential libcap-dev \
                                xz-utils zip unzip imagemagick strace curl \
                                clang discount git
  lxc exec sandstorm -- bash

In container:

root@sandstorm:~# curl https://install.meteor.com/ | sh

(note that in container it's root, while on the host machine root is not required to set up the container)

Also in container:

  git clone https://github.com/sandstorm-io/sandstorm.git
  cd sandstorm
  make
  make install

[kentonv]

So it's failing on:

KJ_SYSCALL(mount("none", "/", nullptr, MS_REC | MS_PRIVATE, nullptr));

This call is supposed to make sure that none of our mounts are marked "shared", because if they are, then any further changes we make to them will be visible outside the sandbox. Note that the meaning of MS_REC and MS_PRIVATE are not documented, but presumably MS_REC means "recursively apply the same change to child mounts" and MS_PRIVATE means "change shared mounts to private".

Apparently this call sometimes fails inside of containers (Docker, LXC), claiming EACCES. Two theories for why that might be:

1. Since we didn't pass MS_READONLY, MS_NOEXEC, MS_NOSUID, nor MS_NODEV as flags, the utterly awful mount() API believe that we are trying to un-set these flags, which may not be allowed when we aren't true root. Of course, it is not actually our intent to change these flags. But I don't think this theory is correct because we aren't specifying MS_REMOUNT, which is normally how you specify that you're trying to mess with the mount flags.
2. It could be that MS_PRIVATE itself is sometimes disallowed in a container. I don't know why this would be. This flag is undocumented.

@amluto any ideas?

[amluto]

I'm guessing it's a selinux bug:

    if (flags & MS_REMOUNT)
        return superblock_has_perm(cred, path->dentry->d_sb,
                       FILESYSTEM__REMOUNT, NULL);
    else
        return path_has_perm(cred, path, FILE__MOUNTON);

Possibly there should be a special case.

@antihec, are you using selinux? If so, do you see anything in your audit log?

If this is the problem, I'll ask the selinux folks to fix it.

[antihec]

@amluto I am not - consciously :) - using selinux. The installation detailled above, I tried on a pretty clean Ubuntu 15.10 box. Maybe Wily has selinux by default? I will check on that when I have a chance.

[amluto]

I doubt that Ubuntu uses selinux.

The EACCES error return in particular is very strange, since almost all the failure paths return EPERM.

@antihec, did you check the kernel logs?

[antihec]

@amluto had a look now. On the Host:

Feb 25 14:46:58 host audit[29954]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-sandstorm_</var/lib/lxd>" name="/" pid=29954 comm="sandstorm" flags="rw, rprivate"
Feb 25 14:46:58 host kernel: audit: type=1400 audit(1456408018.472:48): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-sandstorm_</var/lib/lxd>" name="/" pid=29954 comm="sandstorm" flags="rw, rprivate"

Not sure what the culprit really is here - I may have to look into creating a more specific apparmor profile for the detailed requirements of sandstorm here.

[btheado]

I hit the same error while running via docker. I followed the instructions at https://docs.sandstorm.io/en/latest/install/#option-6-using-sandstorm-within-docker:

$ docker run -i -t --cap-add SYS_ADMIN --security-opt seccomp=unconfined -v sandstorm-data-volume:/opt/sandstorm --name sandstorm-build buildpack-deps bash -c 'useradd --system --user-group sandstorm ; curl https://install.sandstorm.io/ > install.sh && REPORT=no bash install.sh -d -e'
$ docker run -i -t --sig-proxy=true -p 0.0.0.0:6080:6080 --cap-add SYS_ADMIN --security-opt seccomp=unconfined -v sandstorm-data-volume:/opt/sandstorm buildpack-deps bash -c 'useradd --system --user-group sandstorm && /opt/sandstorm/sandstorm start && tail -f /opt/sandstorm/var/log/sandstorm.log & sleep infinity'

And I get this:

** Starting Sandstorm at: Sat Aug 27 23:27:57 2016
*** Uncaught exception ***
sandstorm/run-bundle.c++:1283: failed: mount("none", "/", nullptr, MS_REC | MS_PRIVATE, nullptr): Permission denied
stack: 0x4da553 0x4d9706 0x4f3650 0x4f29b9 0x4f29aa 0x48b1af 0x48acca
** Server monitor died. Aborting.

If I add --privileged to the 2nd docker run, then I don't get the error and it works fine. I tried to check if I needed another capability in addition to SYS_ADMIN, but replacing --privileged with --cap-add ALL gave the error again.

The docker docs at https://docs.docker.com/engine/reference/run/#/runtime-privilege-and-linux-capabilities say

When the operator executes docker run --privileged, Docker will enable to access to all devices on the host as well as set some configuration in AppArmor or SELinux to allow the container nearly all the same access to the host as processes running outside containers on the host.

So if it isn't a capability, then maybe it is a device I need to add? Any ideas?

Using privileged is fine as a solution. Just thought this report might be useful as others following the docker instructions may hit the same issue.

I'm running on Ubuntu 16.04 and docker version 1.11.2.

[paulproteus]

Hi @btheado ,

Thanks for doing this research!

I wonder if SELinux and/or AppArmor are related to the problem. I'm personally hopeful that we can be specific about which privileges we're requesting. Having said all that, I wonder - as a Docker user, do you care if we specify a complex set of --security-optlines? Or do you prefer the simpler "--privileged`?

I also wonder if you can try a couple of other things. First, can you try adding:

--security-opt=label:disable

instead of --privileged and see if that does the right thing?

If that doesn't work, can you try adding:

--security-opt apparmor:unconfined

I'm curious what the root cause is, even if the simplest thing to document is --privileged. If you're willing to try these things, I'd appreciate it! If not, then let me know. Either way, thanks for testing out a variety of things already and leaving a comment on this GitHub issue!

[btheado]

Thanks for the suggestions. I tried them and --security-opt=label:disable didn't help, but --security-opt apparmor:unconfined did.

• --cap-add SYS_ADMIN --security-opt=label:disable --security-opt seccomp=unconfined - same mount error as before
• --cap-add SYS_ADMIN --security-opt=apparmor:unconfined --security-opt seccomp=unconfined - worked
• --cap-add SYS_ADMIN --security-opt=apparmor:unconfined - worked

About whether I prefer the the simpler --privileged, I'm not sure. The finer grained --security-opt lines give the perception of being more secure (principle of least privilege and all), but whether it actually adds anything, I wouldn't know.

[amluto]

I'm mystified. The obvious issue would be a bug in AppArmor's sb_mount hook, but AppArmor doesn't have an sb_mount hook.

@btheado, what kernel are you using? I'm wondering if there's some out-of-tree code involved.

[amluto]

The failing code path is here:

http://kernel.ubuntu.com/git/ubuntu/ubuntu-wily.git/tree/security/apparmor/mount.c#n444

and this doesn't exist in the upstream kernel. I'm pinging the maintainers of this code.

[amluto]

The Ubuntu version appears to have a bug in aa_mount_change_type, which calls match_mnt, which calls match_mnt_flags, which is bogus in this context, because change_type requests don't specify or change flags. (Or maybe I'm confused. I have no idea what the dfa stuff in there is supposed to do.)

[btheado]

@amluto, 4.4.0-31-generic is what shows up in my uname -a output.