Open antihec opened 8 years ago
Thanks for the report!
Just as a bit of preliminary diagnosis: it looks like the problem is that the mount()
call here is failing with EACESS
.
By any chance are you running inside a Docker container?
No docker but yes, I tried to install and start sandstorm inside a container for testing.
It seems to me that the install script should test for this condition so people know what to change before Sandstorm sort-of-starts.
(Note that I maintain the install script!)
@antihec Interesting. What kind of container?
Of the LXC kind, I set it up in userspace using lxd/lxc stack https://linuxcontainers.org/lxd/ like so:
Install lxd/lxc and add image resource:
add-apt-repository ppa:ubuntu-lxc/lxd-stable
apt-get update
apt-get dist-upgrade
apt-get install lxd
lxc remote add images images.linuxcontainers.org
Launch a container to try out sandstorm, and install requirements:
lxc launch images:ubuntu/wily/amd64 sandstorm
lxc exec sandstorm -- apt-get update
lxc exec sandstorm -- apt-get upgrade
lxc exec sandstorm -- apt-get install -y build-essential libcap-dev \
xz-utils zip unzip imagemagick strace curl \
clang discount git
lxc exec sandstorm -- bash
In container:
root@sandstorm:~# curl https://install.meteor.com/ | sh
(note that in container it's root, while on the host machine root is not required to set up the container)
Also in container:
git clone https://github.com/sandstorm-io/sandstorm.git
cd sandstorm
make
make install
So it's failing on:
KJ_SYSCALL(mount("none", "/", nullptr, MS_REC | MS_PRIVATE, nullptr));
This call is supposed to make sure that none of our mounts are marked "shared", because if they are, then any further changes we make to them will be visible outside the sandbox. Note that the meaning of MS_REC and MS_PRIVATE are not documented, but presumably MS_REC means "recursively apply the same change to child mounts" and MS_PRIVATE means "change shared mounts to private".
Apparently this call sometimes fails inside of containers (Docker, LXC), claiming EACCES. Two theories for why that might be:
@amluto any ideas?
I'm guessing it's a selinux bug:
if (flags & MS_REMOUNT)
return superblock_has_perm(cred, path->dentry->d_sb,
FILESYSTEM__REMOUNT, NULL);
else
return path_has_perm(cred, path, FILE__MOUNTON);
Possibly there should be a special case.
@antihec, are you using selinux? If so, do you see anything in your audit log?
If this is the problem, I'll ask the selinux folks to fix it.
@amluto I am not - consciously :) - using selinux. The installation detailled above, I tried on a pretty clean Ubuntu 15.10 box. Maybe Wily has selinux by default? I will check on that when I have a chance.
I doubt that Ubuntu uses selinux.
The EACCES error return in particular is very strange, since almost all the failure paths return EPERM.
@antihec, did you check the kernel logs?
@amluto had a look now. On the Host:
Feb 25 14:46:58 host audit[29954]: AVC apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-sandstorm_</var/lib/lxd>" name="/" pid=29954 comm="sandstorm" flags="rw, rprivate"
Feb 25 14:46:58 host kernel: audit: type=1400 audit(1456408018.472:48): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-sandstorm_</var/lib/lxd>" name="/" pid=29954 comm="sandstorm" flags="rw, rprivate"
Not sure what the culprit really is here - I may have to look into creating a more specific apparmor profile for the detailed requirements of sandstorm here.
I hit the same error while running via docker. I followed the instructions at https://docs.sandstorm.io/en/latest/install/#option-6-using-sandstorm-within-docker:
$ docker run -i -t --cap-add SYS_ADMIN --security-opt seccomp=unconfined -v sandstorm-data-volume:/opt/sandstorm --name sandstorm-build buildpack-deps bash -c 'useradd --system --user-group sandstorm ; curl https://install.sandstorm.io/ > install.sh && REPORT=no bash install.sh -d -e'
$ docker run -i -t --sig-proxy=true -p 0.0.0.0:6080:6080 --cap-add SYS_ADMIN --security-opt seccomp=unconfined -v sandstorm-data-volume:/opt/sandstorm buildpack-deps bash -c 'useradd --system --user-group sandstorm && /opt/sandstorm/sandstorm start && tail -f /opt/sandstorm/var/log/sandstorm.log & sleep infinity'
And I get this:
** Starting Sandstorm at: Sat Aug 27 23:27:57 2016
*** Uncaught exception ***
sandstorm/run-bundle.c++:1283: failed: mount("none", "/", nullptr, MS_REC | MS_PRIVATE, nullptr): Permission denied
stack: 0x4da553 0x4d9706 0x4f3650 0x4f29b9 0x4f29aa 0x48b1af 0x48acca
** Server monitor died. Aborting.
If I add --privileged
to the 2nd docker run, then I don't get the error and it works fine. I tried to check if I needed another capability in addition to SYS_ADMIN
, but replacing --privileged
with --cap-add ALL
gave the error again.
The docker docs at https://docs.docker.com/engine/reference/run/#/runtime-privilege-and-linux-capabilities say
When the operator executes docker run --privileged, Docker will enable to access to all devices on the host as well as set some configuration in AppArmor or SELinux to allow the container nearly all the same access to the host as processes running outside containers on the host.
So if it isn't a capability, then maybe it is a device I need to add? Any ideas?
Using privileged
is fine as a solution. Just thought this report might be useful as others following the docker instructions may hit the same issue.
I'm running on Ubuntu 16.04 and docker version 1.11.2.
Hi @btheado ,
Thanks for doing this research!
I wonder if SELinux and/or AppArmor are related to the problem. I'm personally hopeful that we can be specific about which privileges we're requesting. Having said all that, I wonder - as a Docker user, do you care if we specify a complex set of --security-opt
lines? Or do you prefer the simpler "--privileged`?
I also wonder if you can try a couple of other things. First, can you try adding:
--security-opt=label:disable
instead of --privileged
and see if that does the right thing?
If that doesn't work, can you try adding:
--security-opt apparmor:unconfined
I'm curious what the root cause is, even if the simplest thing to document is --privileged
. If you're willing to try these things, I'd appreciate it! If not, then let me know. Either way, thanks for testing out a variety of things already and leaving a comment on this GitHub issue!
Thanks for the suggestions. I tried them and --security-opt=label:disable
didn't help, but --security-opt apparmor:unconfined
did.
--cap-add SYS_ADMIN --security-opt=label:disable --security-opt seccomp=unconfined
- same mount error as before--cap-add SYS_ADMIN --security-opt=apparmor:unconfined --security-opt seccomp=unconfined
- worked--cap-add SYS_ADMIN --security-opt=apparmor:unconfined
- workedAbout whether I prefer the the simpler --privileged
, I'm not sure. The finer grained --security-opt
lines give the perception of being more secure (principle of least privilege and all), but whether it actually adds anything, I wouldn't know.
I'm mystified. The obvious issue would be a bug in AppArmor's sb_mount
hook, but AppArmor doesn't have an sb_mount
hook.
@btheado, what kernel are you using? I'm wondering if there's some out-of-tree code involved.
The failing code path is here:
http://kernel.ubuntu.com/git/ubuntu/ubuntu-wily.git/tree/security/apparmor/mount.c#n444
and this doesn't exist in the upstream kernel. I'm pinging the maintainers of this code.
The Ubuntu version appears to have a bug in aa_mount_change_type, which calls match_mnt, which calls match_mnt_flags, which is bogus in this context, because change_type requests don't specify or change flags. (Or maybe I'm confused. I have no idea what the dfa stuff in there is supposed to do.)
@amluto, 4.4.0-31-generic
is what shows up in my uname -a output.
Using https://docs.sandstorm.io/en/latest/install/#option-4-installing-from-source
make worked, make install:
strace -f /opt/sandstorm/latest/sandstorm start