Not able to run plash from inside a docker container

[nikvdp]

Hi, Love the idea and approach. I'm primarily interested in using plash inside docker containers, but am getting the below message and am unsure how to proceed. Any ideas what's going on here?

root@2ac09b5c5e7b:/# echo 'plash reference' | plash run --from ubuntu --apt figlet -- figlet
plash error: calling `unshare(CLONE_NEWNS)` returned EPERM
plash error: build failed with exit status 1

For reference, here is what I tried:

$ docker run -it ubuntu bash
root@0409d103a1b5:/# apt-get update && apt-get install -y python3-pip
Get:1 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [107 kB]
Get:3 http://security.ubuntu.com/ubuntu xenial-security/universe Sources [93.8 kB] 
...
Updating certificates in /etc/ssl/certs...
148 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
root@0409d103a1b5:/# pip3 install plash
root@0409d103a1b5:/# plash init
root@0409d103a1b5:/# echo 'plash reference' | plash run --from ubuntu --apt figlet -- figlet
plash: fetching 100%
plash: extracting...
plash error: calling `unshare(CLONE_NEWNS)` returned EPERM
plash error: build failed with exit status 1

In case it's helpful for debugging I also tried running the same container with --privileged, which led to a different result:

$ docker run -it --privileged ubuntu bash
root@0409d103a1b5:/# apt-get update && apt-get install -y python3-pip
Get:1 http://archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [107 kB]
Get:3 http://security.ubuntu.com/ubuntu xenial-security/universe Sources [93.8 kB] 
...
Updating certificates in /etc/ssl/certs...
148 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
root@0409d103a1b5:/# pip3 install plash
root@0409d103a1b5:/# plash init
root@be46c9cc9c83:/# echo 'plash reference' | plash run --from ubuntu --apt figlet -- figlet
mount: wrong fs type, bad option, bad superblock on overlay,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.
plash error: Command '['mount', '-t', 'overlay', 'overlay', '-o', 'lowerdir=/var/lib/plash/index/1/_data/root:/var/lib/plash/index/0/_data/root,workdir=/var/lib/plash/tmp/plashtmp_1_856_s2d2ayka/work,up
perdir=/var/lib/plash/tmp/plashtmp_1_856_s2d2ayka/data', '/var/lib/plash/mnt']' returned non-zero exit status 32
plash error: build failed with exit status 1

[nikvdp]

I found https://github.com/ihucos/docker-plash/ shortly after posting, and with a little fiddling was able to get that to run plash inside a container. I see it's not officially released yet though, so please disregard above and if I can provide any help in getting docker-plash release-ready let me know!

I noticed in https://github.com/ihucos/docker-plash/blob/master/plash#L4 that you're using --privileged, is it possible to run plash inside a docker container without --privileged?

[ihucos]

Hello!

thanks for the issue, I am really happy that people are slowly writing issues and pull requests.

I'll document it in the wiki or so better. Yesterday I put something in the REAMDE at the "Recipes" section that might help. You can't have an overlay mount on top of another overlay mount (but a unionfs-fuse mount works everywhere). So your plash data directory needs to be something else, like a docker volume. It works at https://github.com/ihucos/docker-plash/ because there is the additional flag --volume plashdata:/var/lib/plash. What would also work is making that an--tmpfs, explicitly mounting tmpfs or something else inside the container or alternatively using unionfs-fuse instead of overlay - yeah I definitely need to document these things somewhere.

Regarding --privileged: You can also run plash inside containers that where not started with --privileged. For this you need to export the environment variable export PLASH_NO_UNSHARE=1. That is kind of a failsafe mode that really should work everywhere. During development I ran plash natively in MacOS and FreeBSD. The problem with this mode, is that it will leak into your global mount namespace, which is OK in a short lived (docker) container. It may be possible to keep this approach clean by explicitly unmounting containers that finished running, I want to revisit this at some point.

[...] if I can provide any help in getting docker-plash release-ready let me know!

Thanks, I used that to run plash inside my MacOS laptop with docker. In fact a big portion of plash's development happened with this setup. It works fairly well. It is marked as not released, because first the main focus was in the core components of plash. But maybe I remember some issues because in in mac user are at /Users and not /home. Oh yeah, now I remeber there was some kind of "deal breaker" that I just risked for myself. Plash containers usually mount the /home directory into the container. But I read that you should'nt use a driectory via dockers --volume with multiple instances at the same time because of possible data loss or corruption. So If you want you can try to find out and clarify if that really could happen. If that is safe, I am happy making a nicer README and marking it as released as beta or so

Feel free to write issues or email (mail at irae.me) for issues you encounter and hopefully we can get you going with plash.

EDIT: An additional note, that I'd also test ihucos/docker-plash again to ensure some quality before marking it as beta, of course :-)

[ihucos]

A quick note on this error message: plash error: calling `unshare(CLONE_NEWNS)` returned EPERM It means that a new mount namespace could not be created, which docker seems to disallow when --privileged is not specified. That is where you can as written use PLASH_NO_UNSHARE (leaks mounts). Maybe there is a docker option to allow just that?

[nikvdp]

Thanks for the detailed response! Had some time to experiment a little more today, here's what I found:

• The irae/docker-plash image seems to be out of date, I wasn't able to get anything to work when I pulled that image (perhaps there were changes to the plash CLI since the image was built?), but everything works fine if I do a docker build with the included Dockerfile.

• After some reading through docker's docs on runtime capabilities was able to narrow things down a bit: plash only seems to require the SYS_ADMIN privilege. Running plash in docker via docker run --cap-add SYS_ADMIN instead of docker run --privileged seems to works fine.

• Without either --cap-add SYS_ADMIN or --privileged I couldn't get plash to work from inside docker at all though, even with PLASH_NO_UNSHARE=1 set and exported. Here is what I tried:

  $ docker run -e PLASH_NO_UNSHARE=1 --tmpfs /tmp --volume plashdata:/var/lib/plash  -ti nikplash bash 
  root@4ab0e1e31c01:/# echo $PLASH_NO_UNSHARE
  1
  root@4ab0e1e31c01:/# plash init
  root@4ab0e1e31c01:/# echo hi | plash run --from ubuntu --apt figlet -- figlet
  mount: permission denied
  plash error: Command '['mount', '-t', 'overlay', 'overlay', '-o', 'lowerdir=/var/lib/plash/index/1/_
  data/root:/var/lib/plash/index/0/_data/root,workdir=/var/lib/plash/tmp/plashtmp_1_17_oyctcmfm/work,u
  pperdir=/var/lib/plash/tmp/plashtmp_1_17_oyctcmfm/data', '/var/tmp/plash-mountpoint-19-frbgwcmi']' r
  eturned non-zero exit status 32.
  plash error: build failed with exit status 1

  Am I doing something wrong here?

• While fiddling I noticed that while using a named volume as you do in the plash script in docker-plash works fine (as does using --tmpfs as you mention above) but using a shared volume on macOS (--volume $PWD/plashdata:/var/lib/plash) does not and results in an error message about too many levels of symbolic links. It did work as expected on a native linux host though, so I suspect this is just some side effect of all the magic docker for mac does behind the scenes to share folders with the underlying VM.

• But I read that you should'nt use a driectory via dockers --volume with multiple instances at the same time because of possible data loss or corruption

  To my (admittedly limited) knowledge multiple containers sharing the same docker volume is no different than two processes accessing the same folder. Found a stack overflow post on this that purports to quote docker's docs, but the quoted text never actually appears in the linked doc from docker. However even the quoted text ("multiple containers writing to a single shared volume can cause data corruption. Make sure your applications are designed to write to shared data stores.") seems to be suggesting that application level concurrency concerns are the issue and not any extra complexity added by docker's volume management mechanisms.

  Perhaps a good middle ground would be to release this as alpha/beta with a warning about possible data loss when running multiple plash containers in docker concurrently?

[ihucos]

The irae/docker-plash image seems to be out of date, I wasn't able to get anything to work when I pulled that image (perhaps there were changes to the plash CLI since the image was built?), but everything works fine if I do a docker build with the included Dockerfile.

There was an old version before backward compatibility was introduced, I rebuilded the image just now. In newer versions plash version will print the version being used.

[ihucos]

Also note that since version 1.1392 (released three days ago) plash will automatically try to fallback to unionfs-fuse if a test invocation of overlay results with "operation not permited". So - as you find out - that it does not work, the error message might have changed now.

[ihucos]

• On my primary setup (ubuntu bionic) using only --cap-add SYS_ADMIN does not work:

  $ docker run  --cap-add SYS_ADMIN  --tmpfs /tmp --volume plashdata:/var/lib/plash  -ti irae/docker-plash bash 
  root@a83c99f5ef6e:/# echo hi | plash run --from ubuntu --apt figlet -- figlet
  plash error: calling mounting returned EACCES 
  plash error: build failed with exit status 1

• With --cap-add SYS_ADMIN and PLASH_NO_UNSHARE=1, it fails on the overlay mount

  $ docker run -e PLASH_NO_UNSHARE=1  --cap-add SYS_ADMIN  --tmpfs /tmp --volume plashdata:/var/lib/plash  -ti irae/docker-plash bash
  root@2e09f368ad4f:/# echo hi | plash run --from ubuntu --apt figlet -- figlet
  mount: overlay is write-protected, mounting read-only
  mount: cannot mount overlay read-only
  plash error: Command '['mount', '-t', 'overlay', 'overlay', '-o', 'lowerdir=/var/lib/plash/index/2/_data/root:/var/lib/plash/index/0/_data/root,workdir=/var/lib/plash/tmp/plashtmp_1_15_s8u1w4ub/work,upperdir=/var/lib/plash/tmp/plashtmp_1_15_s8u1w4ub/data', '/var/tmp/plash-mountpoint-17-m_vyfook']' returned non-zero exit status 32.
  plash error: build failed with exit status 1

• Adding yet another flag named --security-opt=apparmor:unconfined, it works

  
  $ docker run --security-opt=apparmor:unconfined -e PLASH_NO_UNSHARE=1 --cap-add SYS_ADMIN --tmpfs /tmp --volume plashdata:/var/lib/plash -ti irae/docker-plash bash
  root@f4549616556a:/# echo finally | plash run --from ubuntu --apt figlet -- figlet
  __ _             _ _       
  / _(_)_ __   __ _| | |_   _ 
  | |_| | '_ \ / _` | | | | | |
  |  _| | | | | (_| | | | |_| |
  |_| |_|_| |_|\__,_|_|_|\__, |
                     |___/ 

So as a conclusion:

Just doing `PLASH_NO_UNSHARE=1` did not work out as expected because overlay failed. In my memory overlay mounts did work without the `--privileged` flag in docker containers, but I was wrong.
Running containers on docker without the `--privileged` flag does only work, adding these three additional flags, which are also security sensitive. Right now I would actually just prefer to say that a privileged container is an requirement out of simplicity and forward comparability needs. And of course document the use of `--privileged` in the README of ihucos/docker-plash . In my understanding that requirement does not invalidate the claim in the README of ihucos/plash that plash runs inside docker. It would be interesting to know if in the use cases of running plash inside docker, the requirement of a `--privileged` container is a significant trade off. I'll also document the need of the `--privileged` flag at the Caveats section in the README.

> Perhaps a good middle ground would be to release this as alpha/beta with a warning about possible data loss when running multiple plash containers in docker concurrently?

Yeah, thank you for your findings. So I'll document the past existence of a warning of possible data loss in the docker homepage. Also the usage of privileged docker containers and do a little bit of cleanup and testing and it should be a beta :-) It would also be nice if the `.travis` script can trigger rebuilds so the version does not get outdated. The good news is that when it's ready you could use something like ihucos/noapt or ihucos/megaman to run linux packages in your mac, which would be pretty cool. Some experimenting with XQuartz could also be interesting.

[ihucos]

Ok, so if you want to use plash inside docker go with:

https://github.com/ihucos/docker-plash (it should be good to go now)

Or alternatively follow the example in the README

Plash only runs in docker containers started with the --privileged flag

@nikvdp Does this satisfy your need or use case of running plash inside docker?

[ihucos]

closing, read last post of how to run plash inside docker

[nikvdp]

Absolutely, thanks for the prompt responses and help getting docker-plash up and running!

[ihucos]

Super, If you find rough edges let me know!