Make the CAmkES Docker images work under `podman`

[maybe-sybr]

This changeset makes it possible to run the Makefile when on a system where docker is actually podman. It ensures that host mounts are given sensible container labels for SELinux with the :z option, and avoids using the in-container dev user in favour of believing that root is actually isolated by the container runtime.

[maybe-sybr]

Accepting that we'll be running as UID 0 in the container under podman isn't great and I did spend some time attempting to force --userns=keep-id to do what it claims to, but it looks like there are a /lot/ of subtle issues which make that hard for a mere human to do. For reference, here's what I tried (in case someone else would like to pick that work up):

• --userns=keep-id results in not being able to run anything in the container (so it dies immediately with EACCES)
• adding --cap-add=DAC_OVERRIDE lets us run programs but leaves the following symptoms:
  • the TERM variable is not understood, so readline features like backspace don't work!
  • cmake can't find its modules and errors out when you run it
  • sudo can't stat the sudoers file so it dies when you run it
• adding --cap-add=ALL doesn't appear to help
• adding --privileged doesn't appear to help

I suspect the fact that the root directory is mounted like:

maybe@1116e66e2363:/$ id
uid=1001(maybe) gid=31337 groups=31337
maybe@1116e66e2363:/$ mount 
fuse-overlayfs on / type fuse.fuse-overlayfs (rw,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)
...

may be problematic, specifically the user_id=0,group_id=0 options. But I'm not incredibly confident making that statement and no idea how to change it since the user friendly options seems to have failed me at this point.

I haven't brought this up on the podman mailing list but if I do, will link the thread from their archive here. In any case, hopefully the approach I ended up taking isn't inelegant enough to block merging this :)

[LukeMondy]

Hello, and thanks for the PR!

I need to take some more time to look at this, but you inspired me to release our own tool we have that uses Podman: https://github.com/SEL4PROJ/imagesh

The short summary of the tool is that it is a shell that, when users ssh in, dumps them into a rootless podman container of their choice.

It does skip the Makefile, and the extras.dockerfile (as it replaces it with its own) - but in regard to mapping in user's UID/GID into the rootless container, we end up doing this:

https://github.com/SEL4PROJ/imagesh/blob/b3815d85d9ad729c59ec0d7abeae8297c399a0d3/bin/add_user.sh#L14

which gets coupled with this: https://github.com/SEL4PROJ/imagesh/blob/b3815d85d9ad729c59ec0d7abeae8297c399a0d3/bin/imagesh#L156

Again, I need to look at your PR more carefully and do some testing, but I thought I'd put up ImageSh quickly to see if it helps.

Thanks!

[maybe-sybr]

Thanks for pushing ImageSh up, @LukeMondy - it looks like the bits you linked are exactly what I was fumbling to avoid learning how to do properly. I'll quickly parrot that into my setup to see if it behaves any better than my previous efforts. If it does, perhaps we can think about cannibalizing some of that logic into this repo!

[maybe-sybr]

Looks like ImageSh has the exact same misbehaviour on my box ): I did enough surgery to get it to run in its checked out location, and see the following output:

$ bin/imagesh
Logging you into: trustworthysystems/camkes-riscv:latest
Checking /../sel4/imagesh/extra.dockerfile for any image customisations... done
Welcome to the TS build server!

This server uses the latest Docker images to give a build environment
the same as what Bamboo uses!

You can customise it by editing the files in /../sel4/imagesh/

Enjoy!
{"msg":"exec the container process: Permission denied","level":"error","time":"2020-02-07T04:06:39.000236497Z"}

Adding --cap-add=DAC_OVERRIDE once again lets me get into the container but the same things are broken as in my previous comment wrt trying to use --userns=keep-id.

A couple of potentially interesting note on my system:

• LUKS on all volumes
• my primary GID != UID, GROUP != USER

[LukeMondy]

Cool, thanks for trying! I'm in the process of setting up a fresh Fedora VM to give this a test, though I might not get to fully investigating until next week.

[LukeMondy]

Hey, so I just tried this on a fresh Fedora 29 VM with podman (symlinked as docker).

The changes I made resulted in:

user_run:
    docker run \
        $(DOCKER_RUN_FLAGS) \
        --hostname in-container \
        --rm \
        --userns=keep-id \
        -v $(HOST_DIR):/host:z \
        -v $(DOCKER_VOLUME_HOME):/home/$(shell whoami) \
        -v /etc/localtime:/etc/localtime:ro \
        $(USER_IMG) $(EXEC)

Note that I followed your lead, and took out the -u $(shell id -u):$(shell id -g) stuff, and put in --userns=keep-id, and added the :z. Those were the only changes I made.

And it worked great! I was able to repo init, repo sync, compile sel4test, and run the resulting image in a simulator. All the permissions worked out fine, both in and outside the container.

It sounds like this didn't work for you though? Can you elaborate more? One issue I did encounter was having the docker volume lmondy-home hanging around from before I changed things, so it had the wrong uids. Doing docker volume rm lmondy-home fixed it up.

[maybe-sybr]

Interesting. I've also not been able to replicate the breakage on another pretty similarly configured machine of mine where I've been attempting to use podman for some other containers. From all the documentation I've been able to track down, it does seem like --userns=keep-id should work fine, but it continues to explode on my laptop. I cleaned up my old podman containers, volumes and images; and still see the same breakage which I noted in the first comment I dropped here. Most problematically, even after getting into the container by adding CAP_DAC_OVERRIDE, cmake can't find its modules which causes immediate problems attempting to build sel4test.

It might be worth dropping cc52f40 from this PR and tacking on a new change to use --userns=keep-id if we do turn out to be on top of podman, and I'll continue to dig into whatever weirdness is happening on my box. I've replicated similar container misbehaviour using alpine so it's doesn't seem to be a problem with the seL4 images.

[maybe-sybr]

In the event that it's useful, I also have a diff which ensures the host UID and GID both get set up rather than auto-assigning a new GID with the UNAME. That was to see if my UID!=GID was causing issues, but it didn't help.

diff --git a/Makefile b/Makefile
index 60c02ed..1795372 100644
--- a/Makefile
+++ b/Makefile
@@ -172,6 +172,8 @@ build_user: run_checks
        --build-arg=EXTRAS_IMG=$(EXTRAS_IMG) \
        --build-arg=UNAME=$(shell whoami) \
        --build-arg=UID=$(shell id -u) \
+       --build-arg=GID=$(shell id -g) \
+       --build-arg=GROUP=$(shell id -gn) \
        -f dockerfiles/user.dockerfile \
        -t $(USER_IMG) .
 build_user_sel4: USER_BASE_IMG = $(SEL4_IMG)
diff --git a/dockerfiles/user.dockerfile b/dockerfiles/user.dockerfile
index 3bf74b5..e64fb22 100644
--- a/dockerfiles/user.dockerfile
+++ b/dockerfiles/user.dockerfile
@@ -4,9 +4,12 @@ FROM $EXTRAS_IMG
 # Get user UID and username
 ARG UID
 ARG UNAME
+ARG GID
+ARG GROUP

 # Crammed a lot in here to make building the image faster
-RUN useradd -u ${UID} ${UNAME} \
+RUN groupadd -g ${GID} ${GROUP} \
+    && useradd -u ${UID} -g ${GID} ${UNAME} \
     && adduser ${UNAME} sudo \
     && passwd -d ${UNAME} \
     && echo 'Defaults        lecture_file = /etc/sudoers.lecture' >> /etc/sudoers \
@@ -33,9 +36,9 @@ RUN useradd -u ${UID} ${UNAME} \
     && echo 'export PATH=/scripts/repo:$PATH' >> /home/${UNAME}/.bashrc \
     && echo 'cd /host' >> /home/${UNAME}/.bashrc \
     && mkdir -p /isabelle \
-    && chown -R ${UNAME}:${UNAME} /isabelle \
+    && chown -R ${UNAME}:${GROUP} /isabelle \
     && ln -s /isabelle /home/${UNAME}/.isabelle \
-    && chown -R ${UNAME}:${UNAME} /home/${UNAME} \
+    && chown -R ${UNAME}:${GROUP} /home/${UNAME} \
     && chmod -R ug+rw /home/${UNAME} 

 # Copy the dev user's `.bashrc` into `root`s home directory too for rootless

[LukeMondy]

Cool, thanks! I'll do some more testing, and see how it goes. The issues you are facing do indeed sound weird, but as you say, sound isolated to your laptop :(

Do you want to make the edits to this PR, so you get authorship on the commits, or would you prefer I modify the commits as needed and merge them?

[maybe-sybr]

Feel free to make whatever edits you need to, @LukeMondy - I'm not too worried about the blame ;)

[maybe-sybr]

I had a bit of time this morning to push an amended changeset, @LukeMondy. The new HEAD commit should be equivalent to what you were running and handle both real docker and podmans hijacking alias. I also added a preceding commit with the diff I dropped above to keep the host/container image group lined up. With the little hacks I still have to apply, I've got the same behaviour so from my perspective, the changeset should be good for you to validate. LMK if there's anything else which needs changing prior to merge.

[LukeMondy]

Cool, looks great! I've merged it internally, and it should get pushed out within a day or so!

[LukeMondy]

A small issue I have found is that my docker --version does not say podman, even though I'm using podman:

[lmondy@defora ok]$ ls -lah $(which docker)
lrwxrwxrwx. 1 lmondy lmondy 15 Feb 10 12:56 /home/lmondy/bin/docker -> /usr/bin/podman
[lmondy@defora ok]$ docker -v
docker version 1.6.2

However, docker --help does mention podman explicitly a few times.

I will add another commit to switch to this one.

[maybe-sybr]

A small issue I have found is that my docker --version does not say podman, even though I'm using podman:

Hmm, interesting. I'm running 1.8.0 and I think I was on 1.7.0 until recently. Might be a pre 1.7 thing and they stopped masquerading as docker. Here's my --version and --help output for reference. Looks like using --help will continue to work fine for me.

$ docker --version
podman version 1.8.0
$ docker --help | grep podman
  podman [flags]
  podman [command]
  events      Show podman events
  info        Display podman system information
  system      Manage podman
      --help                      Help for podman
  -v, --version                   Version of podman
Use "podman [command] --help" for more information about a command.

[LukeMondy]

The changes have been pushed out, after I rebased them.

Thanks for contribution!

[maybe-sybr]

Champion, thanks @LukeMondy!