Permissions Denied (non root)

[aneisch]

Describe the bug Since 4.8.0 with s6, the container fails to start when user is specified in docker run or docker-compose.

Debug Logs

s6-overlay-preinit: fatal: unable to mkdir /var/run/s6: Permission denied
s6-overlay-preinit: fatal: unable to mkdir /var/run/s6: Permission denied
s6-overlay-preinit: fatal: unable to mkdir /var/run/s6: Permission denied
s6-overlay-preinit: fatal: unable to mkdir /var/run/s6: Permission denied

    ring-mqtt:
        container_name: ring-mqtt
        image: tsightler/ring-mqtt
        restart: 'always'
        user: '99'
        environment:
            - MQTTHOST=10.0.1.22
            - ENABLEMODES=true
            - SNAPSHOTMODE=all
            - ENABLECAMERAS=true
        volumes:
            - '/opt/ring-mqtt:/data'

[aneisch]

The obvious answer is "run the container as root" but I'm looking into whether or not their is an easy workaround for this.

[tsightler]

The Docker container is primarily provided for the addon, which runs as root, so that's the only scenario that I test. Because of this, it's likely that attempting to run as non-root will sometimes break as it's not a tested configuration. This is especially true when there are significant changes to the structure of the container image like the move to using s6. In general, the s6 supervision tree expects to run as root, but I could potentially drop privileges for the other processes so that at least the core processes would not be running as root, even when the container is started as root. However, this would still mean that the container would expect to be started as root.

I do see that s6 added some minimal support for running as the non-root docker user fairly recently (within the last year or so), but I'd need to research and understand what that support entails. It's certainly not high on my priority list to do either of these things, but I will keep them in mind. If it's super easy to support, I will try to do it in a future update, but I'm likely taking a break from this project for a while now that 4.8.0 is out there, so it will probably not be right around the corner.

In the interim, the entire project is fully available on Github, including the Dockerfile, so you can easily modify them and build a container to run however you wish, no requirements to use the pre-built container.

[aneisch]

I'm likely taking a break from this project for a while now that 4.8.0 is out there

Right on! Well deserved!

I certainly don't expect this to be anything high priority, I just wanted to see what the general thoughts and technical undertakings would be to support this. I will continue to do some research and testing of my own and submit a PR if I manage to make any reasonable progress and feel like I've come up with an acceptable solution if you're amiable to it.

I think a drop privileges solution would probably be the best way to accomplish this, but again, more research needed.

Thanks for your hard work on this awesome solution, enjoy the break!

[tsightler]

Dropping privileges in S6 is pretty easy, probably just add a line to the init files, but I didn't do it because, well, honestly, I had never used S6 before and I just used examples from other addons and they all (as far as I can tell) just run as root for everything. I just checked all of my addons in my production instance and all processes are root in all of them.

I mainly implemented S6 because that's what all the other addons use and, with the addition of streaming, I now have secondary processes and shell scripts that run on demand and there were potential cases of zombie processes since node only cleans up processes it spawns directly, so I needed a basic init system.

The unfortunate side effect of that is that s6 makes a lot of assumption about running as root. It actually had crossed my mind but, in the normal case, there aren't any exposed ports outside of the container anyway, so it's difficult to see the reason for running non-root. However, I could see it if a user decides to expose, for example, the rtsp-simple-server, but honestly, if that attack is coming internally, well, there are already other big problems in the environment.

[tsightler]

Thinking about this, the biggest issue with dropping permissions within the container is that I'd still have to pick just some random UID/GID to run as and this user would need permissions to write to the persistent volume, but the user running the container would not even know what that UID/GID is because it wouldn't exist on the host. I feel like that's a borderline support nightmare for the people who use this project that barely understand docker well enough to get it running now. I could always tell them to just make the volume with full access to all users I guess, but that feels somehow worse since it has the token.

So I guess that goes back to figuring out how S6 deals with this case and if it's possible to make that work.

[tsightler]

@aneisch Out of curiosity, can you try adding S6_READ_ONLY_ROOT=1 to the environment section of your docker compose. For me this seems to work. It generates some errors on startup because it still tries to chown files to root, but the files simply stay owned by the specified user and everything still seems to work. Perhaps this is the easy workaround you were looking for.

That being said, I'm actually not getting the same error you are seeing so something must be different between our environments.

[aneisch]

I set the variable and everything seems to be running correctly but doesn't seem (or is intended) to have any affect on the user who owns processes. When I specify user=99 and S6_READ_ONLY_ROOT=1 in my compose file I get the same error.

[tsightler]

For me it runs, so I need more details on your setup. Are you perhaps running Docker itself rootless or is there anything otherwise unique about your Docker setup? I basically just created a compose file using what you posted and, for me, adding S6_READ_ONLY_ROOT=1 works fine with user=99, but I don't ever get the message about mkdir failing.

[tsightler]

I think for the 4.8.1 version I'm going to try setting /var/run to 777 and see if that addresses the problem for you. That directory is empty otherwise and I don't see any negative issues with setting it to wide open since it's in the container anyway and, when using the "user" option all of the process will be running as that user so it should be OK.

[tsightler]

@aneisch I did a little bit more research on how s6 init deals with the case when Docker runs with specified user and it's really super simple. There is a small binary, s6-overlay-preinit, which has the SUID sticky bit set so that it always runs as root. This is the very first command run by s6-init, before anything else, and the only thing this binary does is run mkdir to create the /var/run/s6 folder with the proper permissions since, without this, it's not possible to create directories in /var/run as the default permissions on this folder are rwxr-xr-x and it's owned by root.root.

Since s6-overlay-preinit is suid root, it can create the s6 directory and it does so with the run user so that the /var/run/s6 folder is owned by the run user and that user has full rights. This works perfectly in my case, if I start the container as follows I get no errors:

docker run -it --entrypoint /bin/s6-overlay-preinit --user 99 tsightler/ring-mqtt

So clearly the SUID process is working properly as if I just try to create a directory with mkdir, it fails with the same message you receive:

$ docker run -it --entrypoint /bin/mkdir --user 99 tsightler/ring-mqtt /var/run/s6
mkdir: can't create directory '/var/run/s6': Permission denied

However, clearly the SUID process is not working in your case and I can only assume that this is because of some unique security setup that is blocking SUID in the container so I'd be interested in more details around your specific Docker setup. I'm still hopeful that setting /run to 777 will be enough to address this though, and I see no real downside to doing that.

[aneisch]

some unique security setup that is blocking SUID

Your theory is correct, I do have the following in my daemon config:

    "no-new-privileges": true

[tsightler]

Got it, so it makes total sense now why the behavior in my environment is different than what you see and also why it doesn't work for you, because setting no-new-privileges breaks the limited support that S6 has for the Docker "user" feature since S6 can't elevate to create the required directory in /var/run. I'm pretty sure just setting /var/run to 777 will be enough, but now I can test it.

[tsightler]

Pushed 4.8.1 tonight with /var/run set to 777. In my setup this seems to allow it to work with "no-new-priviledges": true as long as you also set S6_READ_ONLY_ROOT=1 in the environment. Admittedly, it was just a smoke test, but it at least made it to the phase that ring-mqtt started. Could you give it a spin when you get a chance?

[aneisch]

Pulled down 4.8.1 and everything looks good!

[aneisch@nuc-nuc docker-compose]$ docker top ring-mqtt
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
nobody              11228               11207               0                   23:10               ?                   00:00:00            s6-svscan -t0 /var/run/s6/services
nobody              11306               11228               0                   23:10               ?                   00:00:00            s6-supervise s6-fdholderd
nobody              11489               11228               0                   23:10               ?                   00:00:00            s6-supervise ring-mqtt
nobody              11492               11489               13                  23:10               ?                   00:00:02            node /app/ring-mqtt/ring-mqtt.js
nobody              11557               11492               0                   23:10               ?                   00:00:00            rtsp-simple-server /app/ring-mqtt/lib/../config/rtsp-simple-server.yml

Processes are running as nobody as expected and all functionality seems to be unaffected, including streaming.

[tsightler]

Awesome, thanks a lot for the feedback!

[aneisch]

Thanks a lot for this amazing project!