Privileged docker images write files with inappropriate UIDs and GIDs

[rpgoldman]

If one mounts a host machine directory (e.g., server/db_data) into a docker container running privileged in a rootless docker, then one can get files created that the owning user on the host machine does not own, and cannot delete, causing failures to start:

error checking context: no permission to read from '/home/rpg/src/planning-as-a-service/server/db_data/#ib_16384_0.dblwr'. 

That file's UID and GID are both 328532966 on the host machine, so I have to bug a sysadmin to delete them before I can restart, or do a really complicated hokey-pokey where I docker exec bash in the MySQL DB container and do the job there....

Is there some way I could set the MYSQL_USER in env to avoid this?

[haz]

As an interim approach, I wonder if we could have a make clean script as part of the Makefile do the hokey pokey for us?

[rpgoldman]

Possibly helpful article: https://vsupalov.com/docker-shared-permissions/`

[rpgoldman]

As an interim approach, I wonder if we could have a make clean script as part of the Makefile do the hokey pokey for us?

Yes, I think one could start up the mysqld image, running bash instead of whatever it's entrypoint is, and smash the files in question.

Note that this shows why privileged docker containers are a horrible security threat! But I have no idea why these containers are run as privileged. Maybe they don't need to be, in which case the techniques in the above article would address the issue.

[haz]

I think, technically, only the worker needs to be run as privileged. It stems from the way singularity runs inside of docker, and most of the planutils packages are singularity images. But maybe this whole thing is moot if the MySQL container doesn't need to run as privileged.

We found it convenient for all to be built on top of the same planutils-parent image, but creating the container doesn't need that escalated mode. Just running the planners (so only the workers).

[rpgoldman]

As an interim approach, I wonder if we could have a make clean script as part of the Makefile do the hokey pokey for us?

OK, with some help from StackOverflow this seems to be the recipe:

1. Start only the mysql server container with docker compose up mysql
2. Stop the mysql daemon either by getting a bash shell and doing mysqld stop or docker exec server-mysql-1 /usr/sbin/mysqld stop
3. Get into the container and do rm -rf /var/lib/mysql/*. Again, this can be done "from outside" by doing docker exec.
4. Probably (since we are in make clean) now shutdown the system with docker compose down

[rpgoldman]

There are documents in the docker hub page about the mysql image that suggest it can be run in userspace, and one might want to ensure that the user-id in the container and the user invoking make are the same, so that ownership of files is correct.

[haz]

For reference, https://medium.com/redbubble/running-a-docker-container-as-a-non-root-user-7d2e00f8ee15

It took longer than I would like to admit, but I've finally managed to make it work over [here]. I'll open another (small) PR for it, once the current one goes through.

Caveat: I wasn't able to figure out how to nuke the folder from within the container...that virtual mount is a powerful thing :P. Plus, the MySQL image is extremely stripped down. Not much available from the inside.

[rpgoldman]

Caveat: I wasn't able to figure out how to nuke the folder from within the container...that virtual mount is a powerful thing :P. Plus, the MySQL image is extremely stripped down. Not much available from the inside.

I should have emphasized that I was able to delete the contents of that folder -- rm -rf /var/lib/mysql/* -- but not the folder itself -- rm -rf /var/lib/mysqld will not work. Does that help?

[haz]

Not if we need the folder removed :P. What got me hung up for so long was that it’s not the permissions in the container that creates the db_data folder ownership, but rather the user space of the docker (or docker compose), which is typically root. Key is to create the directory first (user space), and then fire things up.

You’ll need to find a way to wipe what’s there, but the new PR that’s opened ( #68 ) should spin things up (with make) without causing any funny permissions.