search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
23.45k stars 2.38k forks source link

Clarification, rootless-as-non-root UID/GID remapping #10669

Closed andrewgdunn closed 3 years ago

andrewgdunn commented 3 years ago

Was going to post to the mailing list, but this may be more discoverable... and it may result in a feature/enhancement documentation change.

The shortest form of my question/confusion is:

When a container has a non-root (e.g uid isn't 0) as the container user is there a way to map that back to the initiating host user uid?

The follow up to this would be, how does one handle many non-root users (e.g. gitlab omnibus container has many non-root users writing to disk)?

For example:

[root@vault ~]# useradd -g 606 -u 606 -d /zfs/app/freshrss/ -s /bin/bash freshrss
[root@vault ~]# usermod --add-subuids 60600000-60665535 --add-subgids 60600000-60665535 freshrss
[root@vault ~]#  su - freshrss
[freshrss@vault ~]$ podman info
...snip
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 606
      size: 1
    - container_id: 1
      host_id: 60600000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 606
      size: 1
    - container_id: 1
      host_id: 60600000
      size: 65536
...snip

Now, the freshrss name is because there a good example of a simple container that utilizes the non-root user within itself, there is documentation on their standards of uid and gid here.

We know that if we run this container there is a default uid and gid of 910:

[freshrss@vault ~]$ podman run --detach " --name=freshrss --publish 127.0.0.1:8008:80 --volume ~/data/:/config:Z docker.io/linuxserver/freshrss:latest
376beede3e880b7d8489a0d04ee6f983a3bee13888d1ee3ff1f0a494d8032662
[freshrss@vault ~]$ ls -la data/
total 6
drwxr-xr-x. 8 60600910 60600910 8 Jun 13 09:58 .
drwxr-xr-x. 6 freshrss freshrss 7 Jun 13 09:48 ..
drwxr-xr-x. 2 60600910 60600910 3 Jun 13 09:58 crontabs
drwxr-xr-x. 2 60600910 60600910 4 Jun 13 09:58 keys
drwxr-xr-x. 4 60600910 60600910 4 Jun 13 09:58 log
drwxrwxr-x. 3 60600910 60600910 4 Jun 13 09:58 nginx
drwxr-xr-x. 2 60600910 60600910 4 Jun 13 09:58 php
drwxrwxr-x. 3 60600910 60600910 4 Jun 13 09:58 www

I'd love to know how to "clobber" these filesystem operations back to the initiating host user. What I'd like to see is that these things created on disk are uid and gid of 606. Looking over the documentation I've been unable to figure this out, however I've tried a couple things:

This phenomena gets more complex with "bigger" containers, for example the gitlab omnibus has many users writing to the filesystem so you get a smattering of mapped uid and gid.

giuseppe commented 3 years ago

tried --userns=keep-id but then the container user (910 in the container) can't write to the volume

what error do you have? If the volume is owned by your rootless main user, then it should be able to access it.

When you say "consistent mapping", do you mean the identify mapping?

mheon commented 3 years ago

This is definitely the use-case for --userns=keep-id. Privileged does not and will never do this by default - the default container root to user launching container mapping is a lot safer as container apps run as root (and the the user launching the container) by default. You can certainly duplicate what --userns=keep-id does through --uidmap and --gidmap, but I think you're only duplicating what the flag does as under the hood that is basically what it does. --userns=host doesn't do anything on rootless Podman because we absolutely require a user namespace to set up rootless containers (really need to document this better, maybe emit a warning on it happening).

For debugging your EPERM, I might suggest reading https://www.redhat.com/sysadmin/debug-rootless-podman-mounted-volumes which I put together for cases like this - this is a common problem. Generally speaking, though, it root in the container was mapped to your user and successfully touched a file, and you change to --userns=keep-id to change to an identity mapping, the user in question should also be able to successfully write to that volume.

andrewgdunn commented 3 years ago

Thanks for the response @giuseppe:

what error do you have? If the volume is owned by your rootless main user, then it should be able to access it.

andrewgdunn commented 3 years ago

Thanks for the response @mheon, I did read through that and felt like I was entirely confused about --userns=keep-id (as it seems like exactly what I'm looking for). I didn't debug very long with it. As I'd mentioned just above to @giuseppe, the user that is running the process in this container is not the root user, so it doesn't seem to have proper permissions to write to the mapped volume.

I'd seen this (use of --userns=keep-id) prevent me from launching gitlab-{cc,ee} as well as the container I'd mentioned in the write up. These containers are all built with non-root users interacting on disk.

andrewgdunn commented 3 years ago

@mheon follow up on:

Generally speaking, though, it root in the container was mapped to your user and successfully touched a file, and you change to --userns=keep-id to change to an identity mapping, the user in question should also be able to successfully write to that volume.

I'd tried to toy with this, I was unable to get it to work and I didn't share my logs. I'm away from the machine for a moment but wanted to respond with a question (as I might have had the wrong idea for testing).

Ideally, in the case of freshrss above, I can either just know that the uid and gid are 910 or I an specify with an envar (the way the container is built). In the case of gitlab it's a couple users that I'd have to go discover. I'd wanted to use --uidmap to map specific users back to my host user, such as:

--uidmap=0:606:1 --uidmap=910:606:1 where I want both the container users with id of 0 and 910 to be able to write as the 606 user on the host.

mheon commented 3 years ago

I defer to @giuseppe on whether it's possible to have multiple GIDs in the container map to one GID on the host

andrewgdunn commented 3 years ago

Here is the example I was working with.

[root@vault ~]# useradd -g 606 -u 606 -d /zfs/app/freshrss/ -s /bin/bash freshrss
[root@vault ~]# usermod --add-subuids 60600000-60665535 --add-subgids 60600000-60665535 freshrss
[root@vault ~]# loginctl enable-linger freshrss
[root@vault ~]# su - freshrss
Last login: Mon Jun 14 20:18:46 EDT 2021 on pts/0
[freshrss@vault ~]$ dnf info podman
Installed Packages
Name         : podman
Epoch        : 3
Version      : 3.2.0
Release      : 5.fc34
Architecture : x86_64
Size         : 46 M
Source       : podman-3.2.0-5.fc34.src.rpm
Repository   : @System
From repo    : updates
Summary      : Manage Pods, Containers and Container Images
URL          : https://podman.io/
License      : ASL 2.0
Description  : podman (Pod Manager) is a fully featured container engine that is a simple
             : daemonless tool.  podman provides a Docker-CLI comparable command line that
             : eases the transition from other container engines and allows the management of
             : pods, containers and images.  Simply put: alias docker=podman.
             : Most podman commands can be run as a regular user, without requiring
             : additional privileges.
             : 
             : podman uses Buildah(1) internally to create container images.
             : Both tools share image (not container) storage, hence each can use or
             : manipulate images (but not containers) created by the other.
             : 
             : Manage Pods, Containers and Container Images
             : podman Simple management tool for pods, containers and images
[freshrss@vault ~]$ mkdir data
[freshrss@vault ~]$ ls -la
total 5
drwxr-xr-x. 3 freshrss freshrss     4 Jun 14 20:19 .
drwxr-xr-x. 8 root     root       104 Jun 12 15:23 ..
-rw-------. 1 freshrss freshrss 16689 Jun 14 20:18 .bash_history
drwxr-xr-x. 2 freshrss freshrss     2 Jun 14 20:19 data
[freshrss@vault ~]$ id
uid=606(freshrss) gid=606(freshrss) groups=606(freshrss) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

I'd like to use the --userns=keep-id and have this container be able to write to the data volume, but... it seems to have permissions errors:

[freshrss@vault ~]$ podman run --userns=keep-id --name=freshrss --publish 127.0.0.1:8008:80 --volume ~/data/:/config:Z docker.io/linuxserver/freshrss:latest
Trying to pull docker.io/linuxserver/freshrss:latest...
Getting image source signatures
Copying blob e9d333e5322a done  
Copying blob cbf2c38fca87 done  
Copying blob 664897dd7f3c done  
Copying blob bcd6fb444d70 done  
Copying blob 9a950d26b1f9 done  
Copying blob ba742d4eac8f done  
Copying blob 70ef020bbadd done  
Copying blob 14dd34cd286f done  
Copying blob a538fa58b2b7 done  
Copying blob e482928adb1b done  
Copying config 8f057a25eb done  
Writing manifest to image destination
Storing signatures
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...s6-chown: fatal: unable to chown /var/run/s6/etc/cont-init.d/20-config: Operation not permitted
s6-chown: fatal: unable to chown /var/run/s6/etc/cont-init.d/01-envfile: Operation not permitted
s6-chown: fatal: unable to chown /var/run/s6/etc/cont-init.d/40-install: Operation not permitted
s6-chown: fatal: unable to chown /var/run/s6/etc/cont-init.d/99-custom-files: Operation not permitted
s6-chown: fatal: unable to chown /var/run/s6/etc/cont-init.d/10-adduser: Operation not permitted
s6-chown: fatal: unable to chown /var/run/s6/etc/cont-init.d/30-keygen: Operation not permitted
s6-chmod: fatal: unable to change mode of /var/run/s6/etc/cont-init.d/99-custom-files: Operation not permitted
s6-chmod: fatal: unable to change mode of /var/run/s6/etc/cont-init.d/20-config: Operation not permitted
s6-chmod: fatal: unable to change mode of /var/run/s6/etc/cont-init.d/01-envfile: Operation not permitted
s6-chmod: fatal: unable to change mode of /var/run/s6/etc/cont-init.d/40-install: Operation not permitted
s6-chmod: fatal: unable to change mode of /var/run/s6/etc/cont-init.d/10-adduser: Operation not permitted
s6-chmod: fatal: unable to change mode of /var/run/s6/etc/cont-init.d/30-keygen: Operation not permitted
s6-chown: fatal: unable to chown /var/run/s6/etc/services.d/nginx/run: Operation not permitted
s6-chown: fatal: unable to chown /var/run/s6/etc/services.d/cron/run: Operation not permitted
s6-chown: fatal: unable to chown /var/run/s6/etc/services.d/php-fpm/run: Operation not permitted
s6-chmod: fatal: unable to change mode of /var/run/s6/etc/services.d/nginx/run: Operation not permitted
s6-chmod: fatal: unable to change mode of /var/run/s6/etc/services.d/cron/run: Operation not permitted
s6-chmod: fatal: unable to change mode of /var/run/s6/etc/services.d/php-fpm/run: Operation not permitted
exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 01-envfile: executing... 
foreground: warning: unable to spawn /var/run/s6/etc/cont-init.d/01-envfile: Permission denied
[cont-init.d] 01-envfile: exited 127.
[cont-init.d] 10-adduser: executing... 
foreground: warning: unable to spawn /var/run/s6/etc/cont-init.d/10-adduser: Permission denied
[cont-init.d] 10-adduser: exited 127.
[cont-init.d] 20-config: executing... 
foreground: warning: unable to spawn /var/run/s6/etc/cont-init.d/20-config: Permission denied
[cont-init.d] 20-config: exited 127.
[cont-init.d] 30-keygen: executing... 
foreground: warning: unable to spawn /var/run/s6/etc/cont-init.d/30-keygen: Permission denied
[cont-init.d] 30-keygen: exited 127.
[cont-init.d] 40-install: executing... 
foreground: warning: unable to spawn /var/run/s6/etc/cont-init.d/40-install: Permission denied
[cont-init.d] 40-install: exited 127.
[cont-init.d] 99-custom-files: executing... 
foreground: warning: unable to spawn /var/run/s6/etc/cont-init.d/99-custom-files: Permission denied
[cont-init.d] 99-custom-files: exited 127.
[cont-init.d] done.
[services.d] starting services
s6-supervise (child): fatal: unable to exec run: Permission denied
s6-supervise (child): fatal: unable to exec run: Permission denied
s6-supervise cron: warning: unable to spawn ./run - waiting 10 seconds
s6-supervise nginx: warning: unable to spawn ./run - waiting 10 seconds
s6-supervise (child): fatal: unable to exec run: Permission denied
s6-supervise php-fpm: warning: unable to spawn ./run - waiting 10 seconds
[services.d] done.
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.
^C
[freshrss@vault ~]$ ls -la data/
total 2
drwxr-xr-x. 2 freshrss freshrss 2 Jun 14 20:19 .
drwxr-xr-x. 5 freshrss freshrss 6 Jun 14 20:21 ..

Now without --userns=keep-id:

[freshrss@vault ~]$ podman rm -f freshrss
[freshrss@vault ~]$ podman run  --name=freshrss --publish 127.0.0.1:8008:80 --volume ~/data/:/config:Z docker.io/linuxserver/freshrss:latest
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 01-envfile: executing... 
[cont-init.d] 01-envfile: exited 0.
[cont-init.d] 10-adduser: executing... 
usermod: no changes

-------------------------------------
          _         ()
         | |  ___   _    __
         | | / __| | |  /  \ 
         | | \__ \ | | | () |
         |_| |___/ |_|  \__/

Brought to you by linuxserver.io
-------------------------------------

To support LSIO projects visit:
https://www.linuxserver.io/donate/
-------------------------------------
GID/UID
-------------------------------------

User uid:    911
User gid:    911
-------------------------------------

[cont-init.d] 10-adduser: exited 0.
[cont-init.d] 20-config: executing... 
[cont-init.d] 20-config: exited 0.
[cont-init.d] 30-keygen: executing... 
generating self-signed keys in /config/keys, you can replace these with your own keys if required
Generating a RSA private key
...+++++
...........................................................+++++
writing new private key to '/config/keys/cert.key'
-----
[cont-init.d] 30-keygen: exited 0.
[cont-init.d] 40-install: executing... 
[cont-init.d] 40-install: exited 0.
[cont-init.d] 99-custom-files: executing... 
[custom-init] no custom files found exiting...
[cont-init.d] 99-custom-files: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
^C
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.
[freshrss@vault ~]$ ls -la data/
total 6
drwxr-xr-x. 8 60600910 60600910 8 Jun 14 20:26 .
drwxr-xr-x. 5 freshrss freshrss 6 Jun 14 20:21 ..
drwxr-xr-x. 2 60600910 60600910 3 Jun 14 20:26 crontabs
drwxr-xr-x. 2 60600910 60600910 4 Jun 14 20:26 keys
drwxr-xr-x. 4 60600910 60600910 4 Jun 14 20:26 log
drwxrwxr-x. 3 60600910 60600910 4 Jun 14 20:26 nginx
drwxr-xr-x. 2 60600910 60600910 4 Jun 14 20:26 php
drwxrwxr-x. 3 60600910 60600910 4 Jun 14 20:26 www
giuseppe commented 3 years ago

I defer to @giuseppe on whether it's possible to have multiple GIDs in the container map to one GID on the host

that is not possible. One ID in the user namespace can only map to one ID in the outer environment.

* this is the case I'm trying to understand, when a user that isn't `uid` `0` is in play (e.g. linuxserver containers and something like gitlab) how can I get their disk writes in volumes to not be shifted by `subuid`.

you'd need to allocate IDs 1->N on the host to the rootless user. I really suggest you to not do it, as you allow the unprivileged user to use system IDs.

Why is such a requirement to have the same IDs mapped to the same values on the host?

tobwen commented 3 years ago

Why is such a requirement to have the same IDs mapped to the same values on the host?

Speaking of myself (not as the topic owner): PostgreSQL for example creates an internal UID 999 (postgres) to store the database's data. When trying to backup this data from "outside", the access to this directory fails. So you need to be root or allow the outer user sudo rights. Sure, there is podman unshare, but the file being created again belongs to another user. This can be chowned from outside, but it's pretty ugly to handle by scripting or external processes.

I'm using --userns=keep-id in most cases, but this changes all the volumes/mounts... not what everyone wants.

andrewgdunn commented 3 years ago

Why is such a requirement to have the same IDs mapped to the same values on the host?

It depends on the container composition. The example I expressed with freshrss and containers built by this community was because there are likely many people using these tools and running into this use case. There are other cases of more complex containers, like gitlab, which has uid of 991, 995, 996, 997 and a gid of 998 that shows up. This makes interaction with data on disk more difficult (e.g. requires root, sudo, or unshare).

[root@citadel data]# ls -la
total 173
drwxr-xr-x. 20 gitlab   gitlab    25 Jun  4 12:42 .
drwx------.  7 gitlab   gitlab     8 Mar  4 17:41 ..
drwxr-x---.  3 80000991 gitlab     4 Jun  4 12:44 alertmanager
drwx------.  4 80000997 gitlab    10 Jun  5 05:46 backups
-rw-------.  1 gitlab   gitlab    38 Jul 31  2020 bootstrapped
drwxr-xr-x.  2 80000997 80000997   2 Jul 31  2020 .bundle
drwx------.  3 80000997 gitlab     8 Jun  4 12:44 gitaly
-rw-r--r--.  1 80000997 80000997 367 Jul 31  2020 .gitconfig
drwx------.  3 80000997 80000997   3 Jul 31  2020 git-data
drwxr-xr-x.  3 80000997 gitlab     3 Jul 31  2020 gitlab-ci
drwxr-xr-x.  2 80000997 gitlab     4 Jun  4 12:43 gitlab-exporter
drwxr-xr-x.  9 80000997 gitlab    12 Jun  4 12:42 gitlab-rails
drwx------.  2 80000997 gitlab     3 Jun  4 12:42 gitlab-shell
drwxr-x---.  3 80000997 80000998   5 Jun  4 12:43 gitlab-workhorse
drwx------.  4 80000991 gitlab     7 Jun  4 12:44 grafana
drwx------.  3 gitlab   gitlab     5 Jun 15 08:53 logrotate
drwxr-x---.  9 gitlab   80000998  11 Jun  4 12:43 nginx
drwx------.  2 80000995 gitlab     3 Jun  4 12:44 postgres-exporter
drwxr-xr-x.  4 80000995 gitlab     9 Jun  4 12:42 postgresql
-rw-r--r--.  1 gitlab   gitlab     5 Feb  1 17:22 postgresql-version.old
drwxr-x---.  4 80000991 gitlab     5 Jun  4 12:44 prometheus
-rw-r--r--.  1 gitlab   gitlab   483 Jun  4 12:44 public_attributes.json
drwxr-x---.  2 80000996 80000997  13 Jun 15 09:48 redis
drwx------.  2 80000997 80000997   4 Jul 31  2020 .ssh
-rw-r--r--.  1 gitlab   gitlab    40 Jun  4 12:42 trusted-certs-directory-hash

I'm curious about how to do the mapping when the case is a single non root-user. Like with the linuxserver.io where there is expected to only be one. In this case I'd specifically want to map 910 to my host uid, but all others I'd want to shift in the subuid space. The thing I struggle with is that it appears that the way --uidmap works is sequential. So... Would I have to do three calls to --uidmap like:

--uidmap=0:60600000:909 --uidmap=910:606:1 --uidmap=911:60600911:64626

For a case where there are multiple non-root users... I have no idea what to do. Is it possible to run podman as a non-root user on the host but run it "rootfull" as that non-root user so that it's interactions on disk are all mapped to the host user's uid?

andrewgdunn commented 3 years ago

@tobwen when you use --userns=keep-id does this ensure that the volume for postgres is written as the host uid?

tobwen commented 3 years ago

when you use --userns=keep-id does this ensure that the volume for postgres is written as the host uid?

Yes. But ALL the users inside the container (e.g. 333 666 999 65535) get smashed to host uid.

andrewgdunn commented 3 years ago

Yes. But ALL the users inside the container (e.g. 333 666 999 65535) get smashed to host uid.

This is exactly what I actually want with the example of freshrss above, could you speculate as to why I'm seeing those permissions issues?

tobwen commented 3 years ago

Oh, I was using --user tobwen --userns=keep-id here. You're right, I'm also ending up on IDs like 166536 on the host, when creating another user.

jmpolom commented 3 years ago

keep-id: creates a user namespace where the current rootless user’s UID:GID are mapped to the same values in the container. This option is ignored for containers created by the root user.

From the podman run documentation it would appear that using --userns=keep-id will only affect the uid:gid of the user that starts the command inside of the container once running. How would this result in several other uid/gid getting mapped to a different uid? Similarly, the --user flag only affects the uid:gid of the user that runs the default command/entrypoint for the container image as specified by the appropriate directive in the Containerfile.

andrewgdunn commented 3 years ago

@jmpolom this is what I expected but didn't have a chance to fully verify. My assumption was that --user was implicit when running rootless.

The question I'd have is, how do I "snipe" some UIDs and map them to host IDs? similar to what I'd mentioned above, for a single UID mapping:

--uidmap=0:60600000:909 --uidmap=910:606:1 --uidmap=911:60600911:64626

For multiple?

--uidmap=0:60600000:909 --uidmap=910:606:1 --uidmap=911:607:1 --uidmap=912:60600911:64625

Then I can use group membership of host based users to alleviate management of files on disk.

tobwen commented 3 years ago

I think there's a more difficult thing to consider. When you would be able to assign multiple "users inside the container" to one user "outside the container" - how to get reassign it correctly on the next run?

Let's say: 991, 995, 996, 997 (inside) get mapped to 1001 outside How do you want to recover the internal UIDs, when it's all 1001 outside? You can't "split" it back to the original "internal" UIDs.

Did you consider ACLs on those directories? ACLs often don't get overwritten by containers, since the permissions are missing. But beware of creating new files or chowning permissions. Those have to match the "internal" UID of course.

rhatdan commented 3 years ago

I think you are asking for stuff that user namespace can not do.

if you want to use real UIDs mapped to real UID within the container, then you need to handle this with --uidmap, and --gidmap, which will not be pretty. You would also need all of the entries added to /etc/subuid and /etc/subgid.

rhatdan commented 3 years ago

BTW You can manipulate all of the files in the volume, by simply executing a script with

podman unshare chownand then chown the files to what you want.

giuseppe commented 3 years ago

mapping system IDs (< 1000) in a rootless container is a bad practice as you are allowing your unprivileged user to access system IDs.

If you want 1->1 mapping on the host, could it be done with a a root container without user namespaces?

tobwen commented 3 years ago

podman unshare chownand then chown the files to what you want.

Wouldn't a chmod 777 (or such) be better because a chown might make it unreadable from the container. Or chown :host-user and chmod g+rwX?

giuseppe commented 3 years ago

depends what you are aiming at.

If you want to make the files world writeable/readable then that is fine.

if you want to chown to your own user, then you need a different command, and that probably will make files not accessible from the container.

I am closing the issue as I am not sure what we can do from Podman, but feel free to keep commenting