Document what to do when running into permission issues using `podman compose up`

JonahSussman commented 2 weeks ago

This is what happens when trying to run compose up for me. I thought using use Z instead of z for the volumes in the compose.yaml fixed it, but that was a misnomer.

$ lsb_release -a
LSB Version:    :core-5.0-amd64:core-5.0-noarch
Distributor ID: Fedora
Description:    Fedora release 40 (Forty)
Release:        40
Codename:       Forty

I think that this is unrelated to SELinux, even though I'm running Fedora, as it's disabled on my machine:

$ sestatus && getenforce
SELinux status:                 disabled
Disabled

kai_db-1  | Warning: Can't detect cpuset size from cgroups, will use nproc
kai_db-1  | waiting for server to start....2024-08-29 22:41:11.464 UTC [19] LOG:  redirecting log output to logging collector process
kai_db-1  | 2024-08-29 22:41:11.464 UTC [19] HINT:  Future log output will appear in directory "log".
kai_db-1  |  done
kai_db-1  | server started
kai_db-1  | /var/run/postgresql:5432 - accepting connections
kai_db-1  | => sourcing /usr/share/container-scripts/postgresql/start/set_passwords.sh ...
kai_db-1  | ALTER ROLE
kai_db-1  | waiting for server to shut down.... done
kai_db-1  | server stopped
kai_db-1  | Starting server...
kai_db-1  | 2024-08-29 22:41:11.691 UTC [1] LOG:  redirecting log output to logging collector process
kai_db-1  | 2024-08-29 22:41:11.691 UTC [1] HINT:  Future log output will appear in directory "log".
kai-1     | ################################################
kai-1     | load-data has never been run.                  #
kai-1     | Please wait, this will take a few minutes.     #
kai-1     | ################################################
kai-1     | Using custom config file
kai-1     | Console logging for 'kai' is set to level 'DEBUG'
kai-1     | Traceback (most recent call last):
kai-1     |   File "/kai/./kai/service/incident_store/incident_store.py", line 488, in <module>
kai-1     |     cmd()
kai-1     |   File "/kai/./kai/service/incident_store/incident_store.py", line 462, in cmd
kai-1     |     initLogging(
kai-1     |   File "/kai/kai/kai_logging.py", line 56, in initLogging
kai-1     |     setup_file_handler(parent_log, log_file, log_dir, file_log_level)
kai-1     |   File "/kai/kai/kai_logging.py", line 45, in setup_file_handler
kai-1     |     file_handler = logging.FileHandler(log_file_path)
kai-1     |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kai-1     |   File "/usr/lib64/python3.12/logging/__init__.py", line 1231, in __init__
kai-1     |     StreamHandler.__init__(self, self._open())
kai-1     |                                  ^^^^^^^^^^^^
kai-1     |   File "/usr/lib64/python3.12/logging/__init__.py", line 1263, in _open
kai-1     |     return open_func(self.baseFilename, self.mode,
kai-1     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kai-1     | PermissionError: [Errno 13] Permission denied: '/podman_compose/logs/kai_psql.log'
kai-1     | ################################################
kai-1     | load-data has completed, starting server.      #
kai-1     | ################################################

jwmatthews commented 2 weeks ago

Adding a link related to some of the differences with permissions and root/rootless podman: https://www.tutorialworks.com/podman-rootless-volumes/

This post is likely to help as well: https://www.redhat.com/sysadmin/container-permission-denied-errors

JonahSussman commented 2 weeks ago

I haven't had a chance to look at the links, but I'll just document what I've been doing:

I've been looking at running the containers individually, like

$ podman compose up kai_db

In one terminal and this beast in another

$ podman run -it -v "$(pwd):/podman_compose:z" --entrypoint /bin/bash --uidmap 1001:0:1 --uidmap 0:1:1000 --network=kai-jonah_default localhost/my_kai

(I can't figure out how to pass the uidmap arguments in podman compose)

It's doing... something alright. It's setting the owner of the /podman_compose directory to 1001 (the user in the container) in the container. But I'm still getting some weird errors with logs.

Adding the volume, specifically outlining logs with these flags, either doesn't work when booting or changes the owner to 1001 on the host machine too when running.

My hunch is that if we can get the user mapping right, then rootless podman should work.

JonahSussman commented 1 week ago

I finally found a solution that works, but it's not ideal.

Explanation

Rootless podman containers can run as any user id, which makes things tricky. If the user is root (uid 0) in the container, it gets mapped to your user on the host system. However, containers with other users (e.g. 123) will map to a uid on the host based on the subid offset range found in /etc/subuid (e.g. 10123). ^1

Thus, this is what's happening:

Launch podman compose up as rootless podman.
Inside our container, the user is 1001. From the host's perspective it's running as a mapped user, say 11001.
The volume ${PWD}/logs:/podman_compose/logs:rw,Z passes the kai directory to the container.
This directory is owned by the host user and gets mapped to root inside the container.
User 1001 inside the container cannot read or write to something owned by root, so we get permission errors.

Solutions

1. `podman unshare`

The first solution I came up with is the simplest, but also the least ideal for extensibility. We can simply make the mapped user own the directory on the host with this command:

chown -R jonah:jonah logs
podman unshare -R 1001:0 logs

Podman unshare executes commands as the mapped user, setting the logs directory to be owned by user 1001. (Note, for some reason, the container's group id is 0, which is its own set of weirdness that I didn't dive into). podman compose up works as expected.

This fixes the issue but also adds another layer of config that the end-user has to do. Plus, I'm not too sure if other containers can access the logs directory if we add more.

2. Running each container individually

Open two terminals. In the first, run:

podman compose run kai_db

Wait for it to initialize, then in another terminal do:

[host] $ podman run -it -v "$(pwd):/podman_compose:z" --entrypoint /bin/bash --uidmap 1001:0:1 --uidmap 0:1:1000 --network=kai-jonah_default localhost/my_kai
[container] $ /usr/local/bin/entrypoint.sh

This also works. Something is going on with uid mapping that allows this to work. Unfortunately, I can't figure out how to pass this through into podman compose. You can pass podman arguments to podman-compose, ^3 but not to docker-compose, which is most likely the default for most people.

There's something here about passing --userns arguments instead ^4, but nothing seems to work.

jmontleon commented 1 week ago

This is expected behavior for podman when using rootless containers.. What you did with podman unshare is correct.

konveyor / kai