Open schrammae opened 1 week ago
Hi,
So is it somehow possible to use kas with rootless Docker at all?
we need to look into this, but unfortunately I don't have a rootless docker environment. Many of our users are using podman in rootless mode (including myself) and that definitely works. It can indeed be, that the user id mappings (which are only added for podman) are missing.
Please also have a look at the flags added to podman (just grep for podman in kas-container
) and check if similar options exist for docker. Once we know what is missing, we can add this (given that there is a way to detect if docker is running in rootless mode).
Hi @fmoessbauer,
thanks for your reply.
we need to look into this, but unfortunately I don't have a rootless docker environment. Many of our users are using podman in rootless mode (including myself) and that definitely works.
Can confirm - we used Podman before and are still partially using it. But Docker seems to offer better package support, i.e. one gets the latest Docker versions properly packaged for the current stable Debian/Ubuntu version. That's why our admins decided to go back to Docker and introduce rootless mode.
And they also don't want to maintain multiple container runtimes which is understandable. So at least for productive systems - which also covers the server I'd like to use for Yocto building - this means rootless Docker only.
Please also have a look at the flags added to podman (just grep for podman in
kas-container
) and check if similar options exist for docker.
I did, but besides --userns=keep-id
I don't see anything relevant. --security-opt label=disable
(1688d60):
This is for SELinux enabled systems only.
Once we know what is missing, we can add this (given that there is a way to detect if docker is running in rootless mode).
Detecting this is very easy:
$ docker context show
default
$ docker context show
rootless
It can indeed be, that the user id mappings (which are only added for podman) are missing.
I already stumbled upon that commit. The problem ist that --userns=keep-id
is exclusive to Podman.
Disable namespace remapping for a container (--userns)
host
is the only valid value for the--userns
flag.
Digging around:
$ docker context inspect
[
{
"Name": "rootless",
"Metadata": {
"Description": "Rootless mode"
},
"Endpoints": {
"docker": {
"Host": "unix:///run/user/2117/docker.sock",
"SkipTLSVerify": false
}
},
"TLSMaterial": {},
"Storage": {
"MetadataPath": "/home/schramm/.docker/contexts/meta/12b961af5feb3e9d39f93b2cefb9a1a944f18d02cca0cac2f04f5a982240605f",
"TLSPath": "/home/schramm/.docker/contexts/tls/12b961af5feb3e9d39f93b2cefb9a1a944f18d02cca0cac2f04f5a982240605f"
}
}
]
$ docker inspect a9eec6ca8825 | grep -E 'Privileged|UsernsMode|User'
"Privileged": false,
"UsernsMode": "",
"User": "root",
Removing build/
and poky/
to clean the work dir, running with --userns=host
:
--- Error summary ---
fatal: could not create work tree dir '/work/poky': Permission denied
Verifying the settings:
$ docker run -v /home/schramm/src/kas/:/repo:ro -v /home/schramm/src/kas/:/work:rw -e KAS_WORK_DIR=/work -v /home/schramm/src/kas/build:/build:rw --workdir=/repo -e KAS_BUILD_DIR=/build -e USER_ID=2117 -e GROUP_ID=100 --rm --init -t -i -e TERM=tmux-256color -e SHELL=/bin/bash --log-driver=none --user=root --userns=host ghcr.io/siemens/kas/kas:4.4 /bin/bas
builder@86e8baf1a2cd:/repo$ id
uid=2117(builder) gid=100(users) groups=100(users)
builder@86e8baf1a2cd:/repo$ ls -lad /work/
drwxr-xr-x 3 root root 4096 Jul 3 15:24 /work/
$ docker inspect 86e8baf1a2cd | grep -E 'Privileged|UsernsMode|User'
"Privileged": false,
"UsernsMode": "host",
"User": "root",
Some research revealed Docker rootless mode without userns-remap.
Unfortunately, Docker in rootless mode does currently not support disabling the use of a user namespace; the option
--userns=host
is ignored.
Out of curiosity, testing without --user=root
:
$ docker run -v /home/schramm/src/kas/:/repo:ro -v /home/schramm/src/kas/:/work:rw -e KAS_WORK_DIR=/work -v /home/schramm/src/kas/build:/build:rw --workdir=/repo -e KAS_BUILD_DIR=/build -e USER_ID=2117 -e GROUP_ID=100 --rm --init -t -i -e TERM=tmux-256color -e SHELL=/bin/bash --log-driver=none ghcr.io/siemens/kas/kas:4.4 /bin/bash
groupmod: Permission denied.
groupmod: cannot lock /etc/group; try again later.
usermod: user builder is currently used by process 1
chown: changing ownership of '/builder/.profile': Operation not permitted
chown: changing ownership of '/builder/.bash_logout': Operation not permitted
chown: changing ownership of '/builder/.bashrc': Operation not permitted
chown: changing ownership of '/builder': Operation not permitted
error: failed switching to "builder": operation not permitted
And running as root
with -e USER_ID=0 -e GROUP_ID=0
is also not an option as this violates Bitbake's principles and makes Git barf about dubious file ownerships as shown above.
I'm not an expert in namespace mapping, but to me it looks like using kas with rootless Docker is currently not possible?
So this seems to sum it up perfectly:
Hope and pray that Docker will implement
--userns=keep-id
same as in Podman.
Hi @schrammae,
If I remember right, the time podman was discussing rootless, they introduced fuse-overlayfs to overcome those issues.
Do you have it installed? It is anyway Debian recommended setup for rootless containers: https://docs.docker.com/engine/security/rootless/
Hi @Silvanoc,
Thanks for your reply.
Do you have it installed? It is anyway Debian recommended setup for rootless containers: https://docs.docker.com/engine/security/rootless/
fuse-overlayfs
is currently not installed on the server:
$ apt -qq list fuse-overlayfs
fuse-overlayfs/stable 1.10-1 amd64
The Docker config looks like this:
$ grep storage-driver /etc/docker/daemon-rootless.json
"storage-driver": "overlay2"
$ docker info
...
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: true
Checking the storage driver on a running container:
$ docker inspect --format='{{.Driver}}' e71a71cb6a8d
overlay2
Checking the docs: Docker storage drivers
fuse-overlayfs is preferred only for running Rootless Docker on a host that does not provide support for rootless overlay2. On Ubuntu and Debian 10, the fuse-overlayfs driver does not need to be used, and overlay2 works even in rootless mode.
So overlay2
seems to be ok and should not require fuse-overlayfs
as far as I understand.
overlay2 (only if running with kernel 5.11 or later, or Ubuntu-flavored kernel)
The server is running Debian 6.1.94-1 (2024-06-21), so this should also be fine.
As I can't do any experiments on the server, I installed rootless Docker on my local machine. This is using Ubuntu 22.04.4 LTS and the Ubuntu guide does not mention fuse-overlayfs
at all.
$ lsb_release -d
Description: Ubuntu 22.04.4 LTS
$ docker context show
rootless
$ docker info
...
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: true
This gives the same error as on Debian:
kas ±|master|$ KAS_VERBOSE=1 kas-container --log-level debug build examples/openembedded.yml
+ mkdir -p /home/schramm/src/kas
+ mkdir -p /home/schramm/src/kas/build
+ docker run -v /home/schramm/src/kas:/repo:ro -v /home/schramm/src/kas:/work:rw -e KAS_WORK_DIR=/work -v /home/schramm/src/kas/build:/build:rw --workdir=/repo -e KAS_BUILD_DIR=/build -e USER_ID=2117 -e GROUP_ID=100 --rm --init -t -i -e TERM=xterm-256color -e SHELL=/bin/bash --log-driver=none --user=root ghcr.io/siemens/kas/kas:4.4 -l debug build /repo/examples/openembedded.yml
2024-07-09 08:56:42 - INFO - kas 4.4 started
2024-07-09 08:56:42 - DEBUG - Using selector: EpollSelector
2024-07-09 08:56:42 - DEBUG - /repo/examples$ git rev-parse --show-toplevel
2024-07-09 08:56:42 - DEBUG - /repo/examples$ hg root
2024-07-09 08:56:42 - DEBUG - /repo/examples$ git rev-parse --show-toplevel
2024-07-09 08:56:42 - DEBUG - /repo/examples$ hg root
2024-07-09 08:56:42 - DEBUG - execute setup_dir
2024-07-09 08:56:42 - DEBUG - execute setup_home
2024-07-09 08:56:42 - DEBUG - execute init_setup_repos
2024-07-09 08:56:42 - DEBUG - execute repo_setup_loop
2024-07-09 08:56:42 - DEBUG - Loop repo_setup_loop: execute setup_repos_step
2024-07-09 08:56:42 - DEBUG - execute finish_setup_repos
2024-07-09 08:56:42 - INFO - Cloning repository poky
2024-07-09 08:56:42 - DEBUG - /work$ git clone -q https://git.yoctoproject.org/poky.git /work/poky
2024-07-09 08:56:42 - ERROR - Command "/work$ git clone -q https://git.yoctoproject.org/poky.git /work/poky" failed
--- Error summary ---
fatal: could not create work tree dir '/work/poky': Permission denied
2024-07-09 08:56:42 - ERROR - fetch repos failed: error code 128
Creating a configuration file and testing with fuse-overlayfs
$ docker info
...
Storage Driver: fuse-overlayfs
$ apt -qq list fuse-overlayfs
fuse-overlayfs/jammy,now 1.7.1-1 amd64 [installed]
But this results in the same error as with overlay2
:
kas ±|master|$ KAS_VERBOSE=1 kas-container --log-level debug build examples/openembedded.yml
+ mkdir -p /home/schramm/src/kas
+ mkdir -p /home/schramm/src/kas/build
+ docker run -v /home/schramm/src/kas:/repo:ro -v /home/schramm/src/kas:/work:rw -e KAS_WORK_DIR=/work -v /home/schramm/src/kas/build:/build:rw --workdir=/repo -e KAS_BUILD_DIR=/build -e USER_ID=2117 -e GROUP_ID=100 --rm --init -t -i -e TERM=xterm-256color -e SHELL=/bin/bash --log-driver=none --user=root ghcr.io/siemens/kas/kas:4.4 -l debug build /repo/examples/openembedded.yml
Unable to find image 'ghcr.io/siemens/kas/kas:4.4' locally
4.4: Pulling from siemens/kas/kas
09f376ebb190: Pull complete
2fc2fe69fbd5: Pull complete
3a271d2e7af8: Pull complete
701941e108db: Pull complete
b775f8c62321: Pull complete
132ee506e289: Pull complete
5e4f5ae397c3: Pull complete
54c27b899183: Pull complete
Digest: sha256:0bd8643166f22ad154d8a1cf9e56e9fb94fc08f6e03f42bba0110da8860b554f
Status: Downloaded newer image for ghcr.io/siemens/kas/kas:4.4
2024-07-09 09:05:52 - INFO - kas 4.4 started
2024-07-09 09:05:52 - DEBUG - Using selector: EpollSelector
2024-07-09 09:05:52 - DEBUG - /repo/examples$ git rev-parse --show-toplevel
2024-07-09 09:05:52 - DEBUG - /repo/examples$ hg root
2024-07-09 09:05:52 - DEBUG - /repo/examples$ git rev-parse --show-toplevel
2024-07-09 09:05:52 - DEBUG - /repo/examples$ hg root
2024-07-09 09:05:52 - DEBUG - execute setup_dir
2024-07-09 09:05:52 - DEBUG - execute setup_home
2024-07-09 09:05:52 - DEBUG - execute init_setup_repos
2024-07-09 09:05:52 - DEBUG - execute repo_setup_loop
2024-07-09 09:05:52 - DEBUG - Loop repo_setup_loop: execute setup_repos_step
2024-07-09 09:05:52 - DEBUG - execute finish_setup_repos
2024-07-09 09:05:52 - INFO - Cloning repository poky
2024-07-09 09:05:52 - DEBUG - /work$ git clone -q https://git.yoctoproject.org/poky.git /work/poky
2024-07-09 09:05:52 - ERROR - Command "/work$ git clone -q https://git.yoctoproject.org/poky.git /work/poky" failed
--- Error summary ---
fatal: could not create work tree dir '/work/poky': Permission denied
2024-07-09 09:05:52 - ERROR - fetch repos failed: error code 128
Running it manually:
$ docker run -v /home/schramm/src/kas:/repo:ro -v /home/schramm/src/kas:/work:rw -e KAS_WORK_DIR=/work -v /home/schramm/src/kas/build:/build:rw --workdir=/repo -e KAS_BUILD_DIR=/build -e USER_ID=2117 -e GROUP_ID=100 --rm --init -t -i -e TERM=xterm-256color -e SHELL=/bin/bash --log-driver=none --user=root ghcr.io/siemens/kas/kas:4.4 /bin/bash
$ docker inspect --format='{{.Driver}}' e514522b20f6
fuse-overlayfs
builder@e514522b20f6:/repo$ id
uid=2117(builder) gid=100(users) groups=100(users)
builder@e514522b20f6:/repo$ ls -lad /work/
drwxr-xr-x 12 root root 4096 Jul 2 12:45 /work/
builder@e514522b20f6:/repo$ mkdir /work/poky
mkdir: cannot create directory ‘/work/poky’: Permission denied
Hi.
For security reasons, we're using Docker in rootless mode. As opposed to default Docker mode, kas fails to do builds due to permission problems when creating the work tree dirs.
Running it manually:
Just for testing, I tried running with
--docker-args "-e USER_ID=0 -e GROUP_ID=0"
:Testing further and silencing this error via
touch build/conf/sanity.conf
works to build the example. However, it fails on more complex builds withfatal: detected dubious ownership in repository
:So is it somehow possible to use kas with rootless Docker at all?