Open mfhepp opened 8 months ago
I also tried building the Docker image locally with the host user UID, GID and username included in the build process as mycromamba
, to no avail:
# Build micromamba-docker:1.5.6 locally
docker build . -t mycromamba --build-arg="MAMBA_USER=$USER" \
--build-arg="MAMBA_USER_ID=$(id -u)" \
--build-arg="MAMBA_USER_GID=$(id -g)"
$ docker run --rm -it --user $UID:$GID -v "$(pwd):/home/foobar/data" mycromamba /bin/bash
$ cd /home/foobar/data/
$ ls -la
total 8
drwxr-xr-x 2 root root 4096 Jan 11 03:15 .
drwxrwxrwx 3 foobar foobar 4096 Jan 11 03:38 ..
# Writing to bind mount /home/foobar/data fails:
$ touch test.txt
touch: cannot touch 'test.txt': Permission denied
# Writing to user directory /home/foobar *inside the container* works (but is not bound to host)
$ cd ..
$ touch test.txt
$ ls
data test.txt
Works for me:
$ docker run -it --rm -v $(pwd):/home/mambauser -u $(id -u):$(id -g) mambaorg/micromamba:1.5.6 /bin/bash -c 'touch /home/mambauser/foobar'
$ docker version
Client: Docker Engine - Community
Version: 24.0.6
API version: 1.43
Go version: go1.20.7
Git commit: ed223bc
Built: Mon Sep 4 12:32:16 2023
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 24.0.6
API version: 1.43 (minimum version 1.12)
Go version: go1.20.7
Git commit: 1a79695
Built: Mon Sep 4 12:32:16 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.24
GitCommit: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
runc:
Version: 1.1.9
GitCommit: v1.1.9-0-gccaecfc
docker-init:
Version: 0.19.0
GitCommit: de40ad0
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
On the host, do you have write permission to the working directory where you are executing the docker run ...
command?
this:
drwxr-xr-x 2 root root 4096 Jan 11 03:15 .
seems to indicate that you do not have write permissions in that directory on the host.
Just upgraded docker to the same version you tried, and I still cannot reproduce it when in a host directory where I have write access.
$ docker version
Client: Docker Engine - Community
Version: 24.0.7
API version: 1.43
Go version: go1.20.10
Git commit: afdd53b
Built: Thu Oct 26 09:08:17 2023
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 24.0.7
API version: 1.43 (minimum version 1.12)
Go version: go1.20.10
Git commit: 311b9ff
Built: Thu Oct 26 09:08:17 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.26
GitCommit: 3dd1e886e55dd695541fdcd67420c2888645a495
runc:
Version: 1.1.10
GitCommit: v1.1.10-0-g18a0cb0
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Works fine for me too:
$ docker run --rm -it -v "$(pwd):/tmp" --user $UID:$GID mambaorg/micromamba:1.5.6 /bin/bash
(base) I have no name!@9b2eefcffcd4:/tmp$ touch test && ls
test
Thanks for looking into this! Please note that I am using a rootless Docker installation on the host machine, as is best practice; this may have an effect-
For my py4docker project, I was able to solve the issue as follows:
sudo
group.docker context use rootless
Without a lot of effort, I cannot tell exactly if the UID of 1000 or the sudo group membership of the user on the host cause this behavior, but the cause is most likely one of the two. The CLI context is a requirement to access the Docker daemon, but does not fix the issue (I can still reproduce it by using the UID 1000 user with sudo rights).
Again, thanks for your time and support!
PS: The problem seems to have a long history, hence I add a link to resources and findings on the topic of Docker bind mounts here for future visitors to this issue.
Works fine for me too:
$ docker run --rm -it -v "$(pwd):/tmp" --user $UID:$GID mambaorg/micromamba:1.5.6 /bin/bash (base) I have no name!@9b2eefcffcd4:/tmp$ touch test && ls test
In this test, it is IMO not 100% certain if the write operation takes place inside the mount point or elsewhere (it depends on the WORKDIR). mambauser
can write to many places inside the container, as long as the OS is not set to read-only. In my project, I take quite some effort to minimize persistent, harmful actions by the application code or its dependencies.
Edit: I checked in interactive mode; this indeed works if my user has no sudo membership nor a UID of 1000.
One small addendum:
$GID
enviroment variable, which leads to a GID of 0 and root group membership.--user $(id -u):$(id -g)
than --user $UID:$GID
:With --user $(id -u):$(id -g)
:
docker run --rm -it -v "$(pwd):/tmp" --user $(id -u):$(id -g) mambaorg/micromamba:1.5.6 /bin/bash
(base) I have no name!@8b30d9644121:/tmp$ id
uid=1000 gid=1000 groups=1000
With --user $UID:$GID
:
docker run --rm -it -v "$(pwd):/tmp" --user $UID:$GID mambaorg/micromamba:1.5.6 /bin/bash
(base) I have no name!@1433c5a8d78d:/tmp$ id
uid=1000 gid=0(root) groups=0(root)
I also find that the --user
approach does allow me to access mounts in our rootless podman setup. However, --userns=keep-id:uid=1000,gid=1000
is a working alternative.
(Note that micromamba 1.5.0 and later use UID/GID 57439 instead of 1000, so you need to modify that to --userns=keep-id:uid=57439,gid=57439
)
For others running into similar problems, this tool might be useful:
Still working on the issue... and I think there is an underlying problem that surfaces when you are using micromamba-docker
As far as I understand, the non-root mambauser
conflicts with the way Docker in rootless mode deals with bind mounts and namespace mapping in general.
Docker in rootless mode utilizes rootlesskit
, which maps user namespaces between the host system and the system inside the container.
See how rootlesskit
changes the username, UID, and GID of resources owned by a local user:
# Create a test folder and file
# Default permissions set by umask
$ mkdir foo
$ cd foo
$ mkdir bar
$ touch mytext.txt
# Show permissions from the host user's perspective
$ ls -la
total 12
drwxr-xr-x 3 myusername myusername 4096 Apr 30 19:23 .
drwxr-xr-x 15 myusername myusername 4096 Apr 30 19:23 ..
drwxr-xr-x 2 myusername myusername 4096 Apr 30 19:23 bar
-rw-r--r-- 1 myusername myusername 0 Apr 30 19:23 mytext.txt
$ id
uid=1000(myusername) gid=1000(myusername) groups=1000(myusername),996(docker)
# Now turning on RootlessKit, as used by Docker
# RootlessKit creates user_namespaces and executes newuidmap/newgidmap along with subuid and subgid
# https://github.com/rootless-containers/rootlesskit
$ rootlesskit bash
root@110:~/foo# ls -la
total 12
drwxr-xr-x 3 root root 4096 Apr 30 19:23 .
drwxr-xr-x 15 root root 4096 Apr 30 19:23 ..
drwxr-xr-x 2 root root 4096 Apr 30 19:23 bar
-rw-r--r-- 1 root root 0 Apr 30 19:23 mytext.txt
root@110:~/foo# id
uid=0(root) gid=0(root) groups=0(root),65534(nogroup)
root@110:~/foo# exit
You can see that rootlesskit
changes the owning user and user group from myusername
to root
and maps the UID from 1000 to 0.
This means that the non-root mambauser
cannot access such files and folders, even if the UID/GID of the local user is passed to the container when Docker is run in the rootless mode (for which there are good reasons).
The exact mapping is determined by /etc/subuid
and /etc/subgid
.
I am reopening the issue, because
It may be possible to fix this by
/etc/subuid
and /etc/subgid
, or modifying_entrypoint.sh
shell script or Dockerfile
.Any ideas will be very much appreciated!
PS: The test from earlier on fails now, too, since I changed my system to a rootless Docker configuration:
$ docker run --rm -it -v "$(pwd):/tmp" --user $(id -u):$(id -g) mambaorg/micromamba:1.5.8 /bin/bash
(base) I have no name!@a84cfa7e0ace:/tmp$ touch test
touch: cannot touch 'test': Permission denied
(base) I have no name!@a84cfa7e0ace:/tmp$ id
uid=1000 gid=1000 groups=1000
(base) I have no name!@a84cfa7e0ace:/tmp$ ls -la
total 12
drwxr-xr-x 3 root root 4096 Apr 30 19:23 .
drwxr-xr-x 17 root root 4096 Apr 30 19:54 ..
drwxr-xr-x 2 root root 4096 Apr 30 19:23 bar
-rw-r--r-- 1 root root 0 Apr 30 19:23 mytext.txt
As shown above with rootlesskit
alone, despite the indicated GID and UID being identical, the user namespace mechanism changes the owner to root
from inside the container:
# Permissions from the host user's perspective
$ ls -la
total 12
drwxr-xr-x 3 myusername myusername 4096 Apr 30 19:23 .
drwxr-xr-x 15 myusername myusername 4096 Apr 30 19:23 ..
drwxr-xr-x 2 myusername myusername 4096 Apr 30 19:23 bar
-rw-r--r-- 1 myusername myusername 0 Apr 30 19:23 mytext.txt
$ id
uid=1000(myusername) gid=1000(myusername) groups=1000(myusername),996(docker)
Wow, thanks @mfhepp for your persistence in getting to the bottom of this issue! This looks like it was quite some effort.
I have not yet used rootless mode myself (although I probably should), so this is a bit beyond my current comfort level. Thus I don't have a preference among your suggested approaches for a fix. Perhaps @wholtz has some thoughts?
Have been doing additional experiments today - here is my current summary:
Essentially, there are at least four solutions:
Option 1: Make mambauser
root with --user root
I think the most pragmatic way to deal with this is to use --user root
if, and only if
Then mambauser
will internally be root, but thanks to rootlesskit
etc., this will be mapped to the host user running the container (eg. uid 1001).
This should work and is IMO quite safe. The user is not root on the host.
A similar approach has been added to syzkaller.
Downsides:
mambauser
has root access inside the container and could hence to quite some stuff (e.g. accessing other users, creating new ones. One potential risk is that it could create additional users inside the container that are crafted to be mapped onto existing ones on the host (Example: There is a user with uid=3000 with sudo rights on the host, we create a new user with uid=1500 inside the container - if /etc/subuid/
allows mapping to uid 3000..., then we can do what that user can do on the host. I did not test this, but think it is possible.).Credits:
In my py4docker project, I will likely add this to run_script.sh
as a quick and portable fix with checks for the aforementioned conditions.
As for micromamba-docker
, it is probably best to add a section to the documentation. Again, the WWW is full of desperate ;-) people facing similar problems with volume mounts, non-root users, and rootless Docker ;-). My fear is most of them will simply give up and run everything as root :-(.
See also my answer in https://stackoverflow.com/a/78412890/516699.
Option 2: Mimic --userns=keep-id
on Docker by tweaking /etc/subuid
and using a dedicated non-root user on the host (likely the best approach, not yet tested)
I think I do now understand what happens when rootlesskit
uses /etc/subuid
(and ...gid for the group accordingly):
/etc/subuid
for the host user. Example: /etc/subuid
contains
user:10000:65536
This says that for the user user
on the host, all UID > 0 can / will be mapped to the UID range from 10000 to 10000+65536, like so
UID on container --> UID on host
1 -> 10000
2 -> 10001
...
You can debug the mapping rules from container to host like so :
$ rootlesskit bash
# cat /proc/self/uid_map
0 1000 1
1 100000 65536
# cat /proc/self/gid_map
0 1000 1
1 100000 65536
Possible Solution:
mambauser_host
on the host for running rootless Docker and the micromamba-docker
image with a UID of 57439 (the internal UID of mambauser
). It could be any other sufficiently large UID, but this way it's more transparent.
sudo groupadd -g 57439 mambauser_host
sudo useradd mambauser_host -u 57439 -g 57439 -m -s /bin/bash
sudo paddwd mambauser_host
/etc/subuid
and /etc/subgid
so that the UID and GUID of mambauser
inside the container will map to those on the host. This requires a bit of thinking ;-): We want the UID on the host to be 57439. The UID inside the container should be > 1000, e.g. 1002. In order to map 1002 in the container to 57439 on the host for the user mambauser_host
, we need to add
mambauser_host:56437:1002
to /etc/subuid
and /etc/subgid
on the host system (not your workstation if using ssh
!):
sudo nano /etc/subuid
# Insert this line
mambauser_host:56437:1002
# Save and exit
sudo nano /etc/subgid
# Insert this line
mambauser_host:56437:1002
# Save and exit
# Restart system for the changes to take effect
# There might be a more elegant way
Notes:
If we then run the container with
docker run --rm -it -v "$(pwd):/tmp" --user 1002:1002 mambaorg/micromamba:1.5.8 /bin/bash
the UIDs starting from 1 will be mapped to the UID range starting at 56437, like to
UID 1 -> 56437
UID 2 -> 56438
...
UID 1002 -> 57439
Et voilà!
I have not yet tested this, but wanted to save the intermediate status. **This should basically do what --userns=keep-id:uid=57439,gid=57439
would do on Podman.
Details and Credits: https://rootlesscontaine.rs/how-it-works/userns/
Option 3: Use Podman with --userns=keep-id:uid=57439,gid=57439
# Note that we do not use the --user option, so mambauser remains 57439:57439!
podman run --rm -it -v "$(pwd):/tmp" --userns=keep-id:uid=57439,gid=57439 mambaorg/micromamba:1.5.8 /bin/bash
This is untested as I have no Podman installation at hand, but likely the cleanest approach.
Option 4: Disable user namespace with Docker in rootless mode
In theory, it might be possible to use --userns=host
with Docker and disable the user namespace mechanism, like so:
docker run --rm -it -v "$(pwd):/tmp" --userns=host mambaorg/micromamba:1.5.8 /bin/bash
But it does not work, at least on my test server, and there are a bunch of Docker issues related to this, e.g. this one.
It might also introduce new security issues.
Hope you find this useful :-)! The problem has for long been a huge blocker in my workflow and I would have loved to avoid digging so deeply into this ;-). And of course, the underlying challenges are not specific to micromamba-docker
; it's just that in here the non-root user makes the default recipes for mounting host volumes for write-access with a rootless Docker installation fail.
Addendum:
subuid
and subgid
with an editor, you can better usermod
like so (from here):sudo usermod --add-subuids <start>-<end> <username>
subuid
and subgid
. This makes it a bit less portable - but then again, it is conceptually clean and IMO the most secure option despite using Podman
.Update:
subuid
) does not seem to work, because at least in Debian, there is a conflict when mapping the container user to the UID of the host user.--userns=keep-id
in its source-code.1. The best solution is to install Podman
, which is super-simple, and to use the --userns=keep-id
option. As the additional uid/gid parameters described in the current version are not available in e.g. Podman 3.0.1, you best use this command:
podman run --rm -it -v "$(pwd):/tmp" --userns=keep-id --user $(id -u):$(id -g) docker.io/mambaorg/micromamba:1.5.8 /bin/bash
In there, touch testfile.txt
works like a charm, and the external username and id is also visible from inside.
2. The second-best solution is to use --user root
when starting the container *if and only if Docker is running in rootless mode and if the host user has no sudo rights.***. This can be a quick fix with acceptable risks in most cases.
docker run --rm -it -v "$(pwd):/tmp" --user root mambaorg/micromamba:1.5.8 /bin/bash
touch testfile.txt
works, too, and the container root user is mapped to the local user on the host system.
The other options do not work; tweaking the subuid
and subgid
mapping does not work without brittle and intransparent modifications on the host system.
Hope this is useful for many of you! Frankly, it was a nightmare to get to this solution and insight and I learned many things that are intellectually interesting but were not on my bucket-list to master ;-).
Most important to say: micromamba-docker
is not all all to blame that it was such a painful journey; it is Docker's insufficient documentation and the lack of --userns=host
and/or --userns=keep-id
support; deeply hidden in Docker issues and other sources. The hundreds of related posts and discussions indicates that many, many others are wasting their life-time with this Docker problem. So please spread the word.
TODO: Add summary and pointer to this to the documentation.
@mfhepp thanks for the detailed investigation and clear recommendations. I am certainly in support of increasing our documentation in this area. Do you have any interest in putting together a documentation PR?
@mfhepp thanks for the detailed investigation and clear recommendations. I am certainly in support of increasing our documentation in this area. Do you have any interest in putting together a documentation PR?
Much appreciated! Will try to send a PR when I will have done my homework on py4docker!
If a script wants to write to the current working directory on the host system, an obvious way is to use a bind mount to map a directory on the host to a directory inside the container.
Unfortunately, this does not work with Docker on Linux systems; the non-root
mambauser
cannot write to directories from bind mounts, no matter if we set the UID/GID to that of the user on the host or not:Writing to Docker bind volumes on Linux systems as non-root users is a well-known and complicated topic, but I wonder if there is an elegant way of adding the
mambauser
to the group that has write access to a bind mount point.Or is there any other way of writing to a directory on the host from the
mambauser
?Note: The issue does not appear on Docker Desktop for OSX, as the built-in VM maps between the host system and the Docker environment.
References:
Addendum: Tested with Micromamba:1.5.6, Docker version 24.0.7, build afdd53b on Debian 11.8 Docker version 24.0.7, build afdd53b