docker-library / docker

Docker Official Image packaging for Docker
Apache License 2.0
1.14k stars 582 forks source link

In `dind`, `docker pull` succeeds but `docker run` fails, or `dind-rootless` cannot be executed on Rootless Docker host. #451

Closed hytdsh closed 9 months ago

hytdsh commented 12 months ago

Hi there,

I am facing an error trying to run the dind-rootless image on a Rootless Docker host. I would like to achieve pattern 2 in the following.

Please let me know if there is any information I am missing.

dind or dind-rootless on Rootless Docker host

rootless@rootless:~$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
rootless@rootless:~$ docker version
Client: Docker Engine - Community
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:32:10 2023
 OS/Arch:           linux/amd64
 Context:           rootless

Server: Docker Engine - Community
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:32:10 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 rootlesskit:
  Version:          1.1.1
  ApiVersion:       1.1.1
  NetworkDriver:    slirp4netns
  PortDriver:       builtin
  StateDir:         /tmp/rootlesskit1591481074
 slirp4netns:
  Version:          1.2.0
  GitCommit:        656041d45cfca7a4176f6b7eed9e4fe6c11e8383
rootless@rootless:~$ docker info
Client: Docker Engine - Community
 Version:    24.0.6
 Context:    rootless
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 3
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc version: v1.1.9-0-gccaecfc
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 6.1.0-13-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 1.921GiB
 Name: rootless
 ID: 34a061b9-ae74-41b6-8211-bed2fd9e4f8f
 Docker Root Dir: /home/rootless/.local/share/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No cpuset support
WARNING: No io.weight support
WARNING: No io.weight (per device) support
WARNING: No io.max (rbps) support
WARNING: No io.max (wbps) support
WARNING: No io.max (riops) support
WARNING: No io.max (wiops) support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
rootless@rootless:~$ ls -l /var/run/user/1001/docker.sock
srw-rw---T 1 rootless 166532 0 Oct  9 18:38 /var/run/user/1001/docker.sock
rootless@rootless:~$ docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.
...(snip)...
For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Pattern 1: dind on Rootless Docker host

docker pull succeeds but docker run fails.

rootless@rootless:~/dind$ cat compose.yml
version: "3"

services:
  dind:
    image: docker:24.0.6-dind-alpine3.18
    privileged: true
rootless@rootless:~/dind$ docker compose up -d
[+] Running 2/2
 ✔ Network dind_default   Created                                                                                  0.1s
 ✔ Container dind-dind-1  Started                                                                                  0.1s
rootless@rootless:~/dind$ docker compose exec dind sh
/ #

/ # docker pull ubuntu:latest
latest: Pulling from library/ubuntu
37aaf24cf781: Pull complete
Digest: sha256:9b8dec3bf938bc80fbe758d856e96fdfab5f56c39d44b0cff351e847bb1b01ea
Status: Downloaded newer image for ubuntu:latest
docker.io/library/ubuntu:latest

/ # docker run --rm -it ubuntu:latest bash
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF: unknown.
ERRO[0000] error waiting for container:

Pattern 2: dind-rootless on Rootless Docker host

dind-rootless cannot be executed.

rootless@rootless:~/dind-rootless$ cat compose.yml
version: "3"

services:
  dind-rootless:
    image: docker:24.0.6-dind-rootless
    privileged: true
    environment:
      - DOCKER_HOST=unix:///var/run/user/1000/docker.sock
rootless@rootless:~/dind-rootless$ docker compose up -d
[+] Running 2/2
 ✔ Network dind-rootless_default            Created                                                                0.2s
 ✔ Container dind-rootless-dind-rootless-1  Started                                                                0.1s
rootless@rootless:~/dind-rootless$ docker compose ps
NAME      IMAGE     COMMAND   SERVICE   CREATED   STATUS    PORTS
rootless@rootless:~/dind-rootless$ docker compose logs dind-rootless
dind-rootless-dind-rootless-1  | Certificate request self-signature ok
dind-rootless-dind-rootless-1  | subject=CN = docker:dind server
dind-rootless-dind-rootless-1  | /certs/server/cert.pem: OK
dind-rootless-dind-rootless-1  | Certificate request self-signature ok
dind-rootless-dind-rootless-1  | subject=CN = docker:dind client
dind-rootless-dind-rootless-1  | /certs/client/cert.pem: OK
dind-rootless-dind-rootless-1  | Device "ip_tables" does not exist.
dind-rootless-dind-rootless-1  | ip_tables              36864  2 iptable_nat,iptable_filter
dind-rootless-dind-rootless-1  | x_tables               61440  9 iptable_nat,iptable_filter,xt_nat,xt_tcpudp,xt_conntrack,xt_MASQUERADE,xt_addrtype,nft_compat,ip_tables
dind-rootless-dind-rootless-1  | modprobe: can't change directory to '/lib/modules': No such file or directory
dind-rootless-dind-rootless-1  | [rootlesskit:parent] error: failed to setup UID/GID map: newuidmap 56 [0 1000 1 1 100000 65536] failed: newuidmap: write to uid_map failed: Operation not permitted
dind-rootless-dind-rootless-1  | : exit status 1

dind or dind-rootless on Rootfull Docker host (as a comparison)

root@rootfull:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@rootfull:~# docker version
Client: Docker Engine - Community
 Version:           24.0.6
 API version:       1.43
 Go version:        go1.20.7
 Git commit:        ed223bc
 Built:             Mon Sep  4 12:32:10 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.6
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.7
  Git commit:       1a79695
  Built:            Mon Sep  4 12:32:10 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
root@rootfull:~# docker info
Client: Docker Engine - Community
 Version:    24.0.6
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 3
 Server Version: 24.0.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc version: v1.1.9-0-gccaecfc
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.0-13-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 1.921GiB
 Name: rootfull
 ID: b84f2e9b-21f5-48a2-91b6-d5945b4803f4
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
root@rootfull:~# ls -l /var/run/docker.sock
srw-rw---- 1 root docker 0 Oct  9 18:47 /var/run/docker.sock
root@rootfull:~# docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.
...(snip)...
For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Pattern 3: dind on Rootfull Docker host

Success.

root@rootfull:~/dind# cat compose.yml
version: "3"

services:
  dind:
    image: docker:24.0.6-dind-alpine3.18
    privileged: true
root@rootfull:~/dind# docker compose up -d
[+] Running 2/2
 ✔ Network dind_default   Created                                                                                  0.2s
 ✔ Container dind-dind-1  Started                                                                                  0.1s
root@rootfull:~/dind# docker compose exec dind sh
/ #

/ # docker pull ubuntu:latest
latest: Pulling from library/ubuntu
37aaf24cf781: Pull complete
Digest: sha256:9b8dec3bf938bc80fbe758d856e96fdfab5f56c39d44b0cff351e847bb1b01ea
Status: Downloaded newer image for ubuntu:latest
docker.io/library/ubuntu:latest

/ # docker run --rm -it ubuntu:latest bash
root@d5856a278403:/#

root@d5856a278403:/# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Pattern 4: dind-rootless on Rootfull Docker host

Success with DOCKER_HOST env.

root@rootfull:~/dind-rootless# cat compose.yml
version: "3"

services:
  dind-rootless:
    image: docker:24.0.6-dind-rootless
    privileged: true
    environment:
      - DOCKER_HOST=unix:///var/run/user/1000/docker.sock
root@rootfull:~/dind-rootless# docker compose up -d
[+] Running 2/2
 ✔ Network dind-rootless_default            Created                                                                0.2s
 ✔ Container dind-rootless-dind-rootless-1  Started                                                                0.1s
root@rootfull:~/dind-rootless# docker compose exec dind-rootless sh
/ $

/ $ docker pull ubuntu:latest
latest: Pulling from library/ubuntu
37aaf24cf781: Pull complete
Digest: sha256:9b8dec3bf938bc80fbe758d856e96fdfab5f56c39d44b0cff351e847bb1b01ea
Status: Downloaded newer image for ubuntu:latest
docker.io/library/ubuntu:latest

/ $ docker run --rm -it ubuntu:latest bash
root@8203c3367b95:/#

root@8203c3367b95:/# cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
hytdsh commented 11 months ago

https://github.com/moby/moby/issues/46679

It appears that CAP_SETUID cannot be assigned in Rootless Docker host. Is this what it's supposed to be?

rootless@debian12:~$ ls -l /var/run/user/1001/docker.sock
srw-rw---T 1 rootless 166532 0 Oct 23 20:33 /var/run/user/1001/docker.sock
rootless@debian12:~$ git clone https://github.com/docker-library/docker.git ./dind-source
rootless@debian12:~$ cd dind-source/24/dind-rootless/
rootless@debian12:~/dind-source/24/dind-rootless$ nano Dockerfile
rootless@debian12:~/dind-source/24/dind-rootless$ git diff Dockerfile
diff --git a/24/dind-rootless/Dockerfile b/24/dind-rootless/Dockerfile
index 766214d..edbb62d 100644
--- a/24/dind-rootless/Dockerfile
+++ b/24/dind-rootless/Dockerfile
@@ -8,7 +8,7 @@ FROM docker:24-dind

 # busybox "ip" is insufficient:
 #   [rootlesskit:child ] error: executing [[ip tuntap add name tap0 mode tap] [ip link set tap0 address 02:50:00:00:00:01]]: exit status 1
-RUN apk add --no-cache iproute2 fuse-overlayfs
+RUN apk add --no-cache iproute2 fuse-overlayfs strace

 # "/run/user/UID" will be used by default as the value of XDG_RUNTIME_DIR
 RUN mkdir /run/user && chmod 1777 /run/user
rootless@debian12:~/dind-source/24/dind-rootless$ touch compose.yml
rootless@debian12:~/dind-source/24/dind-rootless$ nano compose.yml
rootless@debian12:~/dind-source/24/dind-rootless$ cat compose.yml
version: "3"

services:
  my-rootless:
    build: .
    image: my-rootless
    privileged: true
    cap_add:
      - SETUID
      - SETGID
    entrypoint: ["tail", "-f", "/dev/null"]
rootless@debian12:~/dind-source/24/dind-rootless$ docker compose build
rootless@debian12:~/dind-source/24/dind-rootless$ docker compose up -d
[+] Running 2/2
 ✔ Network dind-rootless_default          Created                                                                  0.1s
 ✔ Container dind-rootless-my-rootless-1  Started
rootless@debian12:~/dind-source/24/dind-rootless$ docker compose exec my-rootless sh
/ $

/ $ unshare -U sleep 100 &
/ $ strace newuidmap $! 0 $(id -u) 1 1 100000 65536
execve("/usr/bin/newuidmap", ["newuidmap", "12", "0", "1000", "1", "1", "100000", "65536"], 0x7ffc2ec885a8 /* 11 vars */) = 0
arch_prctl(ARCH_SET_FS, 0x7fe8d0946b48) = 0
set_tid_address(0x7fe8d0946fb8)         = 17
brk(NULL)                               = 0x556e626a3000
brk(0x556e626a5000)                     = 0x556e626a5000
mmap(0x556e626a3000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x556e626a3000
mprotect(0x7fe8d0943000, 4096, PROT_READ) = 0
mprotect(0x556e61497000, 4096, PROT_READ) = 0
poll([{fd=0, events=0}, {fd=1, events=0}, {fd=2, events=0}], 3, 0) = 0 (Timeout)
open("/proc/12/", O_RDONLY|O_LARGEFILE|O_DIRECTORY) = 3
getuid()                                = 1000
open("/etc/passwd", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 4
fcntl(4, F_SETFD, FD_CLOEXEC)           = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe8d08ac000
fcntl(4, F_SETFD, FD_CLOEXEC)           = 0
read(4, "root:x:0:0:root:/root:/bin/ash\nb"..., 1024) = 1024
read(4, "gin\nsmmsp:x:209:209:smmsp:/var/s"..., 1024) = 266
close(4)                                = 0
munmap(0x7fe8d08ac000, 4096)            = 0
fstat(3, {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
getuid()                                = 1000
open("/etc/login.defs", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
getgid()                                = 1000
getgid()                                = 1000
open("/etc/subuid", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_LARGEFILE|O_NOFOLLOW) = 4
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe8d08ac000
fcntl(4, F_SETFD, FD_CLOEXEC)           = 0
mmap(NULL, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe8d08a8000
read(4, "dockremap:165536:65536\nrootless:"..., 1024) = 45
read(4, "", 1024)                       = 0
munmap(0x7fe8d08a8000, 16384)           = 0
open("/etc/nsswitch.conf", O_RDONLY|O_LARGEFILE) = 5
read(5, "# musl itself does not support N"..., 1024) = 205
read(5, "", 1024)                       = 0
close(5)                                = 0
open("/etc/passwd", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 5
fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
read(5, "root:x:0:0:root:/root:/bin/ash\nb"..., 1024) = 1024
read(5, "gin\nsmmsp:x:209:209:smmsp:/var/s"..., 1024) = 266
close(5)                                = 0
geteuid()                               = 1000
capset({version=_LINUX_CAPABILITY_VERSION_3, pid=0}, {effective=1<<CAP_SETUID, permitted=1<<CAP_SETUID, inheritable=0}) = -1 EPERM (Operation not permitted)
writev(2, [{iov_base="newuidmap: Could not set caps\n", iov_len=30}, {iov_base=NULL, iov_len=0}], 2newuidmap: Could not set caps
) = 30
exit_group(1)                           = ?
+++ exited with 1 +++
tianon commented 10 months ago

There are definitely a lot of constraints to "rootless" mode, and I'm not sure it supports any kind of nesting, but I do admit I'm not an expert on the subject. I know running "rootless" Docker inside rootful Docker still requires privileged mode, for example.