Rootless Docker support (`container init caused: write sysctl key net.ipv4.ping_group_range: write /proc/sys/net/ipv4/ping_group_range: invalid argument: unknown`)

[igorcalabria]

Is there support for using buildx command with a rootless docker daemon? I tried the default buildkit image and the rootless one but both raises this error while booting up buildkit:

error: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: write sysctl key net.ipv4.ping_group_range: write /proc/sys/net/ipv4/ping_group_range: invalid argument: unknown

As a workaround, I found out that manually running buildkit with rootless config worked

docker run --name buildx_buildkit_builder0 -d --security-opt seccomp=unconfined --security-opt apparmor=unconfined --device /dev/fuse moby/buildkit:rootless --oci-worker-no-process-sandbox

You just gotta make sure that the name of the container matches the name of the buildx builder. In this case, my builder was named "builder" so the buildkit container had to be named "buildx_buildkit_builder0"

[tonistiigi]

I found out that manually running buildkit with rootless

@igorcalabria did you test what was the difference. Was it just rootless image, --privileged or oci-worker-no-process-sandbox. First and third can be set with docker buildx flags.

[igorcalabria]

@tonistiigi I dug a little deeper and found out that the issue is none of those things. Diffing the output of docker inspect for the buildkit container created by buildx and the container created by myself I narrowed the issue down to the --userns=host flag used by buildx. It triggers the same error when buildx creates the container:

igor@desk:~$ docker run --rm --name buildx_buildkit_builder0 --privileged --userns=host moby/buildkit:buildx-stable-1
docker: Error response from daemon: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: write sysctl key net.ipv4.ping_group_range: write /proc/sys/net/ipv4/ping_group_range: invalid argument: unknown.

Without the flag, it seems to run fine(including pushing images with buildx):

igor@desk:~$ docker run -d --privileged --name buildx_buildkit_builder0 moby/buildkit:buildx-stable-1
e8dac3ddc06f83d0ba1dd1af08cbf8a5a2ac23e6e1bafb8b20d747c369d1595d
igor@desk:~$ docker buildx inspect --bootstrap
Name:   builder
Driver: docker-container

Nodes:
Name:      builder0
Endpoint:  unix:///run/user/1000/docker.sock
Status:    running
Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
igor@desk:~$ docker logs e8dac3ddc06f83d0ba1dd1af08cbf8a5a2ac23e6e1bafb8b20d747c369d1595d
time="2021-03-19T13:16:55Z" level=info msg="auto snapshotter: using native"
time="2021-03-19T13:16:55Z" level=info msg="found worker \"uyr2y601gmkbul2pcuyzkzwft\", labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:e8dac3ddc06f org.mobyproject.buildkit.worker.snapshotter:native], platforms=[linux/amd64 linux/arm64 linux/riscv64 linux/ppc64le linux/s390x linux/386 linux/arm/v7 linux/arm/v6]"
time="2021-03-19T13:16:55Z" level=warning msg="rootless mode is not supported for containerd workers. disabling containerd worker."
time="2021-03-19T13:16:55Z" level=info msg="found 1 workers, default=\"uyr2y601gmkbul2pcuyzkzwft\""
time="2021-03-19T13:16:55Z" level=warning msg="currently, only the default worker can be used."
time="2021-03-19T13:16:55Z" level=info msg="running server on /run/buildkit/buildkitd.sock"
igor@desk:~$ 

So maybe we just need a flag to disable passing --userns=host when running the buildkit container against rootless docker. Just a little disclaimer that this may break a ton of things that I didn't test, I'm not that familiar with docker and buildx to be confident of this workaround

[Erisa]

In my case at least, the exact same error appears to be happening when attempting to run docker buildx from inside an LXD container. In this case, buildx is ran as root and the LXD container is given security.nesting=true which works fine for non-buildx things.

The same suggestion (Creating the builder without --userns=host) also fixes it. A buildx flag would definitely be appreciated for this case. Provided it doesn't break anything else.

[tonistiigi]

@AkihiroSuda This works now, right?

[aarongable]

I just did a clean install of docker rootless on Ubuntu (got version 20.10.11) and this still fails.

[+] Building 7.7s (1/1) FINISHED                                                     
 => ERROR [internal] booting buildkit                                                                                                                                 7.7s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                                                    7.2s
 => => creating container buildx_buildkit_cross0                                                                                                                      0.5s
------                                                                               
 > [internal] booting buildkit:                                                      
------                                    
error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: write
 sysctl key net.ipv4.ping_group_range: write /proc/sys/net/ipv4/ping_group_range: invalid argument: unknown

[AkihiroSuda]

Both docker and docker-container drivers seems to work for me

$ $ docker info
Client:
 Context:    rootless
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.7.0-26-g202e9969)
  compose: Docker Compose (Docker Inc., v2.0.1)

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 4
 Server Version: dev
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: true
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1e5ef943eb76627a6d3b6de8cd1ef6537f393a71
 runc version: v1.0.3-0-gf46b6ba2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  rootless
  cgroupns
 Kernel Version: 5.13.0-22-generic
 Operating System: Ubuntu 21.10
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.6GiB
 Name: suda-ws01
 ID: CWVR:KJQU:3CNT:IJF7:FMME:22Y7:GKFW:AFKJ:IVLQ:JOVW:3KZY:S25M
 Docker Root Dir: /home/suda/.local/share/docker
 Debug Mode: false
 Username: akihirosuda
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

$ docker version
Client:
 Version:           20.10.2-327-ga45974a46.m
 API version:       1.42
 Go version:        go1.16.11
 Git commit:        a45974a46
 Built:             Mon Dec 13 12:40:52 2021
 OS/Arch:           linux/amd64
 Context:           rootless
 Experimental:      true

Server:
 Engine:
  Version:          dev
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.17.4
  Git commit:       8955d8da89
  Built:            Mon Dec 13 12:39:07 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.5.8
  GitCommit:        1e5ef943eb76627a6d3b6de8cd1ef6537f393a71
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ docker buildx inspect foo
Name:   foo
Driver: docker-container

Nodes:
Name:      foo0
Endpoint:  rootless
Status:    running
Platforms: linux/amd64, linux/386

[AkihiroSuda]

@aarongable Could you try the master branch?

[aarongable]

Here's logs from my most recent attempt, after updating everything I could.

$ docker info
Client:                                                                                                                  
 Context:    default                                        
 Debug Mode: false                                          
 Plugins:                                                   
  app: Docker App (Docker Inc., v0.9.1-beta3)               
  buildx: Docker Buildx (Docker Inc., v0.7.1-docker)
  scan: Docker Scan (Docker Inc., v0.12.0)                  

Server:                                                     
 Containers: 0                
  Running: 0                                                
  Paused: 0                                                 
  Stopped: 0                                                
 Images: 6                                                  
 Server Version: 20.10.12                                   
 Storage Driver: overlay2    
  Backing Filesystem: extfs                                                                                              
  Supports d_type: true                                                                                                  
  Native Overlay Diff: false                                
  userxattr: true                                                                                                        
 Logging Driver: json-file                                                                                               
 Cgroup Driver: systemd                                                                                                  
 Cgroup Version: 2                                          
 Plugins:                                                   
  Volume: local                                             
  Network: bridge host ipvlan macvlan null overlay                                                                       
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog                                    
 Swarm: inactive                                                                                                         
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc                                                                                                        
 Default Runtime: runc                                      
 Init Binary: docker-init                                                                                                
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d                                                            
 runc version: v1.0.2-0-g52b36a2                            
 init version: de40ad0                                                                                                   
 Security Options:                                                                                                       
  seccomp                                                                                                                
   Profile: default                                                                                                      
  rootless                                                                                                               
  cgroupns                                                                                                               
 Kernel Version: 5.15.5-76051505-generic                                                                                 
 Operating System: Ubuntu 21.10                                                                                           
 OSType: linux                                              
 Architecture: x86_64                                       
 CPUs: 24                                                   
 Total Memory: 31.36GiB 
 Name: callisto                                                                                                          
 ID: SKP2:MDO2:TKSS:CC53:GS4T:DWSZ:JFUR:CTYU:NFVS:W67R:ELVO:235H                                                                                                            
 Docker Root Dir: /home/aaron/.local/share/docker                                     
 Debug Mode: false                                          
 Username: agable                                           
 Registry: https://index.docker.io/v1/                      
 Labels:                      
 Experimental: false                                        
 Insecure Registries:                                       
  127.0.0.0/8                                               
 Live Restore Enabled: false

$ docker buildx create --use --name=cross                   
cross

$ docker buildx build .                                                                                                  
WARN[0000] No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image
 into docker use --load                                                                                                  
[+] Building 1.5s (1/1) FINISHED                                                                                         
 => ERROR [internal] booting buildkit                                                                               1.5s 
 => => pulling image moby/buildkit:buildx-stable-1                                                                  1.1s                                                    
 => => creating container buildx_buildkit_cross0                                                                    0.4s                                                    
------                                                      
 > [internal] booting buildkit:                                                                                          
------                                                                                                                   
error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: write sysctl key net.ipv4.ping_group_range: write /proc/sys/net/ipv4/ping_group_range: invalid argument: unknown

Unfortunately, it appears that I can't build docker from source on the master branch, as it uses buildx as part of its own build, and runs into the same error:

$ cd docker-ce
$ make static DOCKER_BUILD_PKGS=static-linux
make VERSION=20.10.0-beta1 CLI_DIR=/home/aaron/Code/docker-ce/components/cli ENGINE_DIR=/home/aaron/Code/docker-ce/components/engine -C /home/aaron/Code/docker-ce/components/packaging static
make[1]: Entering directory '/home/aaron/Code/docker-ce/components/packaging'
mkdir -p "src/github.com/docker"
cp -r "/home/aaron/Code/docker-ce/components/cli" src/github.com/docker/cli
./scripts/checkout.sh src/github.com/docker/cli "HEAD"
+ SRC=src/github.com/docker/cli
+ REF=HEAD
+ REF_FETCH=HEAD
+ git -C src/github.com/docker/cli ls-remote --refs --heads --tags origin HEAD
+ awk {print $2}
+ REF=
+ [ -n  ]
+ REF=FETCH_HEAD
+ git -C src/github.com/docker/cli fetch --update-head-ok --depth 1 origin HEAD
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
From ssh://github.com/docker/docker-ce
 * branch                  HEAD       -> FETCH_HEAD
+ git -C src/github.com/docker/cli checkout -q FETCH_HEAD
mkdir -p "src/github.com/docker"
cp -r "/home/aaron/Code/docker-ce/components/engine" src/github.com/docker/docker
./scripts/checkout.sh src/github.com/docker/docker "HEAD"
+ SRC=src/github.com/docker/docker
+ REF=HEAD
+ REF_FETCH=HEAD
+ git -C src/github.com/docker/docker ls-remote --refs --heads --tags origin HEAD
+ awk {print $2}
+ REF=
+ [ -n  ]
+ REF=FETCH_HEAD
+ git -C src/github.com/docker/docker fetch --update-head-ok --depth 1 origin HEAD
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
From ssh://github.com/docker/docker-ce
 * branch                  HEAD       -> FETCH_HEAD
+ git -C src/github.com/docker/docker checkout -q FETCH_HEAD
git init src/github.com/docker/scan-cli-plugin
Initialized empty Git repository in /home/aaron/Code/docker-ce/components/packaging/src/github.com/docker/scan-cli-plugin/.git/
git -C src/github.com/docker/scan-cli-plugin remote add origin "https://github.com/docker/scan-cli-plugin.git"
./scripts/checkout.sh src/github.com/docker/scan-cli-plugin "v0.10.0"
+ SRC=src/github.com/docker/scan-cli-plugin
+ REF=v0.10.0
+ REF_FETCH=v0.10.0
+ git -C src/github.com/docker/scan-cli-plugin ls-remote --refs --heads --tags origin v0.10.0
+ awk {print $2}
+ REF=refs/tags/v0.10.0
+ [ -n refs/tags/v0.10.0 ]
+ REF_FETCH=refs/tags/v0.10.0:refs/tags/v0.10.0
+ git -C src/github.com/docker/scan-cli-plugin fetch --update-head-ok --depth 1 origin refs/tags/v0.10.0:refs/tags/v0.10.0
remote: Enumerating objects: 71, done.
remote: Counting objects: 100% (71/71), done.
remote: Compressing objects: 100% (64/64), done.
remote: Total 71 (delta 8), reused 30 (delta 1), pack-reused 0
Unpacking objects: 100% (71/71), 111.10 KiB | 669.00 KiB/s, done.
From https://github.com/docker/scan-cli-plugin
 * [new tag]         v0.10.0    -> v0.10.0
+ git -C src/github.com/docker/scan-cli-plugin checkout -q refs/tags/v0.10.0
for p in static-linux; do \
        make -C static VERSION=20.10.0-beta1 GO_VERSION=1.16.9 TARGETPLATFORM= CONTAINERD_VERSION= RUNC_VERSION= ${p}; \
done
make[2]: Entering directory '/home/aaron/Code/docker-ce/components/packaging/static'
cd /home/aaron/Code/docker-ce/components/cli && VERSION=20.10.0-beta1 docker buildx bake --set binary.platform= --set binary.args.CGO_ENABLED= binary
[+] Building 1.6s (1/1) FINISHED                                                                                                                                            
 => ERROR [internal] booting buildkit                                                                                                                                  1.6s
 => => pulling image moby/buildkit:buildx-stable-1                                                                                                                     1.1s
 => => creating container buildx_buildkit_cross0                                                                                                                       0.5s
------
 > [internal] booting buildkit:
------
error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: write sysctl key net.ipv4.ping_group_range: write /proc/sys/net/ipv4/ping_group_range: invalid argument: unknown
make[2]: *** [Makefile:97: static-cli] Error 1
make[2]: Leaving directory '/home/aaron/Code/docker-ce/components/packaging/static'
make[1]: *** [Makefile:81: static] Error 2
make[1]: Leaving directory '/home/aaron/Code/docker-ce/components/packaging'
make: *** [Makefile:30: static] Error 2

[aarongable]

Aha, I figured out how to build just the buildx plugin from source and install it:

$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.7.0-30-g33c121df)
  scan: Docker Scan (Docker Inc., v0.12.0)
<snip>

$ docker buildx version
github.com/docker/buildx v0.7.0-30-g33c121df 33c121df01e3a7187fecf67445da74ab4bb8668c

Unfortunately it doesn't seem to have changed the situation:

$ docker buildx build .                                 
WARN[0000] No output specified for docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load 
[+] Building 0.3s (1/1) FINISHED                                                                
 => ERROR [internal] booting buildkit                                                      0.3s
 => => starting container buildx_buildkit_cross0                                           0.3s
------
 > [internal] booting buildkit:
------
error: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: write sysctl key net.ipv4.ping_group_range: write /proc/sys/net/ipv4/ping_group_range: invalid argument: unknown

[AkihiroSuda]

This line seems causing the issue with Rootless Docker 20.10 (but not with moby/moby@master) https://github.com/docker/buildx/blob/1927dba42fab34e572771967781e98d2dd4de630/driver/docker-container/driver.go#L115

$ docker run -d --privileged  moby/buildkit:v0.9.3
7a3c5ca457c0f048790c491874aa442ce5c53f42c7f7b7781a569ee387ad9019

$ docker run -d --privileged --userns=host moby/buildkit:v0.9.3
2ccb580191eba64bd57cf409db062b1655dd6e53d1bdd80b2e894f69cd83b479
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: write sysctl key net.ipv4.ping_group_range: write /proc/sys/net/ipv4/ping_group_range: invalid argument: unknown.

(Tested with clean installation of Ubuntu 21.10 with Docker 20.10.12)

[AkihiroSuda]

Should be fixed in https://github.com/moby/moby/pull/43084 (target: Docker 20.10.13)