Docker engine fails to start in rootless mode

[Maronato]

The runner keeps waiting for the engine to start, but it throws failed to start the child: fork/exec /proc/self/exe: invalid argument

compose.yml

  gitea-runner:
    image: vegardit/gitea-act-runner:dind-rootless-0.2.6
    privileged: true
    networks:
      - gitea-runner
    volumes:
      - ${PWD}/misc/gitea/runner/:/data:rw
    environment:
      GITEA_INSTANCE_URL: https://git.${DOMAIN}
      GITEA_RUNNER_REGISTRATION_TOKEN: <token>
      GITEA_RUNNER_NAME: gitea-runner
      GITEA_RUNNER_FETCH_INTERVAL: 5s
      GITEA_RUNNER_FETCH_TIMEOUT: 5s
      GITEA_RUNNER_MAX_PARALLEL_JOBS: 1
      GITEA_RUNNER_UID: 1000
      GITEA_RUNNER_GID: 1000
    restart: on-failure:20
    depends_on:
      gitea:
        condition: service_healthy
    cpu_shares: 1024
    mem_limit: 4g

docker.log

+ [ -w /data/.docker/run ]
+ [ -d /data ]
+ rootlesskit=
+ command -v docker-rootlesskit
+ command -v rootlesskit
+ rootlesskit=rootlesskit
+ break
+ [ -z rootlesskit ]
+ : /data/.docker/run/dockerd-rootless
+ : 
+ : 
+ : builtin
+ : auto
+ : auto
+ net=
+ mtu=
+ [ -z  ]
+ command -v slirp4netns
+ + slirp4netns --help
grep -qw -- --netns-type
+ net=slirp4netns
+ [ -z  ]
+ mtu=65520
+ [ -z slirp4netns ]
+ [ -z 65520 ]
+ dockerd=dockerd
+ [ -z  ]
+ _DOCKERD_ROOTLESS_CHILD=1
+ export _DOCKERD_ROOTLESS_CHILD
+ id -u
+ [ 1000 = 0 ]
+ command -v selinuxenabled
+ exec rootlesskit --state-dir=/data/.docker/run/dockerd-rootless --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave /usr/bin/dockerd-rootless.sh -p /data/.docker/run/docker.pid
time="2024-03-10T08:10:16Z" level=warning msg="[rootlesskit:parent] The host root filesystem is mounted as \"master:1\". Setting child propagation to \"rslave\" is not supported."
[rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: invalid argument

stdout

gitea-runner-1  |    _____ _ _                            _     _____
gitea-runner-1  |   / ____(_) |                 /\       | |   |  __ \
gitea-runner-1  |  | |  __ _| |_ ___  __ _     /  \   ___| |_  | |__) |   _ _ __  _ __   ___ _ __
gitea-runner-1  |  | | |_ | | __/ _ \/ _` |   / /\ \ / __| __| |  _  / | | | '_ \| '_ \ / _ \ '__|
gitea-runner-1  |  | |__| | | ||  __/ (_| |  / ____ \ (__| |_  | | \ \ |_| | | | | | | |  __/ |
gitea-runner-1  |   \_____|_|\__\___|\__,_| /_/    \_\___|\__| |_|  \_\__,_|_| |_|_| |_|\___|_|
gitea-runner-1  | 
gitea-runner-1  | GIT_REPO:    https://github.com/vegardit/docker-gitea-act-runner
gitea-runner-1  | GIT_BRANCH:  main
gitea-runner-1  | GIT_COMMIT:  6edad42 @ 2024-01-23 11:47:32 UTC
gitea-runner-1  | IMAGE_BUILD: 2024-03-06T17:11:14Z
gitea-runner-1  | 
gitea-runner-1  | 2024-03-10 08:20:14 INFO [/opt/run.sh:26] act_runner version v0.2.6
gitea-runner-1  | 2024-03-10 08:20:14 INFO [/opt/run.sh:27] Timezone: UTC +0000
gitea-runner-1  | 2024-03-10 08:20:14 INFO [/opt/run.sh:28] Hostname: 1b75354f5835
gitea-runner-1  | 2024-03-10 08:20:14 INFO [/opt/run.sh:29] IP Addresses: 
gitea-runner-1  |  - <ip>
gitea-runner-1  | 2024-03-10 08:20:14 INFO [/opt/run.sh:31] Config environment variables: 
gitea-runner-1  |  - GITEA_INSTANCE_URL=https://git.<domain>
gitea-runner-1  |  - GITEA_RUNNER_CONFIG_TEMPLATE_FILE=/opt/config.template.yaml
gitea-runner-1  |  - GITEA_RUNNER_FETCH_INTERVAL=5s
gitea-runner-1  |  - GITEA_RUNNER_FETCH_TIMEOUT=5s
gitea-runner-1  |  - GITEA_RUNNER_GID=1000
gitea-runner-1  |  - GITEA_RUNNER_LABELS=
gitea-runner-1  |  - GITEA_RUNNER_LABELS_DEFAULT=ubuntu-latest:docker://catthehacker/ubuntu:runner-22.04,ubuntu-22.04:docker://catthehacker/ubuntu:runner-22.04,ubuntu-20.04:docker://catthehacker/ubuntu:runner-20.04
gitea-runner-1  |  - GITEA_RUNNER_MAX_PARALLEL_JOBS=1
gitea-runner-1  |  - GITEA_RUNNER_NAME=gitea-runner
gitea-runner-1  |  - GITEA_RUNNER_REGISTRATION_RETRY_INTERVAL=5s
gitea-runner-1  |  - GITEA_RUNNER_REGISTRATION_TIMEOUT=30
gitea-runner-1  |  - GITEA_RUNNER_REGISTRATION_TOKEN=******
gitea-runner-1  |  - GITEA_RUNNER_UID=1000
gitea-runner-1  | 2024-03-10 08:20:14 INFO [/opt/run.sh:41] Starting Docker engine (rootless)...
gitea-runner-1  | 2024-03-10 08:20:15 INFO [/opt/run.sh:56] Waiting for Docker engine to start...
gitea-runner-1  | [rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: invalid argument
gitea-runner-1  | 2024-03-10 08:20:17 INFO [/opt/run.sh:56] Waiting for Docker engine to start...
gitea-runner-1  | [rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: invalid argument
gitea-runner-1  | 2024-03-10 08:20:19 INFO [/opt/run.sh:56] Waiting for Docker engine to start...
gitea-runner-1  | [rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: invalid argument
gitea-runner-1  | 2024-03-10 08:20:21 INFO [/opt/run.sh:56] Waiting for Docker engine to start...
gitea-runner-1  | [rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: invalid argument
gitea-runner-1  | 2024-03-10 08:20:23 INFO [/opt/run.sh:56] Waiting for Docker engine to start...
gitea-runner-1  | [rootlesskit:parent] error: failed to start the child: fork/exec /proc/self/exe: invalid argument

[sebthom]

I cannot reproduce this in our environment. Only issue I found for failed to start the child: fork/exec /proc/self/exe: invalid argument is https://github.com/moby/buildkit/issues/1188 from 2019. Maybe you find anything relevant there that fixes the issue. If so please report back!

[sebthom]

You may have to set sysctl -w kernel.unprivileged_userns_clone=1 on the host system. see https://github.com/rootless-containers/rootlesskit?tab=readme-ov-file#sysctl