unable to build images with gitlab runners and sysbox-runc as runtime

Description

I'm having troubles running a buildah build ... from Gitlab-CI on a self-hosted runner with docker as executor and sysbox-runc as docker runtime. Following error is printed in the jobs output:

[...]
STEP 2/3: RUN apk add --no-cache python
time="2024-08-06T15:43:41Z" level=warning msg="pkg/bind: error detaching \"/var/tmp/buildah1882152697/mnt/rootfs\": no such file or directory"
time="2024-08-06T15:43:41Z" level=warning msg="pkg/bind: error removing \"/var/tmp/buildah1882152697/mnt/rootfs\": device or resource busy"
time="2024-08-06T15:43:41Z" level=warning msg="pkg/bind: error removing \"/var/tmp/buildah1882152697/mnt\": directory not empty"
error running subprocess: bind mounting /dev from host into mount namespace: mkdir /var/tmp/buildah1882152697/mnt/rootfs/dev: file exists
Error: deleting build container "96c3a9aad937e16e174792c1ec4edbae3907f1dd469ec0e02339c7eb1d029891": replacing mount point "/var/lib/containers/storage/overlay/76ca95dda33c9e8c343ddf776b4f99b8c9d2fb17c310c59f5cca0f285d9ee100/merged": device or resource busy: building at STEP "RUN apk add --no-cache python": exit status 1

When not using sysbox but the default docker runtime, the docker image build works just fine. I'm able to reproduce it on Debian server as well as my local Arch installation. Running the gitlab-runner in privileged mode instead doesn't change anything.

Steps to reproduce the issue:

On a machine which shall be a gitlab runner, install docker, sysbox-ce and gitlab-runner
Register the gitlab-runner with your project in Gitlab

Configure the docker sysbox-runc runtime:

# cat /etc/docker/daemon.json
{
  "runtimes": {
    "sysbox-runc": {
      "path": "/usr/bin/sysbox-runc",
      "runtimeArgs": ["--no-kernel-check"]
    }
  }
}

Configure the gitlab-runner to use docker as executor with sysbox-runc as runtime

# cat /etc/gitlab-runner/config.toml
concurrent = 1
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "..."
  url = "..."
  id = 408
  token = "..."
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    MaxUploadedArchiveSize = 0
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.docker]
    tls_verify = false
    image = "docker:stable"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    network_mtu = 0
    runtime = "sysbox-runc"
    devices = ["/dev/fuse"]

Ensure the config is running

systemctl start sysbox
systemctl restart docker
systemctl restart gitlab-runner

Example Gitlab CI config .gitlab-ci.yml

stages:
  - build

build-job:
  stage: build
  image: quay.io/containers/buildah:v1.36.0
  script:
    - buildah version
    - buildah info
    - buildah build --file Dockerfile --manifest testimage .

Example Dockerfile

FROM alpine

RUN apk add --no-cache python3
RUN apk add --no-cache curl

Describe the results you received:

The build fails with the error message show in the description.

Describe the results you expected:

buildah builds the image just fine without error.

Output of buildah version (from within a gitlab job):

Version:         1.36.0
Go Version:      go1.22.3
Image Spec:      1.1.0
Runtime Spec:    1.2.0
CNI Spec:        1.0.0
libcni Version:  
image Version:   5.31.0
Git Commit:      
Built:           Mon May 27 13:11:54 2024
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

Output of buildah info (from within a gitlab job):

{
    "host": {
        "CgroupVersion": "v2",
        "Distribution": {
            "distribution": "fedora",
            "version": "40"
        },
        "MemFree": 12764295168,
        "MemTotal": 33671274496,
        "OCIRuntime": "crun",
        "SwapFree": 0,
        "SwapTotal": 0,
        "arch": "amd64",
        "cpus": 8,
        "hostname": "runner-wqxrcczgx-project-3228-concurrent-0",
        "kernel": "6.1.0-21-cloud-amd64",
        "os": "linux",
        "rootless": true,
        "uptime": "410m s",
        "variant": ""
    },
    "store": {
        "ContainerStore": {
            "number": 0
        },
        "GraphDriverName": "overlay",
        "GraphOptions": [
            "overlay.imagestore=/var/lib/shared",
            "overlay.imagestore=/usr/lib/containers/storage",
            "overlay.mount_program=/usr/bin/fuse-overlayfs",
            "overlay.mountopt=nodev,fsync=0"
        ],
        "GraphRoot": "/var/lib/containers/storage",
        "GraphStatus": {
            "Backing Filesystem": "extfs",
            "Native Overlay Diff": "false",
            "Supports d_type": "true",
            "Supports shifting": "true",
            "Supports volatile": "true",
            "Using metacopy": "false"
        },
        "ImageStore": {
            "number": 0
        },
        "RunRoot": "/run/containers/storage"
    }
}

*Output of `cat /etc/release`:**

PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Output of uname -a:

Linux hostname 6.1.0-21-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64 GNU/Linux

containers / buildah

unable to build images with gitlab runners and sysbox-runc as runtime #5669