search

nestybox / sysbox

An open-source, next-generation "runc" that empowers rootless containers to run workloads such as Systemd, Docker, Kubernetes, just like VMs.
Apache License 2.0
2.83k stars 159 forks source link

Issues with pre-loading images in sys container #832

Closed teja156 closed 1 month ago

teja156 commented 2 months ago

Hello there, so I was following this guide https://github.com/nestybox/sysbox/blob/master/docs/quickstart/images.md#building-a-system-container-that-includes-inner-container-images--v012- to test pre-loading the sys container with some images. I used the exact same Dockerfile and docker-pull.sh script as shown in the guide.

Dockerfile:

FROM nestybox/alpine-docker

COPY docker-pull.sh /usr/bin
RUN chmod +x /usr/bin/docker-pull.sh && docker-pull.sh && rm /usr/bin/docker-pull.sh

docker-pull.sh:

#!/bin/sh

# dockerd start
dockerd > /var/log/dockerd.log 2>&1 &
sleep 2

# pull inner images
docker pull busybox:latest
docker pull alpine:latest

# dockerd cleanup (remove the .pid file as otherwise it prevents
# dockerd from launching correctly inside sys container)
kill $(cat /var/run/docker.pid)
kill $(cat /run/docker/containerd/containerd.pid)
rm -f /var/run/docker.pid
rm -f /run/docker/containerd/containerd.pid

Command used to build the image:

sudo docker build -t alpine_test:new . --progress=plain

However, during the build process, dockerd fails to start in the sys container and as a result, the images were not pre-loaded. Here are the logs /var/log/dockerd.log:

#8 2.452 time="2024-08-30T19:26:41.436897298Z" level=info msg="Starting up"
#8 2.452 time="2024-08-30T19:26:41.438264417Z" level=info msg="containerd not running, starting managed containerd"
#8 2.452 time="2024-08-30T19:26:41.439765503Z" level=info msg="started new containerd process" address=/var/run/docker/containerd/containerd.sock module=libcontainerd pid=21
#8 2.452 time="2024-08-30T19:26:41.839883876Z" level=info msg="starting containerd" revision=3a4de459a68952ffb703bbe7f2290861a75b6b67 version=v1.7.17
#8 2.452 time="2024-08-30T19:26:41.862351970Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.867377883Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: can't change directory to '/lib/modules': No such file or directory\\n\"): skip plugin" type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.867427950Z" level=info msg="loading plugin \"io.containerd.event.v1.exchange\"..." type=io.containerd.event.v1
#8 2.452 time="2024-08-30T19:26:41.867442354Z" level=info msg="loading plugin \"io.containerd.internal.v1.opt\"..." type=io.containerd.internal.v1
#8 2.452 time="2024-08-30T19:26:41.870664800Z" level=info msg="loading plugin \"io.containerd.warning.v1.deprecations\"..." type=io.containerd.warning.v1
#8 2.452 time="2024-08-30T19:26:41.870682741Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.blockfile\"..." type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.870839880Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.blockfile\"..." error="no scratch file generator: skip plugin" type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.870850084Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.871051381Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (overlay) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.871062300Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.871079980Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
#8 2.452 time="2024-08-30T19:26:41.871102723Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.871357499Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.872345490Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.872455999Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
#8 2.452 time="2024-08-30T19:26:41.872466542Z" level=info msg="loading plugin \"io.containerd.content.v1.content\"..." type=io.containerd.content.v1
#8 2.452 time="2024-08-30T19:26:41.872632169Z" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1
#8 2.452 time="2024-08-30T19:26:41.872733248Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
#8 2.452 time="2024-08-30T19:26:41.872743510Z" level=info msg="metadata content store policy set" policy=shared
#8 2.452 time="2024-08-30T19:26:41.879597214Z" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1
#8 2.452 time="2024-08-30T19:26:41.879755099Z" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1
#8 2.452 time="2024-08-30T19:26:41.879802526Z" level=info msg="loading plugin \"io.containerd.lease.v1.manager\"..." type=io.containerd.lease.v1
#8 2.452 time="2024-08-30T19:26:41.879867020Z" level=info msg="loading plugin \"io.containerd.streaming.v1.manager\"..." type=io.containerd.streaming.v1
#8 2.452 time="2024-08-30T19:26:41.879901245Z" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1
#8 2.452 time="2024-08-30T19:26:41.880354936Z" level=info msg="loading plugin \"io.containerd.monitor.v1.cgroups\"..." type=io.containerd.monitor.v1
#8 2.452 time="2024-08-30T19:26:41.881266876Z" level=info msg="loading plugin \"io.containerd.runtime.v2.task\"..." type=io.containerd.runtime.v2
#8 2.452 time="2024-08-30T19:26:41.881746701Z" level=info msg="loading plugin \"io.containerd.runtime.v2.shim\"..." type=io.containerd.runtime.v2
#8 2.452 time="2024-08-30T19:26:41.881779007Z" level=info msg="loading plugin \"io.containerd.sandbox.store.v1.local\"..." type=io.containerd.sandbox.store.v1
#8 2.452 time="2024-08-30T19:26:41.881792088Z" level=info msg="loading plugin \"io.containerd.sandbox.controller.v1.local\"..." type=io.containerd.sandbox.controller.v1
#8 2.452 time="2024-08-30T19:26:41.881801433Z" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1
#8 2.452 time="2024-08-30T19:26:41.881832120Z" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1
#8 2.452 time="2024-08-30T19:26:41.881842818Z" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1
#8 2.452 time="2024-08-30T19:26:41.881868949Z" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1
#8 2.452 time="2024-08-30T19:26:41.881892283Z" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1
#8 2.452 time="2024-08-30T19:26:41.881912524Z" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1
#8 2.452 time="2024-08-30T19:26:41.881923155Z" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1
#8 2.452 time="2024-08-30T19:26:41.881930514Z" level=info msg="loading plugin \"io.containerd.service.v1.tasks-service\"..." type=io.containerd.service.v1
#8 2.452 time="2024-08-30T19:26:41.882011295Z" level=info msg="loading plugin \"io.containerd.grpc.v1.containers\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882023020Z" level=info msg="loading plugin \"io.containerd.grpc.v1.content\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882049917Z" level=info msg="loading plugin \"io.containerd.grpc.v1.diff\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882060887Z" level=info msg="loading plugin \"io.containerd.grpc.v1.events\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882081764Z" level=info msg="loading plugin \"io.containerd.grpc.v1.images\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882092302Z" level=info msg="loading plugin \"io.containerd.grpc.v1.introspection\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882099835Z" level=info msg="loading plugin \"io.containerd.grpc.v1.leases\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882118823Z" level=info msg="loading plugin \"io.containerd.grpc.v1.namespaces\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882145355Z" level=info msg="loading plugin \"io.containerd.grpc.v1.sandbox-controllers\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882164716Z" level=info msg="loading plugin \"io.containerd.grpc.v1.sandboxes\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882175468Z" level=info msg="loading plugin \"io.containerd.grpc.v1.snapshots\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882183628Z" level=info msg="loading plugin \"io.containerd.grpc.v1.streaming\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882201888Z" level=info msg="loading plugin \"io.containerd.grpc.v1.tasks\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882223253Z" level=info msg="loading plugin \"io.containerd.transfer.v1.local\"..." type=io.containerd.transfer.v1
#8 2.452 time="2024-08-30T19:26:41.882290734Z" level=info msg="loading plugin \"io.containerd.grpc.v1.transfer\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882310534Z" level=info msg="loading plugin \"io.containerd.grpc.v1.version\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882317375Z" level=info msg="loading plugin \"io.containerd.internal.v1.restart\"..." type=io.containerd.internal.v1
#8 2.452 time="2024-08-30T19:26:41.882657991Z" level=info msg="loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." type=io.containerd.tracing.processor.v1
#8 2.452 time="2024-08-30T19:26:41.882690342Z" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="skip plugin: tracing endpoint not configured" type=io.containerd.tracing.processor.v1
#8 2.452 time="2024-08-30T19:26:41.882700267Z" level=info msg="loading plugin \"io.containerd.internal.v1.tracing\"..." type=io.containerd.internal.v1
#8 2.452 time="2024-08-30T19:26:41.882719923Z" level=info msg="skip loading plugin \"io.containerd.internal.v1.tracing\"..." error="skip plugin: tracing endpoint not configured" type=io.containerd.internal.v1
#8 2.452 time="2024-08-30T19:26:41.882728934Z" level=info msg="loading plugin \"io.containerd.grpc.v1.healthcheck\"..." type=io.containerd.grpc.v1
#8 2.452 time="2024-08-30T19:26:41.882762991Z" level=info msg="loading plugin \"io.containerd.nri.v1.nri\"..." type=io.containerd.nri.v1
#8 2.452 time="2024-08-30T19:26:41.882797884Z" level=info msg="NRI interface is disabled by configuration."
#8 2.452 time="2024-08-30T19:26:41.884176280Z" level=info msg=serving... address=/var/run/docker/containerd/containerd-debug.sock
#8 2.452 time="2024-08-30T19:26:41.884416229Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock.ttrpc
#8 2.452 time="2024-08-30T19:26:41.884623747Z" level=info msg=serving... address=/var/run/docker/containerd/containerd.sock
#8 2.452 time="2024-08-30T19:26:41.884658606Z" level=info msg="containerd successfully booted in 0.049604s"
#8 2.452 time="2024-08-30T19:26:42.531582249Z" level=error msg="failed to mount overlay: operation not permitted" storage-driver=overlay2
#8 2.452 time="2024-08-30T19:26:42.532260220Z" level=error msg="exec: \"fuse-overlayfs\": executable file not found in $PATH" storage-driver=fuse-overlayfs
#8 2.452 time="2024-08-30T19:26:42.533777746Z" level=warning msg="Unable to setup quota: operation not permitted\n"
#8 2.452 time="2024-08-30T19:26:42.536507551Z" level=info msg="Loading containers: start."
#8 2.452 time="2024-08-30T19:26:42.542072234Z" level=info msg="unable to detect if iptables supports xlock: 'iptables --wait -L -n': `iptables v1.8.10 (nf_tables): Could not fetch rule set generation id: Permission denied (you must be root)`" error="exit status 4"
#8 2.452 time="2024-08-30T19:26:42.564851708Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
#8 2.452 time="2024-08-30T19:26:42.565504283Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
#8 2.452 time="2024-08-30T19:26:42.565792447Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby

I have sysbox-runc set as the default docker runtime on the host. Docker version: 27.2.0 Sysbox version: 0.6.4 Host OS: Ubuntu 22.04.4 LTS (Ubuntu Jammy) Host Kernel Version: 6.5.0-44-generic

What am I doing wrong? Can someone guide me in the right direction? Thanks in advance.

ctalledo commented 1 month ago

Hi @teja156, thanks for opening the issue.

Try the following:

1) Set the Docker default runtime to "sysbox-runc" in /etc/docker/daemon.json as shown in the docs.

2) Use the legacy Docker builder:

sudo DOCKER_BUILDKIT=0 docker build --no-cache -t alpine_test:new .

This should work (I just tested it).

The problem you had was that Docker now uses the newer buildkit builder by default, and the process of embedding images inside Sysbox containers does not yet work with the new builder unfortunately.

I will update the Sysbox docs accordingly.

Hope that helps!