search

nestybox / sysbox

An open-source, next-generation "runc" that empowers rootless containers to run workloads such as Systemd, Docker, Kubernetes, just like VMs.
Apache License 2.0
2.76k stars 151 forks source link

Error when attempting to use Docker with Sysbox on ZFS #849

Open nhoefer2 opened 1 week ago

nhoefer2 commented 1 week ago

Using sysbox-ce_0.6.4-0.linux_amd64.deb

lsb_release -ar

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 24.04.1 LTS
Release:        24.04
Codename:       noble

uname -a Linux gil 6.8.0-45-generic #45-Ubuntu SMP PREEMPT_DYNAMIC Fri Aug 30 12:02:04 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

running the following command gives this error docker run --rm -d --runtime=sysbox-runc hello-world

96a24135a6eb0c92c892f3b39a2a1f7a955b775f3aba15211a2ae40485ddd5ab
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: container_linux.go:439: starting container process caused: process_linux.go:608: container init caused: process_linux.go:579: handleReqOp caused: rootfs_init_linux.go:325: chown overlayfs upper layet at %!s(MISSING) caused: failed to shift ACL for /main_pool/docker/overlay2/bb3273127fbba4cce77751c983fb08a2b077a7f55c398d21a3e208335c79580b/diff: failed to get ACL for /main_pool/docker/overlay2/bb3273127fbba4cce77751c983fb08a2b077a7f55c398d21a3e208335c79580b/diff: operation not supported: unknown.

I haven't the slightest idea on what's causing this or how to diagnose and resolve it. Any help would be greatly appreciated.

nhoefer2 commented 1 week ago

While my error is not explicitly mentioned here in the troubleshooting, I managed to find this: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/troubleshoot.md#sysbox-logs

Failed to Setup Docker Volume Manager Error When creating a system container, Docker may report the following error:

docker run --runtime=sysbox-runc -it ubuntu:latest docker: Error response from daemon: OCI runtime create failed: failed to setup docker volume manager: host dir for docker store /var/lib/sysbox/docker can't be on ..." This means that Sysbox's /var/lib/sysbox directory is on a filesystem not supported by Sysbox.

This directory must be on one of the following filesystems:

ext4 btrfs The same requirement applies to the /var/lib/docker directory.

This is normally the case for vanilla Ubuntu installations, so this error is not common.

I should mention that my docker data directory is on a folder in a zfs pool.

I setup my system so all data is on the zfs pool which is completely isolated from the OS disk (which has VERY limited capacity). Am I shit out of luck if I'm trying to use zfs? Will this ever be supported in the future?

nhoefer2 commented 1 week ago

After countless hours of banging my head into the wall I finally figured it out. Sysbox requires POSIX ACLs on the filesystem which can be enabled on zfs using the following commands

zfs set acltype=posixacl poolname/datasetname
zfs set xattr=sa poolname/datasetname
umount /mountpoint
zfs mount poolname

May I suggest to the developers here to perhaps include something about this a bit more explicitly somewhere in the configuration or troubleshooting guides?

ctalledo commented 6 days ago

Thanks @nhoefer2 for trying Sysbox and figuring out the problem with running Docker + Sysbox on ZFS (i.e., Posix ACLs need to be enabled).

I am actually surprised Posix ACLs are not enabled by default on ZFS, given that it's the standard.

Let's keep this issue open in case someone else bumps into the same problem. I've renamed the title based on your findings.

Thanks again!