Containerd aufs/devmapper/zfs snapshotters removed from May 2023 releases

[fidencio]

Some VM based workloads do not have the capability of using a shared filesystem to give the guest access to the container rootfs, which is stored on the host side.

This case described above is exactly what happens when we talk about Kata Containers using Firecracker as VMM, Kata Containers running Confidential VMs (such as Intel TDX, AMD SEV, AMD SNP, IBM SE).

Enabling this on the k3s side is an onliner, here: https://github.com/k3s-io/k3s/blob/7c0a7687c6379cafc589cd740144d6eb5928df98/pkg/containerd/builtins_linux.go#L27

Having this enabled by default would help a lot users of VM based workloads.

[brandond]

Just noting here that we did have a long series of releases that included the zfs snapshotter. That was unintentionally added when we moved to building standalone containerd, so we didn't initially notice when it went away when we moved containerd back into the multicall binary. Some folks had come to rely on its presence.

[wenerme]

I can accept migrate off zfs snapshotter to zfs vol if I know how, (zfs 2.2 support overlayfs), but before this, I hope I can use the stable version of k3s 😭

[brandond]

This will be resolved in the June releases.

[ShylajaDevadiga]

Validated using commit id b66a118362ba0f3fed981722f5d3fba68beb314a from master branch

• Plugins are being loaded on startup.

skip loading plugin due to non zfs filesystem

> sudo ctr plugin ls | grep snapshotter
io.containerd.snapshotter.v1           aufs                     linux/amd64    skip      
io.containerd.snapshotter.v1           btrfs                    linux/amd64    skip      
io.containerd.snapshotter.v1           devmapper                linux/amd64    error     
io.containerd.snapshotter.v1           native                   linux/amd64    ok        
io.containerd.snapshotter.v1           overlayfs                linux/amd64    ok        
io.containerd.snapshotter.v1           fuse-overlayfs           linux/amd64    ok        
io.containerd.snapshotter.v1           stargz                   linux/amd64    ok        
io.containerd.snapshotter.v1           zfs                      linux/amd64    skip  

> sudo grep type=io.containerd.snapshotter.v1 /var/lib/rancher/k3s/agent/containerd/containerd.log |grep plugin
time="2023-06-14T16:57:42.584757200Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.aufs\"..." type=io.containerd.snapshotter.v1
time="2023-06-14T16:57:42.587577630Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found in directory /lib/modules/5.3.18-59.37-default\\n\"): skip plugin" type=io.containerd.snapshotter.v1
time="2023-06-14T16:57:42.587607306Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." type=io.containerd.snapshotter.v1
time="2023-06-14T16:57:42.587918584Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.btrfs (xfs) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2023-06-14T16:57:42.587939794Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1
time="2023-06-14T16:57:42.587978963Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1
time="2023-06-14T16:57:42.588046552Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1
time="2023-06-14T16:57:42.588212621Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.fuse-overlayfs\"..." type=io.containerd.snapshotter.v1
time="2023-06-14T16:57:42.588285135Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.stargz\"..." type=io.containerd.snapshotter.v1
time="2023-06-14T16:57:42.590045553Z" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1
time="2023-06-14T16:57:42.590247429Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1