search

jiayihu / fedra-thesis

Next-generation system for Computing Continuum via WebAssembly (WASM)
7 stars 2 forks source link

Mounting containerd filesystem on k3s Alpine #37

Closed jiayihu closed 3 years ago

jiayihu commented 3 years ago

K3s on Alpine is giving the following error when mounting a fs for a container. Krustlet is not having the issue when mounting volumes.

failed: rpc error: code = Unknown desc = failed to create containerd task: failed to mount rootfs component &{overlay overlay [workdir=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/29/work upperdir=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/29/fs lowerdir=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1/fs]}: invalid argument: unknown
jiayihu commented 3 years ago

K3s doesn't seem to work nicely with Overlay filesystem: https://github.com/k3s-io/k3s/issues/19#issuecomment-461934494

On the other hand, it seems the error happens also with rootless k3s and the support for rootless has improved in latest k3s. But latest k3s isn't working #36

Trying --snapshotter native also requires latest k3s

jiayihu commented 3 years ago

The issue is also likely caused by the overlay filesystem not being ext https://github.com/k3s-io/k3s/issues/495#issuecomment-496366229

jiayihu commented 3 years ago

Updating to v1.19.5+k3s2 and using --snapshotter native works, although there is now the issue invalid capacity 0 on image filesystem, which presented even back with Ubuntu Server. Probably caused by the slow SD cards?

E1227 21:05:03.699917    2384 cri_stats_provider.go:376] Failed to get the info of the filesystem with mountpoint "/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.native": unable to find data in memory cache.
E1227 21:05:03.700114    2384 kubelet.go:1218] Image garbage collection failed once. Stats initialization may not have completed yet: invalid capacity 0 on image filesystem
jiayihu commented 3 years ago

Fixed by reinstalling Alpine with persistent ext fs since nested overlays don't play nice with containerd https://github.com/k3s-io/k3s/issues/2755