ZFS support

[phaer]

Is your feature request related to a problem? Please describe. It's unclear to me whether k3os already supports ZFS after or if this feature is on the roadmap, after skimming the docs. It seems to be supported on rancheros

Describe the solution you'd like Ideally support and documentation to use persistent storage via ZFS with k3os.

Describe alternatives you've considered Well, I could live without it or consider using RancherOS for that usecase.

Thanks for your time and work!

[metahertz]

I spent a little time back in (wow, 2017!) getting ZFS kernel modules loaded into a simple Linuxkit/moby OS build: https://metahertz.co.uk/2017/09/15/using-zfs-with-linuxkit-and-moby/

Out of interest, what are your usecases?

• The K3OS image itself installed onto a ZFS root?
• The ability to have a separate persistence disk(s) loaded via ZFS (for example, for data resilience in a single node cluster environment?)
• Something i've not thought of?

K3OS builds on linuxkit so i'm happy to take a stab at it, tying to understand the usecase!

[phaer]

Hi @matjohn2, thanks for your reply!

My use case would primarily be your second one, persistent storage via ZFS for my volumes. Installing k3os on the same zfs would be awesome but not necessarily needed.

[dweomer]

@phaer @matjohn2 To avoid licensing issues, we currently do not build/ship the ZFS kernel modules (our kernel is basically upstream Ubuntu). (See https://github.com/rancher/k3os-kernel/blob/4.15.0-72.81-rancher1/scripts/build#L16). That said, I would also like to see ZFS support with a working example.

[metahertz]

@phaer Could be as simple as running the k3os-kernel build with do_zfs=true in https://github.com/rancher/k3os-kernel/blob/master/scripts/build#L16

Would then need alpine compatible/compiled userspace binaries added to the k3os userspace image if this is the case. Will have a play with this tomorrow (as i need to understand the K3OS build process anyway, this is a good excuse!)

[liyimeng]

@dweomer

@phaer @matjohn2 To avoid licensing issues, we currently do not build/ship the ZFS kernel modules (our kernel is basically upstream Ubuntu). (See https://github.com/rancher/k3os-kernel/blob/4.15.0-72.81-rancher1/scripts/build#L16). That said, I would also like to see ZFS support with a working example.

I have a conversation with @niusmallnan regarding on this long ago. There is a bug in the building script to turn on zfs. After days of hacking, I find the solution. If rancher believe this is not an license issue anymore, I would consider to post back my solution.

[zimme]

Just wanted to post an update here that I've been able to build zfs (modules) from source during the k3os build process and adding the alpine zfs package to get the zfs userspace binaries, however I need to modprobe zfs manually on boot.

I'll try and refine the steps and try and break it out into it's own build step so the userspace binaries from the source build can be used to avoid version incompatibility between modules and userspace binaries.

[zimme]

Here's my branch (https://github.com/zimme/k3os/tree/zfs) with initial zfs support.

After you've built the iso and started the system you will need to sudo modprobe zfs to load zfs modules.

There's a lot of refinement needed before this could potentially be put up as a PR and I'm thinking that the changes being made to Makefile.in using sed should really be a PR to openzfs.

Try it out and see if it works for you and feel free to improve anything.

[krishofmans]

Great work @zimme I'm going to try this out quite soon and combine this with: @asteven 's https://github.com/asteven/zfs-provisioner

The idea would be to get https://kubernetes.io/docs/concepts/storage/volume-snapshots/ implemented using zfs snapshots behind the scenes locally. And in a later step use zfs replication to send the backups to other places.

The whole idea is that with longhorn we don't really need the HA part on the edge, and zfs underneath would already greatly support our usecases, we just need to cover the gap of creating crd's to trigger underlying zfs operations.

Since linux's mdadm is already included, how high are the chances that we can get zfs in the main k3os as well?

Would it be considered as a possible replacement of the default local path provisioner if we can work out a poc to benefit all k8s volumes using zfs providing lightweight snapshotting?

[zimme]

Just saw this issue in k3s, https://github.com/rancher/k3s/issues/66, in light of k3s not including the zfs snapshotter I'm not sure how useful it would be to include zfs in k3os?

[JOduMonT]

but how TrueNAS SCALE are able to run k3s if k3s don't support ZFS ?