search

helloSystem / hello

Desktop system for creators with a focus on simplicity, elegance, and usability. Based on FreeBSD. Less, but better!
2.31k stars 57 forks source link

Use ZFS features (but simple!) #32

Open probonopd opened 4 years ago

probonopd commented 4 years ago

ZFS is this incredibly powerful tool that 99% of "mere mortals" desktop users will never entirely wrap their head around.

Hence, we should pre-configure it in a way most suitable to desktop users, and provide some really easy tools (ideally with zero configuration) to do the most common jobs, like scheduled local snapshots, rollbacks from within the boot manager, and network backups.

ZFS features I want to make use of

Ideally I want to configure as little as possible as I am not a system administrator but a mere end user (zero configuration).

ZFS complexities to be understood/handled

While ZFS seems incredibly powerful, turns out it is also a lot more complex than needed, at least for desktop use.

Especially when plugging in and out external disks all the time, which I tend to do a lot. For example, I want to take the internal disk, put it on a SATA-to-USB3 adapter, and attach it to another target system, then copy stuff from /home on that disk to the internal disk of the target system.

Datasets

There is a concept of datasets which is some weird mix between partitions (Mac "volumes") and directories but not quite. Why is there yet another concept that we need to deal with? I would probably like to have just one dataset per pool so that things become as simple as possible.

This stuff reminds me of the old times when people set up different partitions for /home, /var, /usr and so on. I never liked having to deal with more than one partition on a disk, because I never knew which one would need how much space over time. With ZFS datasets, I no longer need to define sizes for those, but for some strange reasons, it seems that people still are dealing with different paths individually. I am used to systems where everything is on one partition, which makes everything much simpler for me.

So, is there a way to make it so that there is just one "/" dataset and that is it? Especially I want the whole disk behave the same and not run into unintended surprises like the ones outlined below.

Mounting external disks

When attaching an external disk with ZFS on it, then it doesn't just get mounted as an external disk. Instead, one has to "import the pool" (whatever that really means) and, suprise!, this mounts stuff over the booted system. For example, /home on the external disk doesn't get mounted under /media/external/home or something like it, but under /home.

This is referred to as "clobbering or replacing the current file system.

The behavior can be different for different datasets, depending on on whether some esoteric ZFS options like "canmount" have been set.

What is this good for? I find this dangerous, because it is very unexpected. How do I copy stuff from /home on the external disk to /home on the internal disk?

https://unix.stackexchange.com/questions/296972/how-to-mount-external-zfs-file-system-without-clobbering-altering-current-or-ext

Boot environments

When I played with boot environments, I thought "it does not work" until I found out by watching a YouTube video that actually /home is not covered by boot environments by default. What good is that for? To me, this violates POLA. Why can't I just have the whole system, without any exceptions, covered by boot environments? What good are "backups" when they leave out the most important type of data, the user's own? There should be at least a BIG RED WARNING that boot environments do NOT protect user data. (Probably the inventors thought of boot environments of something that covers only system changes while leaving user data as-is; whereas I experiment with user-side settings a lot, like changing to different desktop environments and playing with their settings, and want to use boot environments for that.)

probonopd commented 4 years ago

Here is a prototype for Boot Environments:

image

It is a GUI for the beadm tool by @vermaden.

probonopd commented 4 years ago

Why can't I just have the whole system, without any exceptions, covered by boot environments?

To get this, follow https://twitter.com/vermaden/status/1321952264735674368:

After default #FreeBSD on #ZFS install do these:

# zfs rename -u zroot/usr zroot/ROOT/default/usr
# zfs rename -u zroot/var zroot/ROOT/default/var

The /home is by default at /usr/home so it will also be moved to BE.

probonopd commented 3 years ago

Automounting existing ZFS pools from hot-plugged external disks

https://docs.oracle.com/cd/E19253-01/819-5461/gbcgl/index.html

Pools can also be imported using an alternate root. This feature allows for recovery situations, where the mount points should not be interpreted in context of the current root, but under some temporary directory where repairs can be performed. This feature also can be used when you are mounting removable media as described in the preceding section.

zpool import -R ...

Probably we should use this for automouting ZFS pools. Or is there an automounter for FreeBSD that can already do this?

Creating ZFS pools on external disks

Previously known as "formatting a disk".

https://docs.oracle.com/cd/E19253-01/819-5461/gaztn/index.html

The most common reason for creating an alternate root pool is for use with removable media. In these circumstances, users typically want a single file system, and they want it to be mounted wherever they choose on the target system. 

zpool create -R ...

Managing ZFS pools the traditional way

https://docs.oracle.com/cd/E19253-01/819-5461/gaztn/index.html

You can manage ZFS file systems with legacy tools by setting the mountpoint property to legacy. Legacy file systems must be managed through the mount and umount commands

Isn't that what we want anyway, for everything but the pool from which the system boots?

Automatic fast and frequent incremental ZFS backups to a backup server

https://www.grendelman.net/wp/fast-frequent-incremental-zfs-backups-with-zrep/

Zrep is a shell script (written in Ksh) that was originally designed as a solution for asynchronous (but continuous) replication of file systems for the purpose of high availability (using a push mechanism).

vermaden commented 3 years ago

From the useful tools that may but also may not) be useful here is rclone which can clone backups between various types, for example it can clone tar.gz file into S3 bucket on Minio or Amazon S3.

probonopd commented 3 years ago

https://twitter.com/vermaden/status/1326097314281353217

I can add ZFS pool attach to sysutils/automount if that is desired?

This would be super useful. My use case: Take FreeBSD ZFS boot disk out of a computer, attach it to another running FreeBSD system using a USB3 adapter. The partitions should be auto-mounted under /media "as usual", like if we were dealing with, e.g., NTFS

probonopd commented 3 years ago

For LAN backups:

According to @darkain, FreeNAS/TrueNAS is a very commonly installed thing, and is FreeBSD+ZFS based. Whatever ZFS scripts are made, all that would be needed is something to automatically put SSH keys on the destination, otherwise, it'll "just work".

So we need to:

  1. Discover FreeNAS/TrueNAS on the network using Zeroconf
  2. Copy our SSH keys there (how)?

For WAN backups:

https://www.tarsnap.com/ --> See https://github.com/helloSystem/Utilities/issues/155.

shilch commented 3 years ago

Want I would find very user friendly is if the operating system was a separate ZFS dataset mounted read-only at /System or so. Updates could then be shipped as ZFS snapshots and only need a proper boot manager that can handle this. Thereby updates would be atomic in nature and can easily be rolled back. Additionally, you have a clear distinction between the operating system and own changes.
Is someone aware of a system (some Linux distribution?) that does it that way?

darkain commented 3 years ago

FreeBSD standard distribution already partially does this. The "root" file system is actually zroot/ROOT/default with a mountpoint of /

If this were made read-only, with other datasets made read-write with other mount points, it could easily solve this situation.

Also, check out how iocage does jails, they have the concept of base jails that other jails can inherit from. They've already done a lot of the work on that front that can probably be replicated for the host OS as well.

probonopd commented 3 years ago

Very interesting!

vermaden commented 3 years ago

Want I would find very user friendly is if the operating system was a separate ZFS dataset mounted read-only at /System or so. Updates could then be shipped as ZFS snapshots and only need a proper boot manager that can handle this. Thereby updates would be atomic in nature and can easily be rolled back. Additionally, you have a clear distinction between the operating system and own changes. Is someone aware of a system (some Linux distribution?) that does it that way?

FreeBSD already has this in the default ZFS install. Its called ZFS Boot Environments and its managed by beadm(8) from sysutils/beadm or bectl(8) from base. If you do not know this concept then please read this - https://is.gd/BECTL - should contain all needed information about ZFS Boot Environments.

shilch commented 3 years ago

I would prefer a read-only mount since even root can not accidentally modify system data. It requires the user to really understand what he's doing to modify the system.
But I don't know if it breaks POSIX if directories like /bin or /usr/bin are not writable to.

Edit: To clarify, have the system mounted read-only at /System and have a bind-mount / nullfs mount for directories at /bin or /usr/bin point into directories inside the /System.

shilch commented 3 years ago

Want I would find very user friendly is if the operating system was a separate ZFS dataset mounted read-only at /System or so. Updates could then be shipped as ZFS snapshots and only need a proper boot manager that can handle this. Thereby updates would be atomic in nature and can easily be rolled back. Additionally, you have a clear distinction between the operating system and own changes. Is someone aware of a system (some Linux distribution?) that does it that way?

FreeBSD already has this in the default ZFS install. Its called ZFS Boot Environments and its managed by beadm(8) from sysutils/beadm or bectl(8) from base. If you do not know this concept then please read this - https://is.gd/BECTL - should contain all needed information about ZFS Boot Environments.

Thanks, I already read about it but I'm not sure if it is what I mean. I will test it out.

It looks pretty much like it. The question is how flexible the tools are.

vermaden commented 3 years ago

The question is how flexible the tools are.

What do you mean?

shilch commented 3 years ago

The question is how flexible the tools are.

What do you mean?

Whether they support different mount points and mount options such as read-only.

vermaden commented 3 years ago

Whether they support different mount points and mount options such as read-only.

Different mount points are supported. You can set in ZFS the readonly property to ON of OFF for any dataset.

probonopd commented 3 years ago

So far, I was unable to use zfs+nullfs+unionfs ro construct a root filesystem that appears to be r/w even though it resides on a r/o ISO.

https://github.com/helloSystem/ISO/blob/pure-zfs-test/overlays/ramdisk/init.sh#L63-L97

The result is unfortunately a kernel panic.

grahamperrin commented 3 years ago

Avoiding clobber

I guess:

  1. include OpenZFS in stable/12 ISOs/distributions of helloSystem
  2. zpool import with option -N to not mount any file system
  3. use temporary mount points to avoid clobber.

@ahrens point (3), is that sane/practical?

IMHO for 13.0-CURRENT, manual page wording is potentially confusing in that temporary mount point properties do not include the mount point mountpoint.

References:

Historical:

grahamperrin commented 3 years ago
  1. use temporary mount points to avoid clobber.

I had an opportunity to try this a few days ago. Either:

Clobber occurred, so I quietly dug myself out of my hole.