Possible solution for dracut-based distros?

[Le0xFF]

Hello and thank you for this massive work!

I'm using Void Linux, today I tried to boot a snapshot from grub to check if this works but I ended up with a black screen, with some /var/log error, even if I have some subvolumes of /var as you can see from my layout:

$ sudo btrfs subvolume list /
ID 256 gen 1055 top level 5 path @
ID 276 gen 1067 top level 5 path @home
ID 257 gen 1055 top level 5 path @snapshots
ID 258 gen 849 top level 256 path var/cache/xbps
ID 259 gen 538 top level 256 path var/tmp
ID 260 gen 1065 top level 256 path var/log

I saw that this could be related to issue #92, so looking better at this README, there are specific instructions for mkinitcpio-based distros, but not for dracut ones, as also I read from issue #160.

Looking at dracut documentation I saw two options that made my snapshots bootable:

• rd.live.overlay.readonly=1
• rd.live.overlay.overlayfs=1

Adding one of these two options to the linux grub line, my snapshot booted without any issue.

The problem is that I don't know if this is the proper way to boot a snapshot the way it should be as I'm not really a Linux savvy. May I ask you if you could take a look at that?

[mbana]

I have this working in Fedora 36.

I'd watch this space: https://github.com/banaio/grub-btrfs. When I find time, I'll detail the steps and configuration required for Fedora on BTRFS.

The annoying thing is that you need to set the default volume back after doing a snapshot. Other than that, it works fine.

[daiaji]

@Antynea Looks like the work has been made simpler, providing an option to add rd.live.overlay.readonly=1 with rd.live.overlay.overlayfs=1 to the kernel command line by the way when generating the grub menu? Does this cause all directories to be read-only?

[slashmach1]

adding either option @daiaji mentioned to this line in 41_snapshots-btrfs: kernel_parameters="$GRUB_CMDLINE_LINUX $GRUB_CMDLINE_LINUX_DEFAULT

before installing grub-btrfs will allow this to work with dracut distros. rd.live.overlay.readonly=1 will make readonly while rd.live.overlay.overlayfs=1 allows read/write but changes are lost on reboot unless comitted to a new snapshot:
example: kernel_parameters="$GRUB_CMDLINE_LINUX $GRUB_CMDLINE_LINUX_DEFAULT rd.live.overlay.overlayfs=1"

NOTE: certain var directories need to be separate subvolumes depending on your window manager or desktop environment; otherwise systemd wont load to GUI. gnome is /var/lib/gdm and /var/lib/AccountsService

[slashmach1]

fedora also may need to install the inotify-tools package. testing on latest nobara

[Schievel1]

adding either option @daiaji mentioned to this line in 41_snapshots-btrfs: kernel_parameters="$GRUB_CMDLINE_LINUX $GRUB_CMDLINE_LINUX_DEFAULT

before installing grub-btrfs will allow this to work with dracut distros. rd.live.overlay.readonly=1 will make readonly while rd.live.overlay.overlayfs=1 allows read/write but changes are lost on reboot unless comitted to a new snapshot: example: kernel_parameters="$GRUB_CMDLINE_LINUX $GRUB_CMDLINE_LINUX_DEFAULT rd.live.overlay.overlayfs=1"

NOTE: certain var directories need to be separate subvolumes depending on your window manager or desktop environment; otherwise systemd wont load to GUI. gnome is /var/lib/gdm and /var/lib/AccountsService

If I add a config parameter like kernel_parameters="$GRUB_CMDLINE_LINUX $GRUB_CMDLINE_LINUX_DEFAULT $GRUB_BTRFS_SNAPSHOT_KERNEL_PARAMETERS"

and you can set that in /etc/default/grub-btrfs/config to whatever you like, would this be enough?

fedora also may need to install the inotify-tools package. testing on latest nobara

It certainly does for the daemon.

[slashmach1]

Have not tested the modification using that file. The file I'm referring to in my previous comment is /etc/grub.d/41_snapshots-btrfs after grub-btrfs is installed. And either rd.live.overlay.overlayfs=1 or rd.live.overlay.readonly=1 needs to be added to the snapshot boot commandline

[Schievel1]

I see, but basically all you need is the ability to add this to the snapshots kernel command line?

See here: https://github.com/Antynea/grub-btrfs/tree/improve-config/initramfs#dracut-based-distros Is this documentation correct like it is? As I understand it, this will not work at all on Dracut based Distros like Void unless you add that kernel command line? But on Fedora it is just added to the command line to make snapshots immutable? I am a bit confused from what you guys are writing here, so maybe you can enlighten me a bit for the documentation.

I am setting up dracut to make my initramfs right now to test it out. But I am running Gentoo, where Dracut is a niche thing. Would be great if someone running Fedora could check if this is correct as well.

/ does not work in the sense that snapshots are still mutable. On the other hand my setup with Dracut generating initramfs it boots them with and without that parameter. So there is that, I need your help getting this into master.

[slashmach1]

This line is incorrect: "After changing this run sudo /etc/grub.d/41_snapshots-btrfs to generate a new snapshot-submenu with the parameter added." That script is executed when grub is regenerating grub.cfg so simply updating grub after installation or changing that parameter should be enough for implementation.

The only thing other than that is to make note of is the /var partition layout. the desktop environment or display manager directories (/var/lib/{display manager or desktop environment application folder}) at least need to be on separate subvolumes. Otherwise even with these changes the system will boot but wont load to GUI from what I have tried.

EXAMPLE: /var/lib/AccountsService and /var/lib/gdm on a separate subvolume for gnome users and /var/lib/AccountsService and /var/lib/sddm on a separate subvolume for KDE users

[Schievel1]

Why is that, though? I mean when we boot without the rd.live.overlay.overlayfs=1 parameter, we would end up with a mutable system. If we boot with rd.live.overlay.overlayfs=1 we would end up with a mutable system. (that is back to it's original state when rebootet) What is happening here?

That is all a bit confusing. Where you got this from? Is there any documentation regarding Dracut and booting btrfs snapshots?

[slashmach1]

That im not 100% sure. I did get grub-btrfs working on Nobara, just sharing my steps to replicate the full picture.

[slashmach1]

I believe the documentation for grub-btrfs calls for var being on a separate partition anyways on dracut distros in setup which would cover the directories I reference

[Schievel1]

Yes when I find the time I will rework what is written in which place and so on to make this a bit less scattered

[daiaji]

Thanks for your work, I think I'll switch back to dracut if dracut's support is available.

I am setting up dracut to make my initramfs right now to test it out. But I am running Gentoo, where Dracut is a niche thing. Would be great if someone running Fedora could check if this is correct as well.

To be honest, dracut is not particularly niche, at least opensuse uses dracut to generate initramfs by default. And this post seems to show some native functions used by dracut, such as overlay without hook script, which is more convenient than mkinitcpio.

[Schievel1]

Gentoo, where Dracut is a niche thing. To be honest, dracut is not particularly niche, at least opensuse uses dracut to generate initramfs by default. And this post seems to show some native functions used by dracut, such as overlay without hook script, which is more convenient than mkinitcpio.

Nah, I meant on Gentoo it is kinda niche. Gentoo has genkernel which can build you the initrd. But I gotta say, Dracut does it's job, all I did was deleting my initrds and runnig drauct --regenerate-all. Problem is rather, that I can not reproduce your problem. I can boot the snapshots all fine, regardless of what I give as kernel command line. That I a bit of a bummer, because I was hoping I could write some proper documentation with this.

/

❯ cat /etc/portage/savedconfig/sys-kernel/gentoo-kernel-6.0.9 | grep OVERLAY
CONFIG_EFI_CUSTOM_SSDT_OVERLAYS=y
# CONFIG_OF_OVERLAY is not set
# CONFIG_OVERLAY_FS is not set

So yeah, without overlayfs support compiled into my kernel, there is no overlayfs for me. :nerd_face:

// nop, snapshot is still mutable even with CONFIG_OVERLAY_FS=y in the kernel.

[daiaji]

What I know about Gentoo is limited to his support for x32 ABI, but x32 ABI is too niche, and using him seems to mean that you need to recompile all packages for x32 ABI, which seems to mean that I have almost no container images available.

To be honest, even genkernel was the first time I knew about it.

The reason I know about Dracut is also because I've used openSUSE.

[daiaji]

https://github.com/dracutdevs/dracut/issues?q=rd.live.overlay.overlayfs

In fact, rd.live.overlay.overlayfs=1 seems to have some failures, because I created a separate sub volume for the/var folder. It seems that the OS is bootable, but the root directory is still not writable.

At least I didn't find the overlayfs was mounted.

A quirky mistake. dracut-pre-udev : sh: /usr/bin/sysctl: No such file or directory

I guess I should add /usr/bin/sysctl for initramfs, but I don't know how.

[LaszloGombos]

@daiaji which dracut version are you on ?

https://github.com/dracutdevs/dracut/pull/1934 is not yet released, hope to get it released with dracut v58.

Also you might be able to just grab the dracut module directory from upstream and try to use it with your local dracut installation version.

I guess I should add /usr/bin/sysctl for initramfs, but I don't know how.

dracut -I sysctl

If you want to see how rd.live.overlay.overlayfs is tested upstream (some example command line options) - https://github.com/dracutdevs/dracut/blob/master/test/TEST-16-DMSQUASH/test.sh#L21

rd.live.overlay.readonly=1 will make readonly while rd.live.overlay.overlayfs=1 allows read/write but changes are lost on reboot unless comitted to a new snapshot

This seems correct to me.

[daiaji]

@daiaji which dracut version are you on ?

dracutdevs/dracut#1934 is not yet released, hope to get it released with dracut v58.

archlinux has an AUR package called dracut-git, which I built and installed.

dracut -I sysctl

If you want to see how rd.live.overlay.overlayfs is tested upstream (some example command line options) - https://github.com/dracutdevs/dracut/blob/master/test/TEST-16-DMSQUASH/test.sh#L21

I did a more detailed test and the glitch seems to go away after adding the dmsquash-live module. And after checking the directory mounting status, I found that rootfs mounted the overlay, but there is a problem that when I add the dmsquash-live module, the --hostonly parameter seems to be unavailable, which means that after adding the dmsquash-live module I can't use dracut-hook.

Also, I can't seem to find a mount point for my btrfs snapshots.

mount | grep -E "overlay"
rootfs on / type overlay (rw,relatime,lowerdir=/tmp.ICyUf1,upperdir=/tmp.2YEua1/upper,workdir=/tmp.2YEua1/work)
rootfs on /var/lib type overlay (rw,relatime,lowerdir=/tmp.ICyUf1,upperdir=/tmp.2YEua1/upper,workdir=/tmp.2YEua1/work)

sudo dracut --hostonly --force /boot/initramfs-6.0-x86_64.img
/etc/dracut.conf.d/myflags.conf:add_dracutmodules+="dmsquash-live"

dracut: WARNING: <key>+=" <values> ": <values> should have surrounding white spaces!
dracut: WARNING: This will lead to unwanted side effects! Please fix the configuration file.

dracut: Executing: /usr/bin/dracut --hostonly --force /boot/initramfs-6.0-x86_64.img
dracut: dracut module 'dash' will not be installed, because command 'dash' could not be found!
dracut: dracut module 'mksh' will not be installed, because command 'mksh' could not be found!
dracut: dracut module 'busybox' will not be installed, because command 'busybox' could not be found!
dracut: dracut module 'dbus-broker' will not be installed, because command 'dbus-broker' could not be found!
dracut: dracut module 'rngd' will not be installed, because command 'rngd' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmanctl' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand-wait-online' could not be found!
dracut: dracut module 'network-wicked' will not be installed, because command 'wicked' could not be found!
dracut: dracut module 'tpm2-tss' will not be installed, because command 'tpm2' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsi-iname' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsiadm' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsid' could not be found!
dracut: dracut module 'biosdevname' will not be installed, because command 'biosdevname' could not be found!
dracut: dracut module 'memstrack' will not be installed, because command 'memstrack' could not be found!
dracut: memstrack is not available
dracut: If you need to use rd.memdebug>=4, please install memstrack and procps-ng
dracut: dracut module 'dmsquash-live' cannot be found or installed.

mount
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
dev on /dev type devtmpfs (rw,nosuid,relatime,size=16334440k,nr_inodes=4083610,mode=755,inode64)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
rootfs on / type overlay (rw,relatime,lowerdir=/tmp.mfEpVb,upperdir=/tmp.dOK9YQ/upper,workdir=/tmp.dOK9YQ/work)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=25439)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
/dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)
/dev/nvme0n1p2 on /btrfsroot type btrfs (rw,relatime,ssd,space_cache=v2,subvolid=5,subvol=/)
/dev/nvme0n1p2 on /home type btrfs (rw,relatime,ssd,space_cache=v2,subvolid=257,subvol=/@home)
/dev/nvme0n1p2 on /var type btrfs (rw,relatime,ssd,space_cache=v2,subvolid=316,subvol=/@var)
/dev/zram1 on /tmp type xfs (rw,relatime,attr2,discard,inode64,logbufs=8,logbsize=32k,noquota)
rootfs on /var/lib type overlay (rw,relatime,lowerdir=/tmp.mfEpVb,upperdir=/tmp.dOK9YQ/upper,workdir=/tmp.dOK9YQ/work)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3278548k,nr_inodes=819637,mode=700,uid=1000,gid=1000,inode64)
portal on /run/user/1000/doc type fuse.portal (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)

grub menu
    menuentry '  vmlinuz-6.0-x86_64 & initramfs-6.0-x86_64.img & intel-ucode.img' --class snapshots --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-snapshots-ca0a7c48-b433-4667-a759-42b305229791' {
        if [ x$feature_all_video_module = xy ]; then
        insmod all_video
        fi
        set gfxpayload=keep
        insmod btrfs
        if [ x$feature_platform_search_hint = xy ]; then
            search --no-floppy --fs-uuid  --set=root  ca0a7c48-b433-4667-a759-42b305229791
        else
            search --no-floppy --fs-uuid  --set=root ca0a7c48-b433-4667-a759-42b305229791
        fi
        echo 'Loading Snapshot: 2022-12-10 19:12:05 snapshots_root/@.20221210T1912'
        echo 'Loading Kernel: vmlinuz-6.0-x86_64 ...'
        linux "/snapshots_root/@.20221210T1912/boot/vmlinuz-6.0-x86_64" root=UUID=ca0a7c48-b433-4667-a759-42b305229791  quiet udev.log_priority=3 rootflags=subvol="snapshots_root/@.20221210T1912"
        echo 'Loading Microcode & Initramfs: intel-ucode.img initramfs-6.0-x86_64.img ...'
        initrd "/snapshots_root/@.20221210T1912/boot/intel-ucode.img" "/snapshots_root/@.20221210T1912/boot/initramfs-6.0-x86_64.img"
    }

[LaszloGombos]

Try to repace

sudo dracut --hostonly --force /boot/initramfs-6.0-x86_64.img

with sudo dracut --hostonly --force /boot/initramfs-6.0-x86_64.img -a "debug dmsquash-live"

You missed this warning and the above would work around it

dracut: WARNING: +=" ": should have surrounding white spaces!

[daiaji]

sudo dracut --hostonly --force /boot/initramfs-6.0-x86_64.img -a "debug dmsquash-live"
dracut: Executing: /usr/bin/dracut --hostonly --force /boot/initramfs-6.0-x86_64.img -a "debug dmsquash-live"
dracut: dracut module 'dash' will not be installed, because command 'dash' could not be found!
dracut: dracut module 'mksh' will not be installed, because command 'mksh' could not be found!
dracut: dracut module 'busybox' will not be installed, because command 'busybox' could not be found!
dracut: dracut module 'dbus-broker' will not be installed, because command 'dbus-broker' could not be found!
dracut: dracut module 'rngd' will not be installed, because command 'rngd' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmanctl' could not be found!
dracut: dracut module 'connman' will not be installed, because command 'connmand-wait-online' could not be found!
dracut: dracut module 'network-wicked' will not be installed, because command 'wicked' could not be found!
dracut: dracut module 'tpm2-tss' will not be installed, because command 'tpm2' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsi-iname' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsiadm' could not be found!
dracut: dracut module 'iscsi' will not be installed, because command 'iscsid' could not be found!
dracut: dracut module 'biosdevname' will not be installed, because command 'biosdevname' could not be found!
dracut: dracut module 'memstrack' will not be installed, because command 'memstrack' could not be found!
dracut: memstrack is not available
dracut: If you need to use rd.memdebug>=4, please install memstrack and procps-ng
dracut: dracut module 'dmsquash-live' cannot be found or installed.

It seems that now that the overlayfs is no longer mounted, things get weird. But I seemed to get the correct mount before.