Issue booting from snapshot - @log subvolume is read-only

[CPU-Blanc]

Hello! I know this seems to be a semi-common problem, and I've followed the usual advice to fix it, but I'm still left scratching my head. I've followed the readme and added the required hooks. The system does mount root as read-write, so the hooks are working. The issue is /var/log, which is a seperate subvolume on my system, is still being mounted read-only and is causing Xorg/Light Display Manager to freak.

Any ideas on how to get overlyfs to work correctly with this subvolume, if that is indeed the problem?

I'm running EndeavourOS

My subvolume layout is:

ID 262 gen 146774 top level 5 path games
ID 443 gen 139401 top level 5 path arch/@
ID 445 gen 139401 top level 5 path arch/@cache
ID 446 gen 139401 top level 5 path arch/@log
ID 455 gen 146781 top level 5 path endeavour/@
ID 456 gen 146781 top level 5 path endeavour/@log
ID 457 gen 146237 top level 5 path endeavour/@cache
ID 462 gen 146754 top level 455 path .snapshots
ID 463 gen 138769 top level 462 path .snapshots/1/snapshot
ID 464 gen 138770 top level 462 path .snapshots/2/snapshot
ID 465 gen 140205 top level 462 path .snapshots/3/snapshot
[...]

Relevant fstab entries:

# <file system>                                 <mount point>                   <type>  <options>                                                       <dump>  <pass>
UUID=244E-C5AA                                  /boot/efi                       vfat    fmask=0137,dmask=0027                                           0       2
UUID=913bf26c-4239-4ae3-a61f-54e24d0b01cc       /                               btrfs   subvol=/endeavour/@,noatime,compress=zstd                       0       0
UUID=913bf26c-4239-4ae3-a61f-54e24d0b01cc       /var/cache                      btrfs   subvol=/endeavour/@cache,noatime,compress=zstd                  0       0
UUID=913bf26c-4239-4ae3-a61f-54e24d0b01cc       /var/log                        btrfs   subvol=/endeavour/@log,noatime,compress=zstd                    0       0
UUID=6c2cad87-8994-4339-b866-ed04c190a259       swap                            swap    defaults                                                        0       0
tmpfs                                           /tmp                            tmpfs   defaults,noatime,mode=1777                                      0       0
UUID=913bf26c-4239-4ae3-a61f-54e24d0b01cc       /home/blanc/Games               btrfs   subvol=/games,noatime,compress=zstd                             0       0
[...]

It would appear as though any subvolume that's on the same disk as / when booting into the snapshot are mounted ro, but root itself is rw. Partitions/subvolumes from other drives are rw.

ie

/ - Write
/var/log - Read-Only
/var/cache - Read-Only
/home/blanc/Games - Read-Only
/home (on another btrfs drive) - Write

Booting into a read-write snapshot does not present this issue, it's only when attempting to boot into a read-only one. Any help troubleshooting would be greatly appreciated.

[bkmo]

I have a hunch on where the issue might be occurring. Is it possible this is happening within the initramfs hook? My knowledge is a little lacking, but is it possible that during the overlayfs mounting step, it's simply mounting the btrfs partition without passing a subvol, and so it's mounting the default subvolume to it?

I will look at the script to see if I can make heads or tails of it.

[HanM23]

I switched from Timeshift to Snapper/btrfs assitant because Timeshit was sometimes messy with restoration. I never had restoration issue with btrfs assistant. But i understand, when you do not trust an app, well, you do not want to use it even-though someone else says it works fine.

You solution with systemd-boot and grub in UEFI sounds interesting, but i really do not trust my UEFI as sometimes it does not keep all entries in NVRAM, i do not why.

For instance, if i boot on a usb key (like ventoy), and then i want to boot again the normal way it cannot find anymore my nvme boot entry and had to boot on live iso and set it with efibootmgr

[HanM23]

initramfs hook?

There is definitely an issue with the overlay

[bkmo]

initramfs hook?

There is definitely an issue with the overlay

That works for me.

[HanM23]

That works for me.

lol What is your distro, is it arch based also ?

[bkmo]

That works for me.

lol What is your distro, is it arch based also ?

Vanilla Arch, SDDM/Plasma6

[HanM23]

initramfs hook?

There is definitely an issue with the overlay

That works for me.

Yes, but by the way, is it the latest overlayfs version ? Or you said it was sd-overlay, maybe it's different. I know there was several overlay versions here.

[bkmo]

initramfs hook?

There is definitely an issue with the overlay

That works for me.

Yes, but by the way, is it the latest overlayfs version ? Or you said it was sd-overlay, maybe it's different. I know there was several overlay versions here.

Works with both my sd-overlay and I tested it with grub-btrfs-overlayfs to be sure. Looking at the current overlay it makes sure there is a ro root ([[ $(btrfs property get ${root_mnt} ro) != "ro=false" ]]) then creates a rw overlay for it in ram. I am not really seeing a smoking gun.

[bkmo]

The other thing is that I do not need overlayfs for a snapshot to boot without errors on SDDM/Plasma.

[CPU-Blanc]

My Arch install is running SDDM/Hyprland, and it cannot run from ro snapshots.

[bkmo]

This is the overlayfs hook if anyone wants to look into it. https://github.com/Antynea/grub-btrfs/blob/master/initramfs/Arch%20Linux/overlay_snap_ro-hook

[Zesko]

mkinitcpio with systemd hook did not support overlayfs-hook a few months ago, AFAIK.

If you need mkinitcpio on Arch, try changing systemd hook to Busybox init.

OR

One solution is Dracut with pure Systemd support and has no issue with overlayfs hook, but its setup must be done by hand.

The overlayfs bug has been fixed in Kernel 6.10.

If you need some Kernel versions with the overlayfs bug, look at a solution:

https://gitlab.com/garuda-linux/pkgbuilds/-/commit/26d7f8496c242265b6902ee9188f134194cd0c0b

[CPU-Blanc]

mkinitcpio with systemd hook did not support overlayfs-hook a few months ago, AFAIK.

If you need mkinitcpio on Arch, try changing systemd hook to Busybox init.

Thanks for chipping in! Sadly I am already using a busybox based init and the does not work unless I manually set the snapshot as read-write.