Issue booting from snapshot - @log subvolume is read-only

[CPU-Blanc]

Hello! I know this seems to be a semi-common problem, and I've followed the usual advice to fix it, but I'm still left scratching my head. I've followed the readme and added the required hooks. The system does mount root as read-write, so the hooks are working. The issue is /var/log, which is a seperate subvolume on my system, is still being mounted read-only and is causing Xorg/Light Display Manager to freak.

Any ideas on how to get overlyfs to work correctly with this subvolume, if that is indeed the problem?

I'm running EndeavourOS

My subvolume layout is:

ID 262 gen 146774 top level 5 path games
ID 443 gen 139401 top level 5 path arch/@
ID 445 gen 139401 top level 5 path arch/@cache
ID 446 gen 139401 top level 5 path arch/@log
ID 455 gen 146781 top level 5 path endeavour/@
ID 456 gen 146781 top level 5 path endeavour/@log
ID 457 gen 146237 top level 5 path endeavour/@cache
ID 462 gen 146754 top level 455 path .snapshots
ID 463 gen 138769 top level 462 path .snapshots/1/snapshot
ID 464 gen 138770 top level 462 path .snapshots/2/snapshot
ID 465 gen 140205 top level 462 path .snapshots/3/snapshot
[...]

Relevant fstab entries:

# <file system>                                 <mount point>                   <type>  <options>                                                       <dump>  <pass>
UUID=244E-C5AA                                  /boot/efi                       vfat    fmask=0137,dmask=0027                                           0       2
UUID=913bf26c-4239-4ae3-a61f-54e24d0b01cc       /                               btrfs   subvol=/endeavour/@,noatime,compress=zstd                       0       0
UUID=913bf26c-4239-4ae3-a61f-54e24d0b01cc       /var/cache                      btrfs   subvol=/endeavour/@cache,noatime,compress=zstd                  0       0
UUID=913bf26c-4239-4ae3-a61f-54e24d0b01cc       /var/log                        btrfs   subvol=/endeavour/@log,noatime,compress=zstd                    0       0
UUID=6c2cad87-8994-4339-b866-ed04c190a259       swap                            swap    defaults                                                        0       0
tmpfs                                           /tmp                            tmpfs   defaults,noatime,mode=1777                                      0       0
UUID=913bf26c-4239-4ae3-a61f-54e24d0b01cc       /home/blanc/Games               btrfs   subvol=/games,noatime,compress=zstd                             0       0
[...]

It would appear as though any subvolume that's on the same disk as / when booting into the snapshot are mounted ro, but root itself is rw. Partitions/subvolumes from other drives are rw.

ie

/ - Write
/var/log - Read-Only
/var/cache - Read-Only
/home/blanc/Games - Read-Only
/home (on another btrfs drive) - Write

Booting into a read-write snapshot does not present this issue, it's only when attempting to boot into a read-only one. Any help troubleshooting would be greatly appreciated.

[CPU-Blanc]

Cannot confirm whether this worked in kernel 6.6, but my symptoms do seem to somewhat resemble those in #328 . Possibly related?

[HanM23]

Yes, i have the same issue, cannot boot in snapshot from grub by default if snapshot are taken with snapper.

It's working if you are using timeshift, because snapshots are not r/o by default. But, i have lot of issues with timeshift for restoring, i do not use it anymore.. Therefore, Snapper (with btrfs-assistant) is perfect for snapshot restoration, but i am in the same situation, i cannot boot a r/o snapshot, although /var/log and var/cache are on separate subvolumes.

What i do each time i create a new snapshot, i set it to be writable:

sudo btrfs property set /.snapshots/<snapshot number>/snapshot ro false

[CPU-Blanc]

Yep, I get it works fine with rw snapshots. The issue is that's quite a flaw, and something I'd like to avoid. A snapshot should be just that, frozen in time. If I boot into a snapshot, accidentally change some files or some sort of application does, if the snapshot is rw then that snapshot is now tainted with no way of reversing it.

Unless there's some mkinitcpio hook for taking a rw snapshot of the ro snapshot at boot time, booting into that, then discarding the rw on shutdown? I was hoping overlayfs would accomplish this, but it doesn't seem to play nice with those other subvolumes... :/

[HanM23]

I have tried to put GRUB_BTRFS_SNAPSHOT_KERNEL_PARAMETERS="systemd.volatile=state" in /etc/default/grub-btrfs/config, but that does not solve the issue either.

@Antynea Could you please help us to find out a workaround ?

Thank you

[bkmo]

What are the mount options in fstab for the @log subvolume? If I add rw to the options I have no issues.

[CPU-Blanc]

What are the mount options in fstab for the @log subvolume? If I add rw to the options I have no issues.

fstab options are listed in the OP subvol=/endeavour/@log,noatime,compress=zstd. I tried adding rw but that did not work, still mounting as ro from within a snapshot

[bkmo]

I don't have this issue on Arch SDDM-Plasma. I did have the problem with @home being ro until I changed fstab options to rw. Have you rebooted and re-snapshotted after adding rw to @log? Try booting a new snapshot maybe.

[CPU-Blanc]

Yep. Tried that. Added rw to options in fstab, did the grub-mkconfig + new snapshot dance, no luck Confirmed it took by tty-ing inside the snapshot and checking fstab did indeed list the rw option

[HanM23]

Arch SDDM-Plasma

what is your subvolume layout ?

I also read that putting /var as subvolume (instead for /var/log and var/cache as i did) could also solved the issue but that's not a good solution as it can cause other problems

[bkmo]

Arch SDDM-Plasma

what is your subvolume layout ?

I also read that putting /var as subvolume (instead for /var/log and var/cache as i did) could also solved the issue but that's not a good solution as it can cause other problems

I have @cache mounted to /var/cache and @log mounted to /var/log. Same as you it seems. What is your DM-DE? I can boot a snapshot without overlayfs with SDDM-Plasma

[HanM23]

boot a snapshot without overlayfs with SDDM-Plasma

You do not have the grub-btrfs-overlayfs hook in your mkinitcpio.conf ? I also have SDDM and Plasma 6 on Manjaro.

I will try without the overlayfs and put rw for @log subvolume in fstab later this evening, it's dinner time right now :) I'll tell you what happened

[bkmo]

I have a systemd based init with an sd-overlayfs right now, but without it a snapshot always booted without issues. I also have rw in my kernel cmdline in systemd-boot entries and /etc/default/grub

[HanM23]

I have a systemd based init with an sd-overlayfs right now, but without it a snapshot always booted without issues.

I see, but, maybe we are confusing. I mean, i am using snapper to take snapshots. Snapper takes r/o snapshots. That's why when booting from a snapshot i have to put it r/w, with or without grub-btrfs-overlayfs. This hook worked until few months ago but it does not work anymore currently, whether on an Arch based distro or Fedora. How do you take your snapshots ?

If i was using TimeShift, for instance, i wouldn't have those issue.

i do not know your overlay, where did you get it ? :+1:

[bkmo]

My overlay is from a post in one of the issues here. I am using snapper and do not have to set snapshots to rw regardless of using overlayfs or not. I can use a udev init with grub-btrfs-overlayfs without issues also, so overlayfs is not the problem. What do you get when booted into a snapshot with this command? stat -f -c '%T' / It should result either btrfs or overlayfs.

[HanM23]

I am using snapper and do not have to set snapshots to rw regardless of using overlayfs or not

I am progressing.

I put rw in /etc/default/grub command line. rw in fstab does not change anything more (or less). --> If i boot my snapshot without overlay, i boot normally, but my filesystem is read-only, and i cannot do anything once booted --> If i boot my snapshot with the overlay, stat -f -c '%T' / gives overlayfs, but that's weird because, i have mount error in journalctl :

systemd-remount-fs[439]: /usr/bin/mount for / exited with exit status 32.
systemd[1]: Failed to start Remount Root and Kernel File Systems.

And i do not know really where i am once booted because/.snapshots is mounted on /. So i have /1, /2, /3, /4, /5 etc....i do not have a regular filesystem. There should be a temporary folder somewhere for the overlay, but it's like i did not booted inside the snapshot chosen in grub menu.

So until now, the key point was to put rw in /etc/default/grub, at least i can boot

[bkmo]

The rw kernel parameter is not needed as / should get remounted rw during init. You error seems to be pointing to the remount not being sucessful. What is your /etc/default/grub GRUB_CMDLINE_LINUX_DEFAULT= ? Als I am not sure why you have /tmp as tmpfs in fstab. Arch should be doing that by default with a systemd tmp.mount service. Just out of curiosity what is your default / subvolume? sudo btrfs subvolume get-default /

[HanM23]

GRUB_CMDLINE_LINUX_DEFAULT='cryptdevice=UUID=d916a464-4abf-46bd-b36a-e81a6381e0fc:luks-d916a464-4abf-46bd-b36a-e81a6381e0fc root=/dev/mapper/luks-d916a464-4abf-46bd-b36a-e81a6381e0fc udev.log_priority=3 nvidia-drm.modeset=1 nvidia_drm.fbdev=1'

sudo btrfs subvolume get-default /                                                                                                                           
ID 5 (FS_TREE)

Als I am not sure why you have /tmp as tmpfs in fstab. Arch should be doing that by default with a systemd tmp.mount service.

Indeed, i have this in my fstab:

tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0

This is like that in my distro by default Manjaro.

[bkmo]

I am trying to wrap my head around / being ID 5 and not subvol /endeavour/@. This is not what I am used to. Maybe you need it for dual boot. Also tmpfs in fstab on Endeavour just seems wrong to me. Tmp is not your issue, but I am not sure about the default subvol. I always see that you should set it to the proper root subvol even after a restore. Again I am not sure if this is needed or even correct for dual boot scenarios.

The following from Timeshift readme to set default subvol.

Make sure, that you have selected subvolume @ or /@ for root. You can check that executing script below, and if output is OK, then everything is alright.

grep -E '^[^#].+/\s+btrfs' /etc/fstab | \
grep -oE 'subvol=[^,]+' | \
cut -d= -f2 | \
grep -qE '^/?@$' && \
echo 'OK' || \
echo 'Not OK'

Default BTRFS subvolume must be /. You can make it using script below.

MP="$(mktemp -d)"
mount | awk '/on \/ type btrfs/{print $1}' | sudo xargs -I{} mount {} "$MP" && \
sudo btrfs subvolume set-default 5 "$MP"; \
sudo umount "$MP"

[HanM23]

I am trying to wrap my head around / being ID 5 and not subvol /endeavour/@

I get it, but, look my fstab :

/dev/mapper/luks-d916a464-4abf-46bd-b36a-e81a6381e0fc / btrfs subvol=/@,defaults,compress=zstd:1 0 0

Arch wiki says :

Mounting subvolume as root

To use a subvolume as the root mountpoint, either make it the default subvolume, or specify the subvolume via a kernel parameter using rootflags=subvol=/path/to/subvolume. Edit the root mountpoint in /etc/fstab and specify the mount option subvol=. Alternatively, the subvolume can be specified with its id, rootflags=subvolid=objectid as kernel parameter and subvolid=objectid as mount option in /etc/fstab. It is preferable to mount using subvol=/path/to/subvolume, rather than the subvolid, as the subvolid may change when restoring #Snapshots, requiring a change of mount configuration, or else the system will not boot.

Changing the default sub-volume

The default sub-volume is mounted if no subvol= mount option is provided. To change the default subvolume, do:

I do not have rootflags in kernel parameter but my fstab seems to be correct. That is my, i think, the right subvolume is mouted at boot eventhough get-deault=5.

But that's right, when using the overlayfs, it seems that it's not mounted correctly, in this case i tend to think that this is subvolid 5 which is mounted.

Your first script gives me OK. I do not understand your second script, because, yes, sudo btrfs subvolume get-default / gives me 5

[bkmo]

I have subvol=@ in my rootflags, rw also. Get default returns sudo btrfs subvolume get-default / ─╯ ID 256 gen 250313 top level 5 path @ I am not sure where your issue lies, but I am looking at mounting being an issue.

[HanM23]

I have subvol=@ in my rootflags, rw also.

This is interesting because, look, in my current subvolume @, in grub.cfg, i also have rootflags=@ rw

But, if i look inside grub-btrfs.cfg, i have rooflags=defaults, and i do not have rw for each snapshot

I will try setting de default subvolume.

If i do sudo btrfs subvolume list /

ID 257 gen 133285 top level 5 path @cache
ID 258 gen 133287 top level 5 path @log
ID 259 gen 23604 top level 5 path @swap
ID 265 gen 131818 top level 5 path @snapshots
ID 268 gen 23604 top level 265 path @snapshots/1/snapshot
ID 269 gen 23604 top level 265 path @snapshots/2/snapshot
ID 270 gen 23604 top level 265 path @snapshots/3/snapshot
ID 271 gen 23605 top level 265 path @snapshots/4/snapshot
ID 272 gen 23605 top level 265 path @snapshots/5/snapshot
ID 273 gen 23605 top level 265 path @snapshots/6/snapshot
ID 274 gen 23605 top level 265 path @snapshots/7/snapshot
ID 285 gen 23883 top level 265 path @snapshots/12/snapshot
ID 291 gen 23986 top level 265 path @snapshots/14/snapshot
ID 292 gen 26861 top level 265 path @snapshots/15/snapshot
ID 294 gen 40438 top level 265 path @snapshots/17/snapshot
ID 297 gen 54888 top level 265 path @snapshots/19/snapshot
ID 299 gen 63550 top level 265 path @snapshots/21/snapshot
ID 302 gen 90860 top level 265 path @snapshots/24/snapshot
ID 304 gen 110984 top level 265 path @snapshots/26/snapshot
ID 306 gen 112367 top level 265 path @snapshots/27/snapshot
ID 310 gen 113698 top level 265 path @snapshots/28/snapshot
ID 311 gen 117178 top level 265 path @snapshots/29/snapshot
ID 312 gen 133286 top level 5 path @
ID 313 gen 128395 top level 265 path @snapshots/30/snapshot
ID 320 gen 131831 top level 265 path @snapshots/31/snapshot

So, i should set it to 312, right ? Thank you

[CPU-Blanc]

I am trying to wrap my head around / being ID 5 and not subvol /endeavour/@. This is not what I am used to. Maybe you need it for dual boot. Also tmpfs in fstab on Endeavour just seems wrong to me. Tmp is not your issue, but I am not sure about the default subvol. I always see that you should set it to the proper root subvol even after a restore. Again I am not sure if this is needed or even correct for dual boot scenarios.

The following from Timeshift readme to set default subvol.

Make sure, that you have selected subvolume @ or /@ for root. You can check that executing script below, and if output is OK, then everything is alright.

grep -E '^[^#].+/\s+btrfs' /etc/fstab | \
grep -oE 'subvol=[^,]+' | \
cut -d= -f2 | \
grep -qE '^/?@$' && \
echo 'OK' || \
echo 'Not OK'

Default BTRFS subvolume must be /. You can make it using script below.

MP="$(mktemp -d)"
mount | awk '/on \/ type btrfs/{print $1}' | sudo xargs -I{} mount {} "$MP" && \
sudo btrfs subvolume set-default 5 "$MP"; \
sudo umount "$MP"

I think you're getting confused a little. It's me that's running /endeavour/@, not @HanM23 , as I'm the OP of this issue :P

[HanM23]

Oh, yes, that is right, my fstab

/dev/mapper/luks-d916a464-4abf-46bd-b36a-e81a6381e0fc /              btrfs   subvol=/@,defaults,compress=zstd:1 0 0
/dev/mapper/luks-d916a464-4abf-46bd-b36a-e81a6381e0fc /var/cache     btrfs   subvol=/@cache,defaults,compress=zstd:1 0 0
/dev/mapper/luks-d916a464-4abf-46bd-b36a-e81a6381e0fc /var/log       btrfs   subvol=/@log,defaults,compress=zstd:1 0 0

[bkmo]

Ahhhhh, I got lost. Sorry to hijack CPU-Blanc. We can take this private if you want.

[CPU-Blanc]

For me, the issue is present in both EndeavourOS and Arch as I'm multibooting from the same btrfs partition. My relevant fstab in Arch

UUID=244E-C5AA                                  /boot/efi                        vfat    fmask=0137,dmask=0027                                      >
UUID=913bf26c-4239-4ae3-a61f-54e24d0b01cc       /                               btrfs   subvol=/arch/@,defaults,noatime,compress=zstd              >
UUID=913bf26c-4239-4ae3-a61f-54e24d0b01cc       /var/cache                      btrfs   subvol=/arch/@cache,defaults,noatime,compress=zstd         >
UUID=913bf26c-4239-4ae3-a61f-54e24d0b01cc       /var/log                        btrfs   subvol=/arch/@log,defaults,noatime,compress=zstd           >
UUID=6c2cad87-8994-4339-b866-ed04c190a259       none                            swap    defaults    

Both in grub.cfg and grub-btrfs.cfg I can see the correct subvolume is set in the kernel root flags, however as @HanM23 pointed out, in grub-btrfs.cfg there is no rw flag specified for each snapshot

[CPU-Blanc]

Ahhhhh, I got lost. Sorry to hijack CPU-Blanc. We can take this private if you want.

Nono it's fine. I have a hunch this is connected somehow so pooling ideas here is a good idea

[HanM23]

grub-btrfs.cfg there is no rw flag specified for each snapshot

In fact, i would say this is normal because, after all, snapper take r/o snapshot. @CPU-Blanc , could you give us the result of sudo btrfs subvolume get-default / ? To see if you are in the same situation as me.

[CPU-Blanc]

grub-btrfs.cfg there is no rw flag specified for each snapshot

In fact, i would say this is normal because, after all, snapper take r/o snapshot. @CPU-Blanc , could you give us the result of sudo btrfs subvolume get-default / ? To see if you are in the same situation as me.

ID 5 (FS_TREE) is what it returns. Though it's to my understanding that the default is only used if no subvolume is specified, but a subvolume has been specified in the kernel parameter. Unless I'm mistaken?

[HanM23]

What's your kernel parameters ?

I think as you when i read the Arch wiki: The default sub-volume is mounted if no subvol= mount option is provided.

So, in theory, when you have subvol=/@ (in my case) or subvol=/endeavour/@ (in your case @CPU-Blanc ), it should be ok, unless things work differently when using an overlayfs, that's what i am thinking

[bkmo]

I have default set to @. This is also what the https://aur.archlinux.org/packages/snapper-rollback/ package does after a rollback. Are you SURE you want to rollback? Type 'CONFIRM' to continue: CONFIRM 2024-06-05 11:56:01,196 - INFO - mkdir -p '/tmp/btrfsroot' 2024-06-05 11:56:01,196 - INFO - mount -o subvolid=5 /dev/mapper/cryptroot /tmp/btrfsroot 2024-06-05 11:56:01,196 - INFO - mv /tmp/btrfsroot/@ /tmp/btrfsroot/@2024-06-05T11:56 2024-06-05 11:56:01,196 - INFO - btrfs subvolume snapshot /tmp/btrfsroot/@snapshots/4865/snapshot /tmp/btrfsroot/@ 2024-06-05 11:56:01,197 - INFO - btrfs subvolume set-default /tmp/btrfsroot/@ 2024-06-05 11:56:01,197 - INFO - [DRY-RUN MODE] Rollback to /tmp/btrfsroot/@snapshots/4865/snapshot complete. Reboot to finish 2024-06-05 11:56:01,197 - INFO - rm -rf /boot/* 2024-06-05 11:56:01,197 - INFO - cp -a /tmp/btrfsroot/@snapshots/4865/snapshot/.bootbackup/boot/* /boot/

[CPU-Blanc]

What's your kernel parameters ?

I think as you when i read the Arch wiki: The default sub-volume is mounted if no subvol= mount option is provided.

So, in theory, when you have subvol=/@ (in my case) or subvol=/endeavour/@ (in your case @CPU-Blanc ), it should be ok, unless things work differently when using an overlayfs, that's what i am thinking

rw rootflags=subvol=arch/@ nvidia-drm.modeset=1 nvidia-drm.fbdev=1 loglevel=3 quiet in grub.cfg. And essentially the same without rw in grub-btrfs.cfg.

I'm thinking along the same lines, is overlayfs somehow not respecting these kernel parameters and always using the default? But I'm not sure how that explains your case, where you're using the standard @ for root if I'm not mistaken?

[HanM23]

But I'm not sure how that explains your case, where you're using the standard @ for root if I'm not mistaken?

You are not mistaken, i am using @ for root. Well, this is my default installation with Manjaro. Does it seem not normal for you ?

[bkmo]

I agree that according to Arch Wiki the kernel rootflag should be all that is needed. Using overlayfs I have no problem. Only difference is I have a default subvolume of @.

[HanM23]

I have default set to @

I see, in fact, i did not change the default subvolume, i let it to be 5, because that is how my Manjaro installation is. I am going to test by setting this default subvolume to the right subvolume ID, and i will tell you the result.

[bkmo]

You have to mount / to subvolid=5 in order to be in the btrfs root to change the default. ex. mount -o subvolid=5 /dev/mapper/cryptroot /tmp/btrfsroot then set the default: btrfs subvolume set-default /tmp/btrfsroot/@

[bkmo]

There must be a reason behind snapper-rollback and Timeshift wanting the default set to the / subvolume.

[CPU-Blanc]

There must be a reason behind snapper-rollback and Timeshift wanting the default set to the / subvolume.

Do you mean / as in top filesystem of the btrfs partition, or as in system root /@ or whatever? Because my default is the top of the btrfs partition, ID 5.

The thing is snapper is fine with my setup. I can take snapshots, and restore them without any kind of issue what-so-ever. The only issue is when trying to boot into ro snapshots

[bkmo]

Yes the BTRFS root, the top of the btrfs ID 5. Snapper should be fine. Arch is not setup like Opensuse. Snapper rollback (the snapper command) should not be used because it does not mv/snapshot the subvolume, but sets it as default. The snapper-rollback AUR package is awesome. I have it patched to backup and restore /boot during the restore.

[CPU-Blanc]

I'm currently using btrfs-assistant to handle rollbacks, it handles them no problems. The only issue is booting via the grub menu

[HanM23]

I'm currently using btrfs-assistant to handle rollbacks, it handles them no problems. The only issue is booting via the grub menu

i do the same with btrfs assistant

[HanM23]

I have it patched to backup and restore /boot during the restore.

i see, i do not have a separate boot partition. I only have a /boot folder in my / partition, encrypted. So when i do a snapshot, /boot folder is snapshotted, and restored if i restore a snapshot.

[bkmo]

I also use btrfs-assistant, but restore is not working with overlayfs, and running from a live iso. I have an issue on the Gitlab. Snapper-rollback on the otherhand will work everywhere, and is patched for a boot restore to keep /boot the same as the restored snapshot. I have a separate /boot partition because I cannot use grub to unlock my Luks2 Argon2 encrypted partition. I use systemd-boot which requires separate boot, and grub just to boot snapshots. The boot restore can be toggled on/off in the .config.

[bkmo]

This also has default set to @ https://waylonwalker.com/setting-up-snapper-on-arch/

If you are interested in my snapper-rollback-root package that includes the bootbackup.hook: git clone https://github.com/bkmo/snapper-rollback-boot cd snapper-rollback-root makepkg -src

[HanM23]

I changed my default-subvolume to /@

Now, my issue is kind of solved because i can boot my r/o snapshots with overlayfs. I still have the remount errors at boot.

systemd-remount-fs[439]: /usr/bin/mount for / exited with exit status 32.
systemd[1]: Failed to start Remount Root and Kernel File Systems.

But then, once it had finished and i am within kde, it seems to be ok, even-though i do not do much tests. Now, changing the default-subvolume led to another issue, indeed :

• By default, my default-subvolume was 5
• Now, i set it to/@ with ID 312 (or whatever ID)

--> If i do a snapshot restoration withbtrfs-assistant: it backups /@ to /@.<date> and mv the snapshot (say /.snapshots/31/to/@, which is fine. But now, the default subvolume is still /@.<date> with ID 312 and your new/@ subvolume is not the default.

Therefore, each time i do a restoration, i have to set the new default subvolume to /@, lol (that's not funny, but i tend to think this is endless....). Maybe this is better to keep this default-subvolume to 5

[CPU-Blanc]

I'd be inclined to agree, although a sort of stop-gap, it doesn't help myself or anyone else in my position; where I cannot set to default subvolume to the OS root as I multiboot on the same partition.

Sounds like something in the chain is not respecting the kernel parameters and instead is using the default btrfs subvolume, which imo feels quite limiting.

[bkmo]

Using snapper-rollback will set the default for you. I am not a fan of btrfs-assistant for restores. I just don't trust it.

[HanM23]

The boot restore can be toggled on/off in the .config.

What do you mean by that ?

So in your setup, you have.

/boot encrypted partition / encrypted partition /home partition

you have systemd-boot and grub, how to you manage both ? How does your machine start, i do not get it :)

[bkmo]

I boot using systemd-boot, but I can change the boot order in bios to boot from Grub when I need to boot a snapshot. My /boot is mounted to the fat32 unencrypted partition. Only way for systemd-boot to work.

[bkmo]

I'd be inclined to agree, although a sort of stop-gap, it doesn't help myself or anyone else in my position; where I cannot set to default subvolume to the OS root as I multiboot on the same partition.

Sounds like something in the chain is not respecting the kernel parameters and instead is using the default btrfs subvolume, which imo feels quite limiting.

This is what I was saying earlier when I was talking about multi-boot. You will need a different default subvol every time you boot a different OS. The only reason I mentiond a default subvol is that was the only difference I have in my setup, and no /var/log issues. I know this doesn't make sense

[CPU-Blanc]

I have a hunch on where the issue might be occurring. Is it possible this is happening within the initramfs hook? My knowledge is a little lacking, but is it possible that during the overlayfs mounting step, it's simply mounting the btrfs partition without passing a subvol, and so it's mounting the default subvolume to it?