Unable to boot into a snapshot since kernel 6.8 (read-only), but i can with 6.6

Hi,

I cannot boot into a snapshot since kernel 6.8. I have two kernels installed in my system 6.6 lts and 6.8. Presets for both are exactly the same, and i can boot into any snapshot if i choose kernel 6.6 in the grub menu after i have chosen the snapshot itself.

If i boot on a snapshot with kernel 6.8, it does not want to start KDE, i can switch with ctrl+alt F to tty, i can then login in, and when i want to start KDE with startx i have the following message :

Can't move /var/log/Xorg.0.log to /var/log/Xorg.0.log.old. This is surprising because/var/log is a subvolume, and i do not make any snapshot of it, only the subvolume /. By checking the logs, yes, my snapshot's filesystem is read-only, therefore i cannot boot to kde. But i really do not understand why,

Once again i do not have this issue at all with 6.6, i can boot my snapshots normally with it.

I do not know really how to investigate more. Do you have any nice idea ? I have Manjaro/KDE with Snapper and Btrfs-Assistant.

Thank you

*Here is the hook line from mkinitcpio.conf :

HOOKS=(base udev autodetect microcode kms modconf block keyboard keymap consolefont plymouth encrypt filesystems grub-btrfs-overlayfs)

*Here is my config

#!/usr/bin/env bash

GRUB_BTRFS_VERSION=4.12-master-2023-04-28T16:26:00+00:00

# Disable grub-btrfs.
# Default: "false"
#GRUB_BTRFS_DISABLE="true"

# Name appearing in the Grub menu.
# Default: "Use distribution information from /etc/os-release."
#GRUB_BTRFS_SUBMENUNAME="Arch Linux snapshots"

# Custom title.
# Shows/Hides "date" "snapshot" "type" "description" in the Grub menu, custom order available.
# Default: ("date" "snapshot" "type" "description")
#GRUB_BTRFS_TITLE_FORMAT=("date" "snapshot" "type" "description")

# Limit the number of snapshots populated in the GRUB menu.
# Default: "50"
#GRUB_BTRFS_LIMIT="50"

# Sort the found subvolumes by "ogeneration" or "generation" or "path" or "rootid".
# # See Sorting section to https://btrfs.wiki.kernel.org/index.php/Manpage/btrfs-subvolume#SUBCOMMAND
# "-rootid" means list snapshot by new ones first.
# Default: "-rootid"
#GRUB_BTRFS_SUBVOLUME_SORT="+ogen,-gen,path,rootid"

# Show snapshots found during run "grub-mkconfig"
# Default: "true"
#GRUB_BTRFS_SHOW_SNAPSHOTS_FOUND="false"

# Show Total of snapshots found during run "grub-mkconfig"
# Default: "true"
#GRUB_BTRFS_SHOW_TOTAL_SNAPSHOTS_FOUND="true"

# By default, "grub-btrfs" automatically detects most existing kernels.
# If you have one or more custom kernels, you can add them here.
# Default: ("")
#GRUB_BTRFS_NKERNEL=("kernel-custom" "vmlinux-custom")

# By default, "grub-btrfs" automatically detects most existing initramfs.
# If you have one or more custom initramfs, you can add them here.
# Default: ("")
#GRUB_BTRFS_NINIT=("initramfs-custom.img" "initrd-custom.img" "otherinit-custom.gz")

# By default, "grub-btrfs" automatically detects most existing microcodes.
# If you have one or more custom microcodes, you can add them here.
# Default: ("")
#GRUB_BTRFS_CUSTOM_MICROCODE=("custom-ucode.img" "custom-uc.img "custom_ucode.cpio")

# Additonal kernel command line parameters that should be passed to the kernel
# when booting a snapshot.
# For dracut based distros this could be useful to pass "rd.live.overlay.overlayfs=1"
# or "rd.live.overlay.readonly=1" to the Kernel for booting snapshots read only.
# Default: ""
#GRUB_BTRFS_SNAPSHOT_KERNEL_PARAMETERS="rd.live.overlay.overlayfs=1"

# Comma seperated mount options to be used when booting a snapshot.
# They can be defined here as well as in the "/" line inside the respective snapshots'
# "/etc/fstab" files.  Mount options found in both places are combined, and this variable
# takes priority over `fstab` entries.
# NB: Do NOT include "subvol=..." or "subvolid=..." here.
# Default: ""
#GRUB_BTRFS_ROOTFLAGS="space_cache,commit=10,norecovery"

# Ignore specific path during run "grub-mkconfig".
# Only exact paths are ignored.
# e.g : if `specific path` = @, only `@` snapshot will be ignored.
# Default: ("@")
GRUB_BTRFS_IGNORE_SPECIFIC_PATH=("@")

# Ignore prefix path during run "grub-mkconfig".
# Any path starting with the specified string will be ignored.
# e.g : if `prefix path` = @, all snapshots beginning with "@/..." will be ignored.
# Default: ("var/lib/docker" "@var/lib/docker" "@/var/lib/docker")
GRUB_BTRFS_IGNORE_PREFIX_PATH=("var/lib/docker" "@var/lib/docker" "@/var/lib/docker")

# Ignore specific type/tag of snapshot during run "grub-mkconfig".
# For snapper:
# Type = single, pre, post.
# For Timeshift:
# Tag = boot, ondemand, hourly, daily, weekly, monthly.
# Default: ("")
#GRUB_BTRFS_IGNORE_SNAPSHOT_TYPE=("")

# Ignore specific description of snapshot during run "grub-mkconfig".
# e.g: timeline
# Default: ("")
#GRUB_BTRFS_IGNORE_SNAPSHOT_DESCRIPTION=("")

# By default "grub-btrfs" automatically detects your boot partition,
# either located at the system root or on a separate partition or in a subvolume,
# Change to "true" if your boot partition isn't detected as separate.
# Default: "false"
#GRUB_BTRFS_OVERRIDE_BOOT_PARTITION_DETECTION="true"

# Location of the folder containing the "grub.cfg" file.
# Might be grub2 on some systems.
# Default: "/boot/grub"
#GRUB_BTRFS_GRUB_DIRNAME="/boot/grub2"

# Location of kernels/initramfs/microcode.
# Use by "grub-btrfs" to detect the boot partition and the location of kernels/initrafms/microcodes.
# Default: "/boot"
#GRUB_BTRFS_BOOT_DIRNAME="/boot"

# Location where grub-btrfs.cfg should be saved.
# Some distributions (like OpenSuSE) store those files at the snapshot directory
# instead of boot. Be aware that this direcory must be available for grub during
# startup of the system.
# Default: $GRUB_BTRFS_GRUB_DIRNAME
#GRUB_BTRFS_GBTRFS_DIRNAME="/boot/grub"

# Location of the directory where Grub searches for the grub-btrfs.cfg file.
# Some distributions (like OpenSuSE) store those file at the snapshot directory
# instead of boot. Be aware that this direcory must be available for grub during
# startup of the system.
# Default: "\${prefix}" # This is a grub variable that resolves to where grub is
# installed. (like /boot/grub, /boot/efi/grub)
# NOTE: If variables of grub are used here (like ${prefix}) they need to be escaped
# with `\` before the `$`
#GRUB_BTRFS_GBTRFS_SEARCH_DIRNAME="\${prefix}"

# Name/path of grub-mkconfig command, use by "grub-btrfs.service"
# Might be 'grub2-mkconfig' on some systems (Fedora ...)
# Default paths are /sbin:/bin:/usr/sbin:/usr/bin,
# if your path is missing, report it on the upstream project.
# For example, on Fedora : "/sbin/grub2-mkconfig"
# You can use only name or full path.
# Default: grub-mkconfig
#GRUB_BTRFS_MKCONFIG=/usr/bin/grub2-mkconfig

# Name of grub-script-check command, use by "grub-btrfs"
# Might be 'grub2-script-check' on some systems (Fedora ...)
# For example, on Fedora : "grub2-script-check"
# Default: grub-script-check
#GRUB_BTRFS_SCRIPT_CHECK=grub2-script-check

# Path of grub-mkconfig_lib file, use by "grub-btrfs"
# Might be '/usr/share/grub2/grub-mkconfig_lib' on some systems (Opensuse ...)
# Default: /usr/share/grub/grub-mkconfig_lib
#GRUB_BTRFS_MKCONFIG_LIB=/usr/share/grub2/grub-mkconfig_lib

# Password protection management for submenu,snapshots
# Refer to the Grub documentation https://www.gnu.org/software/grub/manual/grub/grub.html#Authentication-and-authorisation
# and this comment https://github.com/Antynea/grub-btrfs/issues/95#issuecomment-682295660
#
# Add authorized usernames separate by comma (foo,bar)
# When Grub's password protection is enabled, the superuser is authorized by default, it isn't necessary to add it
# Default: ""
#GRUB_BTRFS_PROTECTION_AUTHORIZED_USERS="foo,bar"
#
# Disable authentication support for submenu of Grub-btrfs only (--unrestricted)
# doesn't work if GRUB_BTRFS_PROTECTION_AUTHORIZED_USERS isn't empty
# Default: "false"
#GRUB_BTRFS_DISABLE_PROTECTION_SUBMENU="true"

I get some more information once i boot into my snapshot in tty :

lsinitcpio -a /boot/initramfs-6.8-x64_64.img

==> Image: /boot/initramfs-6.8-x86_64.img
==> Created with mkinitcpio 38.1
==> Kernel: 6.8.3-1-MANJARO
==> Early CPIO: 220 KiB
==> Size: 62.51 MiB
==> Compressed with: gzip
  -> Uncompressed size: 125 MiB (.500 ratio)
  -> Estimated decompression time: 0,659s

==> Included modules:
  842                 cast6-avx-x86_64        echainiv            nhpoly1305              sm3-avx-x86_64
  842_compress            cast6_generic           ecrdsa_generic          nhpoly1305-avx2         sm3_generic
  842_decompress          cast_common             encrypted-keys          nhpoly1305-sse2         sm4
  adiantum            cbc                 essiv               nouveau             sm4-aesni-avx-x86_64
  aegis128            ccm                 fcrypt              nvme                sm4-aesni-avx2-x86_64
  aegis128-aesni          ccp                 gcm                 nvme-auth           sm4_generic
  aes_ti              ccp-crypto              geniv               nvme-core           spi-intel
  aesni-intel             cec                 gf128mul            overlay             spi-intel-pci
  af_alg              chacha-x86_64           ghash-clmulni-intel         padlock-aes             streebog_generic
  algif_aead              chacha20poly1305        ghash-generic           padlock-sha             tcrypt
  algif_hash              chacha_generic          gpu-sched           pcbc                tee
  algif_rng           chcr                hctr2               pcrypt              tls
  algif_skcipher          cmac                hid-generic             pkcs8_key_parser        trusted
  amlogic-gxl-crypto          crc16               i2c-algo-bit            poly1305-x86_64         ttm
  ansi_cprng              crc32-pclmul            i8042               poly1305_generic        twofish-avx-x86_64
  aria-aesni-avx-x86_64       crc32_generic           iaa_crypto              polyval-clmulni         twofish-x86_64
  aria-aesni-avx2-x86_64      crc32c-intel [explicit]     idxd                polyval-generic         twofish-x86_64-3way
  aria-gfni-avx512-x86_64     crc32c_generic          idxd_bus            pps_core            twofish_common
  aria_generic            crc8                intel_qat           ptp                 twofish_generic
  asn1_encoder            crct10dif-pclmul        keywrap             qat_420xx           uas
  async_memcpy            cryptd              libarc4             qat_4xxx            usb-storage
  async_pq            crypto_engine           libchacha           qat_c3xxx           usbhid
  async_raid6_recov       crypto_null             libchacha20poly1305         qat_c3xxxvf             video
  async_tx            crypto_safexcel         libcrc32c           qat_c62x            virtio_blk
  async_xor           crypto_simd             libcurve25519           qat_c62xvf              virtio_crypto
  atkbd               crypto_user             libcurve25519-generic       qat_dh895xcc            virtio_scsi
  atmel-ecc           cts                 libdes              qat_dh895xccvf          vivaldi-fmap
  atmel-i2c           curve25519-generic          libpoly1305             raid6_pq            vmac
  atmel-sha204a           curve25519-x86_64       libps2              raid6test           vmd
  authenc             cxgb4               lrw                 rmd160              wmi
  authencesn              deflate             lz4                 seqiv               wp512
  blake2b_generic         des3_ede-x86_64         lz4_compress            serio               xcbc
  blowfish-x86_64         des_generic             lz4hc               serio_raw           xctr
  blowfish_common         dm-bufio            lz4hc_compress          serpent-avx-x86_64          xhci-pci
  blowfish_generic        dm-crypt            lzo                 serpent-avx2            xhci-pci-renesas
  btrfs               dm-integrity            lzo-rle             serpent-sse2-x86_64         xor
  camellia-aesni-avx-x86_64   dm-mod              md4                 serpent_generic         xts
  camellia-aesni-avx2         drm_display_helper          michael_mic             sha1-ssse3              xxhash_generic
  camellia-x86_64         drm_exec            mmc_block           sha256-ssse3
  camellia_generic        drm_gpuvm           mmc_core            sha512-ssse3
  cast5-avx-x86_64        drm_ttm_helper          mxm-wmi             sm2_generic
  cast5_generic           ecdh_generic            n5pf                sm3

==> Included binaries:
  blkid           cryptsetup          loginctl        plymouthd       systemd-tmpfiles
  btrfs           dmsetup         mount           setfont         udevadm
  busybox         kmod            plymouth        switch_root

==> Early hook run order:
  udev

==> Hook run order:
  udev keymap plymouth encrypt

==> Late hook run order:
  plymouth grub-btrfs-overlayfs

==> Cleanup hook run order:
  udev

==> Emergency hook run order:
  plymouth

dmesg | grep overlay
[    5.316566] overlayfs: "xino" feature enabled using 2 upper inode bits.

findmnt


TARGET                        SOURCE                                                             FSTYPE     OPTIONS
/                             rootfs                                                             overlay    rw,relatime,lowerdir=/tmp.JoTBFm,upperdir=/tmp.vQzsno/upper,workdir=/tmp.vQzsno/work,uuid=on
├─/proc                       proc                                                               proc       rw,nosuid,nodev,noexec,relatime
│ └─/proc/sys/fs/binfmt_misc  systemd-1                                                          autofs     rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3856
├─/sys                        sys                                                                sysfs      rw,nosuid,nodev,noexec,relatime
│ ├─/sys/firmware/efi/efivars efivarfs                                                           efivarfs   rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/security      securityfs                                                         securityfs rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/cgroup            cgroup2                                                            cgroup2    rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot
│ ├─/sys/fs/pstore            pstore                                                             pstore     rw,nosuid,nodev,noexec,relatime
│ ├─/sys/fs/bpf               bpf                                                                bpf        rw,nosuid,nodev,noexec,relatime,mode=700
│ ├─/sys/kernel/debug         debugfs                                                            debugfs    rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/tracing       tracefs                                                            tracefs    rw,nosuid,nodev,noexec,relatime
│ ├─/sys/kernel/config        configfs                                                           configfs   rw,nosuid,nodev,noexec,relatime
│ └─/sys/fs/fuse/connections  fusectl                                                            fusectl    rw,nosuid,nodev,noexec,relatime
├─/dev                        dev                                                                devtmpfs   rw,nosuid,relatime,size=8068464k,nr_inodes=2017116,mode=755,inode64
│ ├─/dev/pts                  devpts                                                             devpts     rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
│ ├─/dev/shm                  tmpfs                                                              tmpfs      rw,nosuid,nodev,inode64
│ ├─/dev/hugepages            hugetlbfs                                                          hugetlbfs  rw,nosuid,nodev,relatime,pagesize=2M
│ └─/dev/mqueue               mqueue                                                             mqueue     rw,nosuid,nodev,noexec,relatime
├─/run                        run                                                                tmpfs      rw,nosuid,nodev,relatime,mode=755,inode64
│ └─/run/user/1000            tmpfs                                                              tmpfs      rw,nosuid,nodev,relatime,size=1621584k,nr_inodes=405396,mode=700,uid=1000,gid=1000,inode64
├─/.snapshots                 /dev/mapper/luks-d916a464-4abf-46bd-b36a-e81a6381e0fc[/@snapshots] btrfs      ro,noatime,compress=zstd:1,ssd,space_cache=v2,subvolid=265,subvol=/@snapshots
├─/swap                       /dev/mapper/luks-d916a464-4abf-46bd-b36a-e81a6381e0fc[/@swap]      btrfs      ro,noatime,compress=zstd:1,ssd,space_cache=v2,subvolid=259,subvol=/@swap
├─/var/cache                  /dev/mapper/luks-d916a464-4abf-46bd-b36a-e81a6381e0fc[/@cache]     btrfs      ro,relatime,compress=zstd:1,ssd,space_cache=v2,subvolid=257,subvol=/@cache
├─/var/log                    /dev/mapper/luks-d916a464-4abf-46bd-b36a-e81a6381e0fc[/@log]       btrfs      ro,relatime,compress=zstd:1,ssd,space_cache=v2,subvolid=258,subvol=/@log
├─/tmp                        tmpfs                                                              tmpfs      rw,noatime,inode64
├─/boot/efi                   /dev/nvme1n1p1                                                     vfat       rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro
├─/home                       /dev/mapper/luks-c3e0e757-8f71-4642-bce0-fa52246af02a              ext4       rw,relatime
├─/mnt/storage                /dev/mapper/luks-2aafa376-c7c9-4928-b649-a853af9ea859              ext4       rw,relatime

Does anyone also have some issues with grub-btrfs-overlayfs and kernel 6.8 ?

Thanks

Antynea / grub-btrfs

Unable to boot into a snapshot since kernel 6.8 (read-only), but i can with 6.6 #328