sysroot.readonly caused unbootable system

[aengelke]

After an unknown system failure (but that might just be a spurious correlation; that was the second attempt to boot Fedora Silverblue 37 after an upgrade from 36), the next boot failed here; the system kept looping with mounting and unmounting the file system (and IIRC the logs indicated attempts to start an emergency shell -- which didn't succeed). Changing the config to readonly=false from a live system made the system bootable again.

1. Why is the check fatal? (IOW, how broken is my system now?)
2. What is the best way to debug this kind of problem without an external live system?

System: Fedora Silverblue 37.20221130.0, libostree: Version '2022.6' Git dd030c57ca76a08d2db33902d08a4050a4e506eb.

[jlebon]

Sounds like something might've gone wrong with the migration script for the read-only sysroot change. Check the journal logs for fedora-silverblue-readonly-sysroot.service and verify that you have the rw karg set.

[aengelke]

The script ran and rw is shown in rpm-ostree kargs; but apparently did not end up in the grub config (maybe because of the unclean shutdown?).

$ journalctl -u fedora-silverblue-readonly-sysroot.service
XX systemd[1]: Starting fedora-silverblue-readonly-sysroot.service - Fedora Silverblue Read-Only Sysroot Migration...
XX fedora-silverblue-readonly-sysroot[847]: Setting up the sysroot.readonly option in the ostree repo config
XX systemd[1]: Finished fedora-silverblue-readonly-sysroot.service - Fedora Silverblue Read-Only Sysroot Migration.
-- Boot 0f49f24f8a3d49f19a2de3e45388c14f --
XX systemd[1]: fedora-silverblue-readonly-sysroot.service - Fedora Silverblue Read-Only Sysroot Migration was skipped because of a failed condition check (ConditionPathExists=!/var/lib/.fedora_silverblue_readonly_sysroot).
$ rpm-ostree kargs
resume=UUID=c5b595c8-xxxx-xxxx-xxxx-xxxxxxxxxxxx rhgb root=UUID=27c26a30-xxxx-xxxx-xxxx-xxxxxxxxxxxx usbcore.autosuspend=-1 quiet ostree=/ostree/boot.0/fedora-workstation/719ea517590ef51ffc12c5bf3b43611f868a0dfd008d6a33933df9481237f808/0 rw

grub.cfg excerpt:

menuentry 'Fedora Linux 37.20221130.0 (Silverblue) (ostree:0)' --class gnu-linux --class gnu --class os --unrestricted 'ostree-0-5d31793d-2b5d-49be-bb9b-0cbf2b57acf3' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod ext2
search --no-floppy --fs-uuid --set=root 5d31793d-2b5d-49be-bb9b-0cbf2b57acf3
linuxefi /ostree/fedora-workstation-719ea517590ef51ffc12c5bf3b43611f868a0dfd008d6a33933df9481237f808/vmlinuz-6.0.10-300.fc37.x86_64 resume=UUID=c5b595c8-xxxx-xxxx-xxxx-xxxxxxxxxxxx rhgb root=UUID=27c26a30-xxxx-xxxx-xxxx-xxxxxxxxxxxx
 usbcore.autosuspend=-1 quiet ostree=/ostree/boot.0/fedora-workstation/719ea517590ef51ffc12c5bf3b43611f868a0dfd008d6a33933df9481237f808/0
initrdefi /ostree/fedora-workstation-719ea517590ef51ffc12c5bf3b43611f868a0dfd008d6a33933df9481237f808/initramfs-6.0.10-300.fc37.x86_64.img
}

[jlebon]

Hmm, it seems like you're not using blscfg. Do you have

insmod blscfg
blscfg

in your GRUB config?

But anyway, try rerunning another e.g. upgrade to regenerate the GRUB config. The rw karg should show up then.

[aengelke]

Hmm, it seems like you're not using blscfg. Do you have [...] in your GRUB config?

Yes, it comes in from 10_linux.