Default BE doesn't automatically boot on RAID0 pool

[ximGBu4cyQss5P]

ZFSBootMenu build source

Release EFI

ZFSBootMenu version

2.3.0

Boot environment distribution

OpenSUSE Tumbleweed

Problem description

First of all, great work on an awesome project! I've previously set up ZBM on a single disk on my laptop, and a multiboot with shared keys in a VM just to get somewhat familiar with the process. I ran into an issue trying to use a RAID0 pool, and I assume it's more likely that I've just done something wrong than an issue with ZBM.

I installed Tumbleweed on a RAID0 pool, and the usual install stuff went fine. But on the first reboot and boot into that pool, I didn't see the usual ZBM count down, and instead got a few repeating sequences of ^[[26~, then a brief splash of what looked like the ZBM countdown screen. Then I get the usual password prompt for zroot. Once I enter my password, my default (and only) BE doesn't boot, I get dropped into ZBM. Here, the first time I hit ENTER to boot my BE, I got dropped into the chroot for it. I assumed things were borked, but the pool and datasets looked fine, /boot looked good, the ZFS properties seemed to be set okay, and I even scrubbed the pool for fun with no errors. I rebooted and got dropped into ZBM instead of the BE booting again, but this time when I hit ENTER the BE booted fine. And that's been what's happening ever since.

The only real differences to anything I've done before that seem important are...

• It's a 2 disk RAID0 pool
• I duplicated the EFI partition contents to both disks
• I set the EFI partition to 4x as big as normal just in case I want to screw around with things that want to go into EFI later

Are there any gotchas or tips to be aware of for RAID0 pools?

Steps to reproduce

Here's the steps I took. This was done from a real Tumbleweed host on /dev/nvme0n1 that already uses ZFS for non-root stuff. Kernel 6.6.6, ZFS 2.2.2, ZBM 2.3.0.

```bash export ID="tumbleweed" export BOOT_DISK="/dev/nvme1n1" export BOOT_PART="1" export BOOT_DEVICE="${BOOT_DISK}p${BOOT_PART}" # Clear our disks wipefs -a /dev/nvme1n1 wipefs -a /dev/nvme2n1 sgdisk --zap-all /dev/nvme1n1 sgdisk --zap-all /dev/nvme2n1 # Create our partitions sgdisk -n 1:1m:+2048m -t 1:ef00 /dev/nvme1n1 sgdisk -n 2:0:-10m -t 2:bf00 /dev/nvme1n1 # Copy our disk layout from disk 1 to disk 2 sgdisk /dev/nvme1n1 -R /dev/nvme2n1 -G # Format our boot partitions mkfs.vfat -F32 /dev/nvme1n1p1 mkfs.vfat -F32 /dev/nvme2n1p1 # Temporarily create our key location / file so we can create the pool mkdir -p /etc/zfs/keys echo 'SOME_KEY' > /etc/zfs/keys/zroot.key # Create our RAID0 pool sudo zpool create -o ashift=12 \ -O compression=lz4 \ -O acltype=posixacl \ -O xattr=sa \ -O relatime=on \ -O encryption=aes-256-gcm \ -O keylocation=file:///etc/zfs/keys/zroot.key \ -O keyformat=passphrase \ -o autotrim=on \ -o compatibility=openzfs-2.1-linux \ -m none zroot /dev/disk/by-id/nvme-TEAM_TM8FP4004T_112303150060286-part2 /dev/disk/by-id/nvme-TEAM_TM8FP4004T_112304280070783-part2 # Nuke our temp key location / file rm -rf /etc/zfs/keys # Create our datasets zfs create -o mountpoint=none zroot/ROOT zfs create -o mountpoint=/ -o canmount=noauto zroot/ROOT/${ID} zfs create -o mountpoint=/home zroot/home zfs create -o mountpoint=/etc/zfs/keys zroot/keystore # Recreate our key where it belongs echo "SOME_KEY" > /etc/zfs/keys/zroot.key chmod 000 /etc/zfs/keys/zroot.key # Pool and dataset settings zfs set keylocation=file:///etc/zfs/keys/zroot.key zroot zfs set org.zfsbootmenu:commandline="quiet loglevel=4 rhgb" zroot/ROOT zfs set org.zfsbootmenu:keysource=zroot/keystore zroot zpool set bootfs=zroot/ROOT/${ID} zroot # Export and reimport our pool to start the fun mkdir /mnt/tumbleweed zpool export zroot zpool import -N -R /mnt/tumbleweed zroot zfs load-key -L prompt zroot zfs mount zroot/ROOT/${ID} zfs mount zroot/home zfs mount zroot/keystore # Make sure our filesystems show up mount | grep mnt # Update device symlinks udevadm trigger # Add our repos zypper --root /mnt/tumbleweed ar http://download.opensuse.org/tumbleweed/repo/oss/ oss zypper --root /mnt/tumbleweed ar http://download.opensuse.org/tumbleweed/repo/non-oss/ non-oss zypper --root /mnt/tumbleweed ar http://download.opensuse.org/update/tumbleweed/ update zypper --root /mnt/tumbleweed ar https://download.opensuse.org/repositories/filesystems/openSUSE_Tumbleweed/ filesystems zypper --root /mnt/tumbleweed --gpg-auto-import-keys ref zypper --root /mnt/tumbleweed lr -Pu # Install the enhanced_base pattern zypper --root /mnt/tumbleweed in --recommends -t pattern enhanced_base zypper --root /mnt/tumbleweed in --recommends zypper yast2 # should already be installed with enhanced_base cp /etc/hostid /mnt/tumbleweed/etc cp -L /etc/resolv.conf /mnt/tumbleweed/etc # Get ready and chroot mount -t proc proc /mnt/tumbleweed/proc mount -t sysfs sys /mnt/tumbleweed/sys mount -B /dev /mnt/tumbleweed/dev mount -t devpts pts /mnt/tumbleweed/dev/pts chroot /mnt/tumbleweed /bin/bash update-ca-certificates echo 'LANG=en_US.UTF-8' > /etc/locale.conf echo 'SOME_HOSTNAME' > /etc/hostname echo -e '127.0.1.1\tSOME_HOSTNAME' >> /etc/hosts passwd # Make sure dracut adds the zfs module to our initrd cat << EOF > /etc/dracut.conf.d/zol.conf nofsck="yes" add_dracutmodules+=" zfs " omit_dracutmodules+=" btrfs " install_items+=" /etc/zfs/keys/zroot.key " EOF # Add our /boot/efi partition to fstab cat << EOF >> /etc/fstab $( blkid | grep "$BOOT_DEVICE" | cut -d ' ' -f 2 ) /boot/efi vfat defaults 0 0 EOF cat /etc/fstab # Mount or boot partition (earlier just to silence an error while installing stuff below) mkdir -p /boot/efi mount /boot/efi # Install our kernel zypper in --recommends kernel-default kernel-default-devel kernel-firmware # The purge-kernels service just touched this file but there were no kernel symlinks to clean up so we'll remove it manually rm /boot/do_purge_kernels zypper in --recommends zfs zfs-kmp-default # Build our initrd dracut -f # Make sure our initrd has our zfs goodies inside lsinitrd /boot/initrd-$(uname -r) | grep -i zfs # If the dracut initrd above fails and says it can't find the zfs module, force reinstall the kernel, then run the dracut command again #zypper in -f --recommends kernel-default kernel-default-devel kernel-firmware #dracut -f # Without grub2 installed, the system doesn't symlink the installed kernels into /boot/, let's manually copy them there as a rescue kernel/initrd pair cp /lib/modules/$(uname -r)/vmlinuz /boot/vmlinuz-rescue cp /lib/modules/$(uname -r)/config /boot/config-rescue cp /lib/modules/$(uname -r)/System.map /boot/System.map-rescue cp /lib/modules/$(uname -r)/sysctl.conf /boot/sysctl.conf-rescue cp /boot/initrd-$(uname -r) /boot/initrd-rescue # Install NetworkManager and set it to run on next boot sudo zypper in --recommends NetworkManager systemctl enable NetworkManager.service # Install some handy utils with no external deps zypper in --no-recommends btop fd iotop nano nnn # Create our zypper locks for patterns we never want installed cat << EOF >> /etc/zypp/locks type: pattern match_type: exact case_sensitive: on solvable_name: laptop type: pattern match_type: exact case_sensitive: on solvable_name: office type: pattern match_type: exact case_sensitive: on solvable_name: kde_pim type: pattern match_type: exact case_sensitive: on solvable_name: games type: pattern match_type: exact case_sensitive: on solvable_name: multimedia EOF # Grab ZBM and put it where it belongs mkdir -p /boot/efi/EFI/ZBM curl -o /boot/efi/EFI/ZBM/VMLINUZ.EFI -L https://get.zfsbootmenu.org/efi cp /boot/efi/EFI/ZBM/VMLINUZ.EFI /boot/efi/EFI/ZBM/VMLINUZ-BACKUP.EFI mount -t efivarfs efivarfs /sys/firmware/efi/efivars zypper in efibootmgr # should already be installed # echo $BOOT_DISK echo $BOOT_PART efibootmgr -c -d "$BOOT_DISK" -p "$BOOT_PART" \ -L "ZFSBootMenu (Backup)" \ -l '\EFI\ZBM\VMLINUZ-BACKUP.EFI' efibootmgr -c -d "$BOOT_DISK" -p "$BOOT_PART" \ -L "ZFSBootMenu" \ -l '\EFI\ZBM\VMLINUZ.EFI' # Unmount our /boot/efi location and copy partition 1 from drive 1 over to partition 1 of drive 2 (this will take a little while since we made it a 2GB partition above) umount /dev/nvme1n1p1 dd if=/dev/nvme1n1p1 of=/dev/nvme2n1p1 exit umount -n -R /mnt/tumbleweed zpool export zroot reboot ```

[zdykstra]

RAID0 doesn't matter; If OpenZFS can import it, ZFSBootMenu can boot it. How many zpools does this system have? If it's more than one, and bootfs is set for zroot, setting zbm.prefer=zroot on the ZFSBootMenu command line via zbm-kcl should resolve the issue.

[ximGBu4cyQss5P]

Thanks. There's 2 pools on that machine. The RAID0 NVMe one I created for ZBM, and a 4 disk RAID10 for data storage. I added zbm.prefer=zroot to the default command line but I'm still seeing the same result where I get dropped into ZBM instead of it automatically booting zroot/ROOT/tumbleweed which is set as bootfs for that pool.

[zdykstra]

Where exactly did you add zbm.prefer=zroot ? If you added it to org.zfsbootmenu:commandline on a pool, that won't do anything. It needs to be added to the kernel command line for the EFI bundle that you're booting, via zbm-kcl -e /path/to/your/efi.

[ximGBu4cyQss5P]

I had updated the command line for the ZBM EFI bundle, so it now reads quiet loglevel=0 nomodeset zbm.prefer=zroot. I thought maybe the system was actually finding ZBM on nvme2n1 instead of nvme1n1 since the EFI partition is duplicated to both. But I updated those command lines too with zbm-kcl, so now all 4 possible EFI bundles (main and backup on both disks) have the same command line. But I'm still getting the same results. The weird repeated ^[[26~ instead of the countdown, then after my password I get dropped into ZBM.

[zdykstra]

Can you upload a copy of the EFI bundle that you're using to boot your system? Can you also share the output of zpool get bootfs from your booted machine?

[ximGBu4cyQss5P]

Sure thing. Here's the output of zpool get bootfs.

NAME PROPERTY VALUE SOURCE HDD bootfs - default zroot bootfs zroot/ROOT/tumbleweed local

Is there a specific place you want the EFI bundle uploaded? It looks like the max here is 25MB.

[zdykstra]

Dropbox or a Google files account would work. Otherwise you can make a repository on GitHub and upload it there.

[ahesford]

Can you also post the output of zreport? A photo of the console is fine if you don't want to jump through hoops to capture the text.

[ximGBu4cyQss5P]

Sure thing, here you go...

zreport.txt

VMLINUZ.EFI

MD5 for the EFI bundle: 93553b9d07d8eed4c85edead5f7abb0b

[zdykstra]

Thank you for uploading both of those! I've retrieved both of the files.

The zreport data looks fine at first glance. Do you connect to ZBM through an iDRAC/IPMI interface, or is it directly through a local display/keyboard?

If you're willing to reboot this system again, you could try setting 'zbm.skip' on the EFI's KCL. That'll completely bypass the countdown screen and should result in it directly booting your bootfs value on the preferred pool. Obviously this isn't a permanent solution, but it can possibly help us narrow down what might be happening here.

[ximGBu4cyQss5P]

I finally figured this out, and the culprit is interesting and surprising to me. Your question piqued my interest. While this particular machine doesn't have an IPMI interface, last night I began suspecting the problem may have something to do with the HDMI KVM that it's hooked up to since it seems like ZBM is getting some kind of spurious input. After many many reboots today testing different cabling configurations, it turns out the KVM isn't to blame. It was the Logitech H820e headset. It seems like ZBM is getting some kind of USB input from this device, and it happens even without the KVM involved and the headset hooked up directly to the computer. I thought it may be some weird chipset interaction quirk on this old dual Xeon Z840 workstation, but the exact same thing happens on a much newer AMD Zen 3 laptop where I have a minimal Xubutnu install with ZBM. I also happen to have a second one of these Logitech H820e headsets in the house in my wife's office so I tested that one too. Same result, so it's not just some weird defect in mine. Short of the "throw the headset in the trash and buy another one" solution, is there any way to blacklist / block certain devices by USB device ID within the ZBM EFI bundle similar to how I might do it with udev rules in a fully booted system? Also, zbm.prefer=zroot isn't needed in the end once the problematic device is removed, ZBM does the right thing with the 2 pool setup so that's another positive.

[ahesford]

There is no way for you to ignore this device in a pre-built release image, but you can build a custom image with whatever udev rules or driver blacklists are appropriate.

[zdykstra]

I'm glad you were able to track down what was happening! I suspected that something was giving you spurious input, interferring with the countdown screen. I would not have guessed in a million years that it was a USB headset!

ZFSBootMenu itself doesn't have a mechanism to blacklist a USB device. If you build a custom image, you can add the appropriate udev rules file to it. https://docs.zfsbootmenu.org/en/v2.3.x/guides/general/container-building.html is probably the best option for you - building on top of openSuSE could be tough.