Open Ryushin opened 8 years ago
Richard Yao mentioned that it had to do with It has to do with grub2-probe failing silently and that the problem file was /etc/grub.d/10_linux.
@Ryushin and I did some debugging on his system. /usr/sbin/grub-probe --device /dev/sda1 --target=fs
is not detecting ZFS. His grub2-probe is linked to zfs, but it isn't detecting it:
linux-vdso.so.1 (0x00007ffd2edb3000)
libzfs.so.2 => /lib64/libzfs.so.2 (0x00007f388b117000)
libnvpair.so.1 => /lib64/libnvpair.so.1 (0x00007f388af01000)
libc.so.6 => /lib64/libc.so.6 (0x00007f388ab66000)
libzfs_core.so.1 => /lib64/libzfs_core.so.1 (0x00007f388a961000)
libzpool.so.2 => /lib64/libzpool.so.2 (0x00007f388a462000)
libuutil.so.1 => /lib64/libuutil.so.1 (0x00007f388a24d000)
libm.so.6 => /lib64/libm.so.6 (0x00007f3889f49000)
libblkid.so.1 => /lib64/libblkid.so.1 (0x00007f3889d06000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3889aea000)
/lib64/ld-linux-x86-64.so.2 (0x00007f388b35b000)
libz.so.1 => /lib64/libz.so.1 (0x00007f38898d4000)
librt.so.1 => /lib64/librt.so.1 (0x00007f38896cc000)
libuuid.so.1 => /lib64/libuuid.so.1 (0x00007f38894c7000)
My guess is that he has the latest feature flags on his pool and grub2-probe only reports zfs when it fully understands the disk format, despite it not needing to know anything about the disk format in this situation.
Updated information on my systems. This is occurring on three systems. All three systems are configured similarly.
System has GRUB installed to a mdraid1 device made up of two 16GB USB sticks. GRUB is install in the MBR of both USB sticks. A 1GB partition created on each USB stick that is mirrored using mdraid1. /boot is installed on this partition.
System has two pools. rpool and netshares:
zfs list: NAME USED AVAIL REFER MOUNTPOINT netshares 7.44T 46.1T 219K /netshares netshares/movies 7.44T 46.1T 7.44T /netshares/movies netshares/temp 219K 46.1T 219K /netshares/temp netshares/timemachine 219K 46.1T 219K /netshares/timemachine rpool 74.7G 4.12T 192K none rpool/ROOT 9.54G 4.12T 7.85G / rpool/speed_test 50.9G 4.12T 192K /speed_test rpool/virtual_machines 14.2G 4.12T 14.2G /var/lib/libvirt
zpool status -v: pool: netshares state: ONLINE scan: none requested config:
NAME STATE READ WRITE CKSUM
netshares ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
wwn-0x5000cca249d1979e ONLINE 0 0 0
wwn-0x5000cca24cea12a6 ONLINE 0 0 0
wwn-0x5000cca24cea415b ONLINE 0 0 0
wwn-0x5000cca249d084aa ONLINE 0 0 0
wwn-0x5000cca24ce90c6a ONLINE 0 0 0
wwn-0x5000cca24ce83a07 ONLINE 0 0 0
wwn-0x5000cca24cea0c42 ONLINE 0 0 0
wwn-0x5000cca24cea1261 ONLINE 0 0 0
wwn-0x5000cca249d168bc ONLINE 0 0 0
wwn-0x5000cca24cebeb5b ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
wwn-0x5000cca24ce7585e ONLINE 0 0 0
wwn-0x5000cca249d13c38 ONLINE 0 0 0
wwn-0x5000cca24cec2348 ONLINE 0 0 0
wwn-0x5000cca24ceb91ba ONLINE 0 0 0
wwn-0x5000cca24cec23f1 ONLINE 0 0 0
wwn-0x5000cca24cec23e9 ONLINE 0 0 0
wwn-0x5000cca24ce92b76 ONLINE 0 0 0
wwn-0x5000cca24ce9864a ONLINE 0 0 0
wwn-0x5000cca249d1478f ONLINE 0 0 0
wwn-0x5000cca24ce85bac ONLINE 0 0 0
errors: No known data errors
pool: rpool state: ONLINE scan: none requested config:
NAME STATE READ WRITE CKSUM
rpool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
wwn-0x5000cca0320d7c60 ONLINE 0 0 0
wwn-0x5000cca0320d7ab8 ONLINE 0 0 0
wwn-0x5000cca0320d84d0 ONLINE 0 0 0
wwn-0x5000cca0320d87c8 ONLINE 0 0 0
wwn-0x5000cca0320d870c ONLINE 0 0 0
wwn-0x5000cca0320d7310 ONLINE 0 0 0
Something like this could work here:
diff --git a/util/grub.d/10_linux.in b/util/grub.d/10_linux.in
index 00d1931..b9a39a6 100644
--- a/util/grub.d/10_linux.in
+++ b/util/grub.d/10_linux.in
@@ -60,5 +60,5 @@ case x"$GRUB_FS" in
fi;;
xzfs)
- rpool=`${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true`
+ rpool=`blkid -p -o value -s LABEL /dev/sda1 /dev/sdb1 | uniq 2>/dev/null || true`
bootfs="`make_system_path_relative_to_its_root / | sed -e "s,@$,,"`"
LINUX_ROOT_DEVICE="ZFS=${rpool}${bootfs}"
It can be applied to /etc/grub.d/10_linux
on a live system.
That said, this is not quite right because there is likely a chance that ${GRUB_DEVICE}
would have multiple devices from different pools that I have not ruled out, but this likely only makes sense in the context of the Ubuntu PPA's GRUB2 patchset that uses blkid to avoid the necessity of linking to libzfs. I am unfamiliar with that patch set, so I did not make an attempt to handle that edge case.
I'm running Debian (Devuan) actually and the patch works great for me. Thanks much.
The patch really only worked for certain situations. I think a requirement would be to have the zpool bootfs parameter set. I wrote a new patch that works against the current version (2.02~beta2-22+deb8) of grub in Debian Jessie.
diff -c /tmp/10_linux.orig /etc/grub.d/10_linux *** /tmp/10_linux.orig 2017-02-02 08:02:36.926542777 -0700 --- /etc/grub.d/10_linux 2017-02-02 15:26:02.138285042 -0700
* 77,85 **
GRUB_CMDLINE_LINUX="rootflags=subvol=${rootsubvol} ${GRUB_CMDLINE_LINUX}"
fi;;
xzfs)
! rpool=${grub_probe} --device ${GRUB_DEVICE} --target=fs_label 2>/dev/null || true
! bootfs="make_system_path_relative_to_its_root / | sed -e "s,@$,,"
"
! LINUX_ROOT_DEVICE="ZFS=${rpool}${bootfs}"
;;
esac
--- 77,96 ----
GRUB_CMDLINE_LINUX="rootflags=subvol=${rootsubvol} ${GRUB_CMDLINE_LINUX}"
fi;;
xzfs)
! zfsbootfs=zpool get bootfs | sed '2,100!d' | grep -v "-" | awk {'print $3'} 2>/dev/null || true
! if [ "${zfsbootfs}" = "" ]
! then
! echo "zpool bootfs parameter not set. System will fail to boot!" 1>&2
! elif [ "$(echo ${zfsbootfs} | wc -w)" -gt "1" ]
! then
! echo "Multiple zpool bootfs parameters detected:" 1>&2
! echo ${zfsbootfs} 1>&2
! echo 1>&2
! zfsbootfs=echo ${zfsbootfs} | awk {'print $1'}
! echo "Using the first bootfs listed: ${zfsbootfs}" 1>&2
! echo "Final LINUX_ROOT_DEVICE=ZFS=${zfsbootfs}" 1>&2
! fi
! LINUX_ROOT_DEVICE="ZFS=${zfsbootfs}"
;;
esac
Why is the default behaviour to ignore the bootfs setting? I was experimenting with snapshots and clones (debian zfs initramfs support is nice!), then found that changing bootfs did nothing and that grub scripts always generate the same bootfs= string. Debian 9.4 (root on zfs) with zfs-dkms and zfs-initramfs from backports (version 0.7.6-1~bpo9+1)
Same issue on Debian Bullseye. In my case, I overrode root=ZFS
in /etc/default/grub
, but it would be great if it could respect bootfs
property.
Running Debian Jessie. GRUB 2.02-beta2.9-ZOL11-7 is installed.
GRUB is creating the root device incorrectly. Running update grub produces the root device as "root=ZFS=/ROOT" instead of the proper root device of "root=ZFS=rpool/ROOT"