Grub history menu for zsys state entries will not be created if installer chooses efi partition located on another drive

Update: I have history menu in grub now. The Ubuntu installer created an efi partition on the same drive as rpool, but then proceeded to install grub on another drive with an existing efi partition. E.g.:

/dev/nvme0n1p1 # <--- Ubuntu installed grub here: existing efi from another distro
/dev/nvme0n1p2 # lv from another distro

/dev/nvme2n1p1 # <--- Ubuntu created this partition, but then didn't use it
/dev/nvme2n1p2 # bpool
/dev/nvme2n1p3 # rpool, etc.

Further illustrated by entry auto-generated by the installer in /etc/fstab:

# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# Note (added by installer): /boot/efi was on /dev/nvme0n1p1 during installation
# Note (added by user): should have been on /dev/nvme2n1p1 - moved 2-21-2022
UUID=<orig incorrect>  /boot/efi       vfat    umask=0022,fmask=0022,dmask=0022      0       1

Note: I thought I'd leave this as a history trail, in case it can help you guys. I'm going to go track down the installer maintainers since I think that's where this issue probably belongs.

Issue thread:

Installer created efi partition on same nvme as rpool, but then used another efi partition on another drive
grub-mkconfig would not use anything other than default grub config on efi if located on another drive (is this normal behavior?)
grub history menu entry not displayed, leading to the trail of exploration I engaged in below:

Deprecated but left in case useful - I have judged there is no problem with zsys, and rather was user + installer error.

Describe the bug Grub history menu entry not present on freshly installed Ubuntu 22.04 dev snapshot

To Reproduce Steps to reproduce the behavior: Turn on computer Examine grub menu notice History entry not available / shown

Expected behavior Grub History menu to be present during boot Grub history entries to be created by zsys state updates, which are being generated after every apt install / upgrade command

For ubuntu users, please run and copy the following:

ubuntu-bug zsys --save=/tmp/report
Copy paste below /tmp/report content:

zsys-no-grub-history-apport-report-2-21-2022.txt

Screenshots Not applicable

Installed versions:

OS:

PRETTY_NAME="Ubuntu Jammy Jellyfish (development branch)"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04 (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Zsysd running version:
```
zsysctl 0.5.8build1
zsysd   0.5.8build1
```

Additional context I am going through a read-only copy of /etc/grub.d/10_linux_zfs and trying out some of the commands with which grub-mkconfig is directed to search for zfs snapshots, but the syntax is not acquiring any snapshots. E.g. line 107 10_linux_zfs:

zfs list -H -o name,canmount,mountpoint -t filesystem | grep -E '^'"${p}"'(\s|/[[:print:]]*\s)(on|noauto)\s'"${rel_pool_root}"'$' | awk '{print $1}'

Produces no output. However, zfs list -t filesystem canmount=on shows there are plenty of mountable filesystems (sorry, could not grep for canmount=noauto):

zfs list -H -o name,canmount,mountpoint -t filesystem | grep -v off
bpool/BOOT/ubuntu_pd3ehl        on      /boot
rpool/ROOT/ubuntu_pd3ehl        on      /
rpool/ROOT/ubuntu_pd3ehl/srv    on      /srv
rpool/ROOT/ubuntu_pd3ehl/usr/local      on      /usr/local
rpool/ROOT/ubuntu_pd3ehl/var/games      on      /var/games
rpool/ROOT/ubuntu_pd3ehl/var/lib        on      /var/lib
rpool/ROOT/ubuntu_pd3ehl/var/lib/AccountsService        on      /var/lib/AccountsService
rpool/ROOT/ubuntu_pd3ehl/var/lib/NetworkManager on      /var/lib/NetworkManager
rpool/ROOT/ubuntu_pd3ehl/var/lib/apt    on      /var/lib/apt
rpool/ROOT/ubuntu_pd3ehl/var/lib/dpkg   on      /var/lib/dpkg
rpool/ROOT/ubuntu_pd3ehl/var/log        on      /var/log
rpool/ROOT/ubuntu_pd3ehl/var/mail       on      /var/mail
rpool/ROOT/ubuntu_pd3ehl/var/snap       on      /var/snap
rpool/ROOT/ubuntu_pd3ehl/var/spool      on      /var/spool
rpool/ROOT/ubuntu_pd3ehl/var/www        on      /var/www
rpool/USERDATA/avery_wjn2jt     on      /home/avery
rpool/USERDATA/avery_wjn2jt/containers  on      /home/avery/containers
rpool/USERDATA/root_wjn2jt      on      /root

I don't know if this is because of how 10_linux_zfs deconstructs zfs list output and puts it back together based on certain criteria, but if I had to guess I would think something regarding detecting available snapshots not happening in /etc/grub.d/10_linux_zfs.

Does this appear to be more of a grub 10_linux_zfs bug rather than zsys, and should I instead take my concerns there?

Also, I'm noticing zsysctl boot command removed entirely, does that mean zsys is no longer responsible for creating entries for boot history? If so, what is an alternative?

Thanks

Update:

I enabled the jammy-proposed repo, hoping there might be a fix. Sure enough, there was a an update of grub packages, which I installed.

I haven't rebooted yet to see if the History entry is present in the grub menu yet, but I did have the presence of mind to look at /boot/grub/grub.cfg, and it looks promising:

egrep -i history /boot/grub/grub.cfg 
function zsyshistorymenu {
submenu 'History for Ubuntu Jammy Jellyfish (development branch)' ${menuentry_id_option} 'gnulinux-history-rpool/ROOT/ubuntu_pd3ehl' {
        submenu 'Revert to 02/21/2022 @ 13:32' ${menuentry_id_option} 'gnulinux-history-rpool/ROOT/ubuntu_pd3ehl@autozsys_flyrl2' {
                zsyshistorymenu "rpool/ROOT/ubuntu_pd3ehl@autozsys_flyrl2" "/dev/nvme2n1p3" "/BOOT/ubuntu_pd3ehl@autozsys_flyrl2/initrd.img-5.15.0-18-generic" "/BOOT/ubuntu_pd3ehl@autozsys_flyrl2/vmlinuz-5.15.0-18-generic" "5.15.0-18-generic"
        submenu 'Revert to 02/21/2022 @ 13:29' ${menuentry_id_option} 'gnulinux-history-rpool/ROOT/ubuntu_pd3ehl@autozsys_rznzih' {
                zsyshistorymenu "rpool/ROOT/ubuntu_pd3ehl@autozsys_rznzih" "/dev/nvme2n1p3" "/BOOT/ubuntu_pd3ehl@autozsys_rznzih/initrd.img-5.15.0-18-generic" "/BOOT/ubuntu_pd3ehl@autozsys_rznzih/vmlinuz-5.15.0-18-generic" "5.15.0-18-generic"
        submenu 'Revert to 02/21/2022 @ 10:34' ${menuentry_id_option} 'gnulinux-history-rpool/ROOT/ubuntu_pd3ehl@autozsys_jc4ivf' {
                zsyshistorymenu "rpool/ROOT/ubuntu_pd3ehl@autozsys_jc4ivf" "/dev/nvme2n1p3" "/BOOT/ubuntu_pd3ehl@autozsys_jc4ivf/initrd.img-5.15.0-18-generic" "/BOOT/ubuntu_pd3ehl@autozsys_jc4ivf/vmlinuz-5.15.0-18-generic" "5.15.0-18-generic"
        submenu 'Revert to 02/21/2022 @ 01:33' ${menuentry_id_option} 'gnulinux-history-rpool/ROOT/ubuntu_pd3ehl@autozsys_ttxtsj' {
                zsyshistorymenu "rpool/ROOT/ubuntu_pd3ehl@autozsys_ttxtsj" "/dev/nvme2n1p3" "/BOOT/ubuntu_pd3ehl@autozsys_ttxtsj/initrd.img-5.15.0-18-generic" "/BOOT/ubuntu_pd3ehl@autozsys_ttxtsj/vmlinuz-5.15.0-18-generic" "5.15.0-18-generic"
        submenu 'Revert to 02/21/2022 @ 01:33' ${menuentry_id_option} 'gnulinux-history-rpool/ROOT/ubuntu_pd3ehl@autozsys_rszxx0' {
                zsyshistorymenu "rpool/ROOT/ubuntu_pd3ehl@autozsys_rszxx0" "/dev/nvme2n1p3" "/BOOT/ubuntu_pd3ehl@autozsys_rszxx0/initrd.img-5.15.0-18-generic" "/BOOT/ubuntu_pd3ehl@autozsys_rszxx0/vmlinuz-5.15.0-18-generic" "5.15.0-18-generic"
        submenu 'Revert to 02/21/2022 @ 00:55' ${menuentry_id_option} 'gnulinux-history-rpool/ROOT/ubuntu_pd3ehl@autozsys_ngspi6' {
                zsyshistorymenu "rpool/ROOT/ubuntu_pd3ehl@autozsys_ngspi6" "/dev/nvme2n1p3" "/BOOT/ubuntu_pd3ehl@autozsys_ngspi6/initrd.img-5.15.0-18-generic" "/BOOT/ubuntu_pd3ehl@autozsys_ngspi6/vmlinuz-5.15.0-18-generic" "5.15.0-18-generic"
        submenu 'Revert to 02/21/2022 @ 00:51' ${menuentry_id_option} 'gnulinux-history-rpool/ROOT/ubuntu_pd3ehl@autozsys_56uuqi' {
                zsyshistorymenu "rpool/ROOT/ubuntu_pd3ehl@autozsys_56uuqi" "/dev/nvme2n1p3" "/BOOT/ubuntu_pd3ehl@autozsys_56uuqi/initrd.img-5.15.0-18-generic" "/BOOT/ubuntu_pd3ehl@autozsys_56uuqi/vmlinuz-5.15.0-18-generic" "5.15.0-18-generic"
        submenu 'Revert to 02/21/2022 @ 00:47' ${menuentry_id_option} 'gnulinux-history-rpool/ROOT/ubuntu_pd3ehl@autozsys_xdm1tn' {
                zsyshistorymenu "rpool/ROOT/ubuntu_pd3ehl@autozsys_xdm1tn" "/dev/nvme2n1p3" "/BOOT/ubuntu_pd3ehl@autozsys_xdm1tn/initrd.img-5.15.0-18-generic" "/BOOT/ubuntu_pd3ehl@autozsys_xdm1tn/vmlinuz-5.15.0-18-generic" "5.15.0-18-generic"
        submenu 'Revert to 02/21/2022 @ 00:45' ${menuentry_id_option} 'gnulinux-history-rpool/ROOT/ubuntu_pd3ehl@autozsys_5i19hf' {
                zsyshistorymenu "rpool/ROOT/ubuntu_pd3ehl@autozsys_5i19hf" "/dev/nvme2n1p3" "/BOOT/ubuntu_pd3ehl@autozsys_5i19hf/initrd.img-5.15.0-18-generic" "/BOOT/ubuntu_pd3ehl@autozsys_5i19hf/vmlinuz-5.15.0-18-generic" "5.15.0-18-generic"
        submenu 'Revert to 02/21/2022 @ 00:39' ${menuentry_id_option} 'gnulinux-history-rpool/ROOT/ubuntu_pd3ehl@autozsys_2xfo2r' {
                zsyshistorymenu "rpool/ROOT/ubuntu_pd3ehl@autozsys_2xfo2r" "/dev/nvme2n1p3" "/BOOT/ubuntu_pd3ehl@autozsys_2xfo2r/initrd.img-5.15.0-18-generic" "/BOOT/ubuntu_pd3ehl@autozsys_2xfo2r/vmlinuz-5.15.0-18-generic" "5.15.0-18-generic"
        submenu 'Revert to 02/19/2022 @ 20:52' ${menuentry_id_option} 'gnulinux-history-rpool/ROOT/ubuntu_pd3ehl@autozsys_60vc9s' {
                zsyshistorymenu "rpool/ROOT/ubuntu_pd3ehl@autozsys_60vc9s" "/dev/nvme2n1p3" "/BOOT/ubuntu_pd3ehl@autozsys_60vc9s/initrd.img-5.15.0-18-generic" "/BOOT/ubuntu_pd3ehl@autozsys_60vc9s/vmlinuz-5.15.0-18-generic" "5.15.0-18-generic"
        submenu 'Revert to 02/19/2022 @ 20:51' ${menuentry_id_option} 'gnulinux-history-rpool/ROOT/ubuntu_pd3ehl@autozsys_t7y21n' {
                zsyshistorymenu "rpool/ROOT/ubuntu_pd3ehl@autozsys_t7y21n" "/dev/nvme2n1p3" "/BOOT/ubuntu_pd3ehl@autozsys_t7y21n/initrd.img-5.15.0-18-generic" "/BOOT/ubuntu_pd3ehl@autozsys_t7y21n/vmlinuz-5.15.0-18-generic" "5.15.0-18-generic"
        submenu 'Revert to 02/19/2022 @ 20:31' ${menuentry_id_option} 'gnulinux-history-rpool/ROOT/ubuntu_pd3ehl@autozsys_a4ni54' {
                zsyshistorymenu "rpool/ROOT/ubuntu_pd3ehl@autozsys_a4ni54" "/dev/nvme2n1p3" "/BOOT/ubuntu_pd3ehl@autozsys_a4ni54/initrd.img-5.15.0-18-generic" "/BOOT/ubuntu_pd3ehl@autozsys_a4ni54/vmlinuz-5.15.0-18-generic" "5.15.0-18-generic"
        submenu 'Revert to 02/19/2022 @ 20:31' ${menuentry_id_option} 'gnulinux-history-rpool/ROOT/ubuntu_pd3ehl@autozsys_4yc3yk' { . . .

grub and zfs related packages I just upgraded:

$ apt-history install | egrep -i 'grub|zfs'
2022-02-02 07:44:05 install grub-common:amd64 <none> 2.06-2ubuntu4
2022-02-02 07:44:05 install grub2-common:amd64 <none> 2.06-2ubuntu4
2022-02-02 07:44:05 install grub-pc-bin:amd64 <none> 2.06-2ubuntu4
2022-02-02 07:44:05 install grub-pc:amd64 <none> 2.06-2ubuntu4
2022-02-02 07:44:05 install grub-gfxpayload-lists:amd64 <none> 0.7
2022-02-02 07:46:17 install libzfs4linux:amd64 <none> 2.1.2-1ubuntu1
2022-02-02 07:46:17 install zfsutils-linux:amd64 <none> 2.1.2-1ubuntu1
2022-02-02 07:46:17 install zfs-zed:amd64 <none> 2.1.2-1ubuntu1
2022-02-02 07:46:19 install grub-efi-amd64-bin:amd64 <none> 2.06-2ubuntu4
2022-02-02 07:46:19 install grub-efi-amd64-signed:amd64 <none> 1.177+2.06-2ubuntu4
2022-02-02 07:46:31 install zfs-initramfs:amd64 <none> 2.1.2-1ubuntu1
2022-02-19 04:24:20 install grub-efi-amd64:amd64 <none> 2.06-2ubuntu4
2022-02-19 04:28:31 install grub-efi:amd64 <none> 2.06-2ubuntu4
2022-02-19 04:28:56 install grub-efi-amd64-signed-template:amd64 <none> 2.06-2ubuntu4
2022-02-19 15:47:08 install grubzfs-testsuite:amd64 <none> 0.4.15build1
2022-02-19 20:23:22 install grub-pc-bin:amd64 <none> 2.06-2ubuntu4
2022-02-19 20:23:22 install grub-gfxpayload-lists:amd64 <none> 0.7
2022-02-19 20:23:23 install grub-pc:amd64 2.06-2ubuntu4 2.06-2ubuntu4
2022-02-19 20:23:23 install grub2:amd64 <none> 2.06-2ubuntu4

current versions:

grub-common/jammy-proposed,now 2.06-2ubuntu5 amd64 [installed]
grub-efi-amd64-bin/jammy,now 2.06-2ubuntu5 amd64 [installed]
grub-efi-amd64-signed-template/jammy-proposed,now 2.06-2ubuntu5 amd64 [installed]
grub-efi-amd64-signed/jammy,now 1.178+2.06-2ubuntu5 amd64 [installed]
grub-gfxpayload-lists/jammy,now 0.7 amd64 [installed,automatic]
grub-pc-bin/jammy-proposed,now 2.06-2ubuntu5 amd64 [installed,automatic]
grub-pc/jammy-proposed,now 2.06-2ubuntu5 amd64 [installed,automatic]
grub2-common/jammy-proposed,now 2.06-2ubuntu5 amd64 [installed]
grub2/jammy-proposed,now 2.06-2ubuntu5 amd64 [installed]
grubzfs-testsuite/jammy,now 0.4.15build1 amd64 [installed]
libzfs4linux/jammy,now 2.1.2-1ubuntu3 amd64 [installed]
zfs-initramfs/jammy,now 2.1.2-1ubuntu3 amd64 [installed]
zfs-zed/jammy,now 2.1.2-1ubuntu3 amd64 [installed]
zfsutils-linux/jammy,now 2.1.2-1ubuntu3 amd64 [installed]

I will reboot and report back whether or not grub history menu indeed present as expected now.

ubuntu / zsys

Grub history menu for zsys state entries will not be created if installer chooses efi partition located on another drive #223