meilon commented 1 year ago

@ne9z, I have the following issue with the Arch Linux ZFS HOWTO:

I updated my install last night to the current kernel and zfs combo and now my system won't boot anymore. Grub complains that the compression algorithm inherit not supported. The pools mount fine in arch live or alpine live. I guess I must've accidentally enabled a feature or something and GRUB won't work anymore.

From what I read ZFS and GRUB where never that good together, so what do you think about a switch to ZFSBootMenu?

ghost commented 1 year ago

Christian Arnold @.***> writes:

I updated my install last night

I would like to know what did your "update" procedure entail. Did you use "zpool upgrade"?

If you did not do anything besides updating the packages, and Grub nevertheless fails to read the pool, then we've got a serious bug to fix.

Grub complains that the compression algorithm inherit not supported. The pools mount fine in arch live or alpine live. I guess I must've accidentally enabled a feature or something and GRUB won't work anymore.

Well, there is no "accident"; if the computer does not boot after I run "zpool destroy bpool", I can not say there is a fault with the guide.

One thing I will admit it that, the guide should use pool feature flags. It is currently not used due to historical reasons. I think this will prevent issues like yours from happening.

From what I read ZFS and GRUB where never that good together,

Please substantiate your claim with evidence.

so what do you think about a switch to ZFSBootMenu?

You are free to switch to ZFSBootMenu for your computers.

For me, it is not nearly as universal or battle-tested as GRUB, and addes unnecessary complication to the guide.

GRUB is actively developed and its security is constantly scrutinized by companies around the world. Support for TPM2 is being developed, for example.

[1] https://lists.gnu.org/archive/html/grub-devel/2023-10/threads.html

ghost commented 1 year ago

Christian Arnold @.***> writes:

From what I read ZFS and GRUB where never that good together

Please substantiate your claim with evidence.

I will correct myself here: in your case, the computer did not boot after an update, and that is indeed bad.

But the problem is not with GRUB: GRUB only supports a subset of pool features, which are clearly specified in the GRUB source code. If boot pool sticks to that set of features, everything would work correctly.

This limitation is seen as a disadvantage for many: but not me. I consider this limitation as a feature, because it guarantees compatibility between any version of GRUB and any boot pool.

meilon commented 1 year ago

I just updated linux, zfs and zfs-utils and did a reboot as always. Worked with 6.5.7 (and a lot of previous kernel releases), not with 6.5.8. Didn't do a zpool upgrade.

I didn't do anything weird between those two boots (6.5.7 update was the last one), especially doing anything zfs related.

Here's a paste of grub-probe: https://pastebin.com/dJWrj482

And yeah, I meant the supported features. Like booting snapshots for instance.

ghost commented 1 year ago

@gmelikov ZFS package update without zpool upgrade breaks GRUB. What's your take on this?

gmelikov commented 1 year ago

Hm, there are some history about external problems https://forum.proxmox.com/threads/grub-error-compression-algorithm-inherit-not-supported-on-reboot.38629/ https://github.com/pbatard/efifs/issues/27

We need a reproducer, @ne9z maybe you have already tried? And it's just to try to use Arch instruction? I didn't try it yet. We didn't have large changes on compression side in 2.2.

ghost commented 1 year ago

George Melikov @.***> writes:

Hm, there are some history about external problems https://forum.proxmox.com/threads/grub-error-compression-algorithm-inherit-not-supported-on-reboot.38629/ pbatard/efifs#27

Thanks for the links! The first link was due to a hardware problem;

The second link used a EFI driver that is known to be broken -- I tried that driver back in 2020, didn't work back then, no surprise it is still broken today.

We need a reproducer, @ne9z maybe you have already tried?

I tried and could not reproduce the problem.

We didn't have large changes on compression side in 2.2.

Thanks for the info.

To OP: can you try to boot from the same disk on a different computer?

gmelikov commented 1 year ago

@meilon @ne9z I've found issue about that https://github.com/openzfs/zfs/issues/15261

EDIT: oops, @meilon you was faster :)

ghost commented 1 year ago

George Melikov @.***> writes:

@meilon @ne9z I've found issue about that openzfs/zfs#15261

Thanks. I will update the documentation to warn about top-level snapshots. Should we do this also for Debian/Ubuntu?

To OP: you can switch to ZBM easily by:

Boot from live CD
Destroy bpool
Import the rpool, mount at /mnt
Reinstall kernel, at this point /boot should be populated
Install ZBM

meilon commented 1 year ago

Thank you, I don't know why I clung to keeping the bpool. Without it it works great!

ghost commented 1 year ago

Christian Arnold @.***> writes:

Thank you, I don't know why I clung to keeping the bpool. Without it it works great!

There are two options: either you destroy and rebuild bpool by following steps in the guide, after which GRUB will work again; or destroy bpool then use ZBM, which sees bpool as an impediment.

ghost commented 1 year ago

Now I can reproduce this issue:

dd if=/dev/zero of=test.img bs=1M count=1024

zpool create tpool -O mountpoint=legacy -o compatibility=grub2 /root/test.img

mount -t zfs tpool /mnt

grub-probe /mnt

zfs

zfs snap @.***

grub-probe /mnt

grub-probe: error: compression algorithm inherit not supported .

But, using compatibility=legacy seems to solve this issue:

zpool destroy tpool

zpool create tpool -O mountpoint=legacy -o compatibility=legacy /root/test.img

mount -t zfs tpool /mnt

grub-probe /mnt

zfs

zfs snap @.***

grub-probe /mnt

zfs #

I'm now more inclined to just setting compatibility=legacy on bpool instead of adding warning messages.

ahesford commented 1 year ago

You are free to switch to ZFSBootMenu for your computers. For me, it is not nearly as universal or battle-tested as GRUB, and addes unnecessary complication to the guide. GRUB is actively developed and its security is constantly scrutinized by companies around the world.

This is the funniest thing I've read all week. Thanks for that.

ghost commented 1 year ago

Thanks for your long-time advocacy of ZBM and total and complete denigration of GRUB, the Grand Unified Bootloader! Really appreciate that. We all know GRUB is a totally worthless piece of garbage that should be stamped on, burned at the stake and forgotten by history.

ZBM totally does not have to deal with feature-compatibility regardless of ZFS version, no sir! No manual intervention ever needed.

If you are so keen, I invite you to maintain the guides here for greater public good, with ZBM if that's your choice. I will quietly leave the scene because I have better things to do.

"Andrew J. Hesford" @.***> writes:

You are free to switch to ZFSBootMenu for your computers. For me, it is not nearly as universal or battle-tested as GRUB, and addes unnecessary complication to the guide. GRUB is actively developed and its security is constantly scrutinized by companies around the world.

This is the funniest thing I've read all week. Thanks for that.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

ahesford commented 1 year ago

Note that, if you are switching to ZFSBootMenu, you shouldn't destroy your boot pool or reinstall your kernel packages. Just copy the contents of /boot to /boot.new, unset the mountpoint on your existing /boot and confirm that it's unmounted, rmdir /boot && mv /boot.new /boot.

Needless destruction before trying something new is a terrible practice. If you migrate to ZFSBootMenu and like it, you can then decide what you want to do with the vestigial boot pool.

ghost commented 1 year ago

"Andrew J. Hesford" @.***> writes:

Needless destruction before trying something new is a terrible practice. If you migrate to ZFSBootMenu and like it, you can then decide what you want to do with the vestigial boot pool.

I thought separate boot pool would interfere with ZBM's detection mechanism. Good to hear it doesn't. Great job, well done.

rlaager commented 1 year ago

Is this an actual problem or just a spurious message?

ghost commented 1 year ago

Richard Laager @.***> writes:

Is this an actual problem or just a spurious message?

Most likely a spurious message. Because I also get this message about compression algorithm "inherit" not supported when GRUB successfully boots.

To get to the actual cause of why a root dataset snapshot renders the bpool permanently GRUB-unreadable needs some serious GRUB-fu to get the debug details, it seems.

colmbuckley commented 1 year ago

Note: your reproduction above is not sound - you are creating the bpool pool on a zeroed-out file inside a (presumably not grub-compatible) dataset, then snapshotting that dataset. GRUB is trying to map from this back into raw disk offsets, which I expect would be highly unreliable. I think that the grub-probe errors here are red herrings (I'm frankly surprised it works even with compatibility=legacy).

Has anyone been able to reproduce this problem when creating bpool on a separate disk partition as the guide suggests?

colmbuckley commented 1 year ago

Ah, I see that the original reporter did. Hmm. @meilon do you still have access to the bpool you originally had trouble with? Could you show the output of zpool get all bpool if so?

meilon commented 1 year ago

I destroyed the pool, but didn't delete the partitions. I re-imported it just now, here's the requested output:

$ zpool get all bpool
NAME   PROPERTY                       VALUE                          SOURCE
bpool  size                           3.75G                          -
bpool  capacity                       0%                             -
bpool  altroot                        -                              default
bpool  health                         ONLINE                         -
bpool  guid                           7872762533682099877            -
bpool  version                        -                              default
bpool  bootfs                         -                              default
bpool  delegation                     on                             default
bpool  autoreplace                    off                            default
bpool  cachefile                      -                              default
bpool  failmode                       wait                           default
bpool  listsnapshots                  off                            default
bpool  autoexpand                     off                            default
bpool  dedupratio                     1.00x                          -
bpool  free                           3.75G                          -
bpool  allocated                      988K                           -
bpool  readonly                       off                            -
bpool  ashift                         12                             local
bpool  comment                        -                              default
bpool  expandsize                     -                              -
bpool  freeing                        0                              -
bpool  fragmentation                  0%                             -
bpool  leaked                         0                              -
bpool  multihost                      off                            default
bpool  checkpoint                     -                              -
bpool  load_guid                      14349335285769238702           -
bpool  autotrim                       on                             local
bpool  compatibility                  off                            default
bpool  bcloneused                     0                              -
bpool  bclonesaved                    0                              -
bpool  bcloneratio                    1.00x                          -
bpool  feature@async_destroy          enabled                        local
bpool  feature@empty_bpobj            active                         local
bpool  feature@lz4_compress           active                         local
bpool  feature@multi_vdev_crash_dump  disabled                       local
bpool  feature@spacemap_histogram     active                         local
bpool  feature@enabled_txg            active                         local
bpool  feature@hole_birth             active                         local
bpool  feature@extensible_dataset     enabled                        local
bpool  feature@embedded_data          active                         local
bpool  feature@bookmarks              enabled                        local
bpool  feature@filesystem_limits      enabled                        local
bpool  feature@large_blocks           enabled                        local
bpool  feature@large_dnode            disabled                       local
bpool  feature@sha512                 disabled                       local
bpool  feature@skein                  disabled                       local
bpool  feature@edonr                  disabled                       local
bpool  feature@userobj_accounting     disabled                       local
bpool  feature@encryption             disabled                       local
bpool  feature@project_quota          disabled                       local
bpool  feature@device_removal         disabled                       local
bpool  feature@obsolete_counts        disabled                       local
bpool  feature@zpool_checkpoint       disabled                       local
bpool  feature@spacemap_v2            disabled                       local
bpool  feature@allocation_classes     disabled                       local
bpool  feature@resilver_defer         disabled                       local
bpool  feature@bookmark_v2            disabled                       local
bpool  feature@redaction_bookmarks    disabled                       local
bpool  feature@redacted_datasets      disabled                       local
bpool  feature@bookmark_written       disabled                       local
bpool  feature@log_spacemap           disabled                       local
bpool  feature@livelist               disabled                       local
bpool  feature@device_rebuild         disabled                       local
bpool  feature@zstd_compress          disabled                       local
bpool  feature@draid                  disabled                       local
bpool  feature@zilsaxattr             disabled                       local
bpool  feature@head_errlog            disabled                       local
bpool  feature@blake3                 disabled                       local
bpool  feature@block_cloning          disabled                       local
bpool  feature@vdev_zaps_v2           disabled                       local

ghost commented 1 year ago

Colm @.***> writes:

Note: your reproduction above is not sound - you are creating the bpool pool on a zeroed-out file inside a (presumably not grub-compatible) dataset, then snapshotting that dataset. GRUB is trying to map from this back into raw disk offsets, which I expect would be highly unreliable. I think that the grub-probe errors here are red herrings (I'm frankly surprised it works even with compatibility=legacy).

You might be right about the part of testing a file based pool. But now I actually use my own guides on my physical laptop, I can conclusively demonstrate that compat=legacy on a baremental machine fixes the "bpool snapshot breaks GRUB compatibility" issue:

@.***:~]# zpool status bpool pool: bpool state: ONLINE config:

    NAME                                                    STATE     READ WRITE CKSUM
    bpool                                                   ONLINE       0     0     0
      nvme-SAMSUNG_MZVLV256HCHP-000H1_S2CSNA0J547878-part2  ONLINE       0     0     0

errors: No known data errors

@.:~]# NAME PROPERTY VALUE bpool size 3.75G bpool capacity 3% bpool altroot - bpool health ONLINE bpool guid 17697956629229637003 bpool version - bpool bootfs - bpool delegation on bpool autoreplace off bpool cachefile - bpool failmode wait bpool listsnapshots off bpool autoexpand off bpool dedupratio 1.00x bpool free 3.64G bpool allocated 117M bpool readonly off bpool ashift 12 bpool comment - bpool expandsize - bpool freeing 0 bpool fragmentation - bpool leaked 0 bpool multihost off bpool checkpoint - bpool load_guid 1900602576278683098 bpool autotrim on bpool compatibility legacy bpool bcloneused 0 bpool bclonesaved 0 bpool bcloneratio 1.00x bpool @._destroy bpool @._bpobj bpool @._compress bpool @._v bpool @._histogram bpool @._txg bpool @._birth bpool @._dataset bpool @._data bpool @. bpool @._limits bpool @._blocks bpool @._dnode bpool @. bpool @. bpool @. bpool @._accounting bpool @. bpool @._quota bpool @._removal bpool @._counts bpool @._checkpoint bpool @._v2 bpool @._classes bpool @._defer bpool @._v2 bpool @._bookmarks bpool @._datasets bpool @._written bpool @._spacemap bpool @. bpool @._rebuild bpool @._compress bpool @. bpool @. bpool @._errlog bpool @. bpool @._cloning bpool @._zaps_v2 zpool get all bpool SOURCE - - default - - default default default default default default default default - - - - local default - - - - default - - local local - - - disabled local disabled local disabled local dev_crash_dump disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local disabled local

@.:~]# zfs snap @.

@.:~]# zfs list -t snapshot bpool NAME USED AVAIL REFER MOUNTPOINT @. 0B - 176K -

@.***:~]# grub-probe -vvv /boot grub-probe: info: cannot open /boot/grub/device.map': No such file or directory. grub-probe: info: /dev/nvme0n1p2 is not present. grub-probe: info: Looking for /dev/nvme0n1p2. grub-probe: info: /dev/nvme0n1 is a parent of /dev/nvme0n1p2. grub-probe: info: /dev/nvme0n1p2 starts from 2097152. grub-probe: info: opening the device hostdisk//dev/nvme0n1. grub-core/kern/disk.c:196:disk: Openinghostdisk//dev/nvme0n1'... grub-probe: info: drive = 0. grub-probe: info: the size of hostdisk//dev/nvme0n1 is 500118192. grub-core/kern/disk.c:196:disk: Opening hostdisk//dev/nvme0n1'... grub-probe: info: drive = 0. grub-probe: info: the size of hostdisk//dev/nvme0n1 is 500118192. grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:379:hostdisk: opening the device/dev/nvme0n1' in open_device() grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1' grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1' grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1' grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1' grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-core/partmap/gpt.c:93:gpt: Read a valid GPT header grub-core/partmap/gpt.c:115:gpt: GPT entry 0: start=4096, length=2093056 grub-core/partmap/gpt.c:115:gpt: GPT entry 1: start=2097152, length=8388608 grub-core/partmap/gpt.c:115:gpt: GPT entry 2: start=10485760, length=479146672 grub-core/partmap/gpt.c:115:gpt: GPT entry 3: start=489632432, length=8388608 grub-core/partmap/gpt.c:115:gpt: GPT entry 4: start=2048, length=2048 grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/gpt.c:93:gpt: Read a valid GPT header grub-core/partmap/gpt.c:115:gpt: GPT entry 0: start=4096, length=2093056 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:379:hostdisk: opening the device /dev/nvme0n1p1' in open_device() grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1p1' grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1p1' grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/dfly.c:69:partition: bad magic (found 0x41615252; wanted 0xc4464c59) grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/apple.c:120:partition: bad magic (found 0xeb58; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 1: start=2097152, length=8388608 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:379:hostdisk: opening the device/dev/nvme0n1p2' in open_device() grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1p2' grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1p2' grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x0; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x0; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 2: start=10485760, length=479146672 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:379:hostdisk: opening the device /dev/nvme0n1p3' in open_device() grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1p3' grub-core/osdep/hostdisk.c:379:hostdisk: opening the device /dev/nvme0n1' in open_device() grub-core/osdep/hostdisk.c:379:hostdisk: opening the device/dev/nvme0n1p3' in open_device() grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1p3' grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x0; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x4c55; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 3: start=489632432, length=8388608 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:379:hostdisk: opening the device/dev/nvme0n1p4' in open_device() grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1p4' grub-core/osdep/hostdisk.c:379:hostdisk: opening the device/dev/nvme0n1' in open_device() grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:379:hostdisk: opening the device /dev/nvme0n1p4' in open_device() grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x1c5a8c1a; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x8bac; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 4: start=2048, length=2048 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:379:hostdisk: opening the device/dev/nvme0n1p5' in open_device() grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1p5' grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1p5' grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x821cea; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x5256; wanted 0x4552) grub-core/partmap/dfly.c:69:partition: bad magic (found 0x20494645; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0xeb63; wanted 0x4552) grub-core/kern/disk.c:299:disk: Closing hostdisk//dev/nvme0n1'. grub-core/partmap/gpt.c:93:gpt: Read a valid GPT header grub-core/partmap/gpt.c:115:gpt: GPT entry 0: start=4096, length=2093056 grub-probe: info: Partition 0 starts from 4096. grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/dfly.c:69:partition: bad magic (found 0x41615252; wanted 0xc4464c59) grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/apple.c:120:partition: bad magic (found 0xeb58; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 1: start=2097152, length=8388608 grub-probe: info: Partition 1 starts from 2097152. grub-core/kern/disk.c:299:disk: Closinghostdisk//dev/nvme0n1'. grub-probe: info: /dev/nvme0n1p2 is present. grub-probe: info: Looking for /dev/nvme0n1p2. grub-probe: info: /dev/nvme0n1 is a parent of /dev/nvme0n1p2. grub-probe: info: /dev/nvme0n1p2 starts from 2097152. grub-probe: info: opening the device hostdisk//dev/nvme0n1. grub-core/kern/disk.c:196:disk: Opening hostdisk//dev/nvme0n1'... grub-probe: info: drive = 0. grub-probe: info: the size of hostdisk//dev/nvme0n1 is 500118192. grub-core/kern/disk.c:196:disk: Openinghostdisk//dev/nvme0n1'... grub-probe: info: drive = 0. grub-probe: info: the size of hostdisk//dev/nvme0n1 is 500118192. grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:379:hostdisk: opening the device /dev/nvme0n1' in open_device() grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1' grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-core/partmap/gpt.c:93:gpt: Read a valid GPT header grub-core/partmap/gpt.c:115:gpt: GPT entry 0: start=4096, length=2093056 grub-core/partmap/gpt.c:115:gpt: GPT entry 1: start=2097152, length=8388608 grub-core/partmap/gpt.c:115:gpt: GPT entry 2: start=10485760, length=479146672 grub-core/partmap/gpt.c:115:gpt: GPT entry 3: start=489632432, length=8388608 grub-core/partmap/gpt.c:115:gpt: GPT entry 4: start=2048, length=2048 grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/gpt.c:93:gpt: Read a valid GPT header grub-core/partmap/gpt.c:115:gpt: GPT entry 0: start=4096, length=2093056 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/dfly.c:69:partition: bad magic (found 0x41615252; wanted 0xc4464c59) grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/apple.c:120:partition: bad magic (found 0xeb58; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 1: start=2097152, length=8388608 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x0; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x0; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 2: start=10485760, length=479146672 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x0; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x4c55; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 3: start=489632432, length=8388608 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x1c5a8c1a; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x8bac; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 4: start=2048, length=2048 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x821cea; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x5256; wanted 0x4552) grub-core/partmap/dfly.c:69:partition: bad magic (found 0x20494645; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0xeb63; wanted 0x4552) grub-core/kern/disk.c:299:disk: Closing hostdisk//dev/nvme0n1'. grub-core/partmap/gpt.c:93:gpt: Read a valid GPT header grub-core/partmap/gpt.c:115:gpt: GPT entry 0: start=4096, length=2093056 grub-probe: info: Partition 0 starts from 4096. grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/dfly.c:69:partition: bad magic (found 0x41615252; wanted 0xc4464c59) grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/apple.c:120:partition: bad magic (found 0xeb58; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 1: start=2097152, length=8388608 grub-probe: info: Partition 1 starts from 2097152. grub-core/kern/disk.c:299:disk: Closinghostdisk//dev/nvme0n1'. grub-probe: info: /dev/nvme0n1p2 is present. grub-probe: info: Looking for /dev/nvme0n1p2. grub-probe: info: /dev/nvme0n1 is a parent of /dev/nvme0n1p2. grub-probe: info: /dev/nvme0n1p2 starts from 2097152. grub-probe: info: opening the device hostdisk//dev/nvme0n1. grub-core/kern/disk.c:196:disk: Opening hostdisk//dev/nvme0n1'... grub-probe: info: drive = 0. grub-probe: info: the size of hostdisk//dev/nvme0n1 is 500118192. grub-core/kern/disk.c:196:disk: Openinghostdisk//dev/nvme0n1'... grub-probe: info: drive = 0. grub-probe: info: the size of hostdisk//dev/nvme0n1 is 500118192. grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:379:hostdisk: opening the device /dev/nvme0n1' in open_device() grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1' grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-core/partmap/gpt.c:93:gpt: Read a valid GPT header grub-core/partmap/gpt.c:115:gpt: GPT entry 0: start=4096, length=2093056 grub-core/partmap/gpt.c:115:gpt: GPT entry 1: start=2097152, length=8388608 grub-core/partmap/gpt.c:115:gpt: GPT entry 2: start=10485760, length=479146672 grub-core/partmap/gpt.c:115:gpt: GPT entry 3: start=489632432, length=8388608 grub-core/partmap/gpt.c:115:gpt: GPT entry 4: start=2048, length=2048 grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/gpt.c:93:gpt: Read a valid GPT header grub-core/partmap/gpt.c:115:gpt: GPT entry 0: start=4096, length=2093056 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/dfly.c:69:partition: bad magic (found 0x41615252; wanted 0xc4464c59) grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/apple.c:120:partition: bad magic (found 0xeb58; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 1: start=2097152, length=8388608 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x0; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x0; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 2: start=10485760, length=479146672 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x0; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x4c55; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 3: start=489632432, length=8388608 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x1c5a8c1a; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x8bac; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 4: start=2048, length=2048 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x821cea; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x5256; wanted 0x4552) grub-core/partmap/dfly.c:69:partition: bad magic (found 0x20494645; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0xeb63; wanted 0x4552) grub-core/kern/disk.c:299:disk: Closing hostdisk//dev/nvme0n1'. grub-core/partmap/gpt.c:93:gpt: Read a valid GPT header grub-core/partmap/gpt.c:115:gpt: GPT entry 0: start=4096, length=2093056 grub-probe: info: Partition 0 starts from 4096. grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/dfly.c:69:partition: bad magic (found 0x41615252; wanted 0xc4464c59) grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/apple.c:120:partition: bad magic (found 0xeb58; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 1: start=2097152, length=8388608 grub-probe: info: Partition 1 starts from 2097152. grub-core/kern/disk.c:299:disk: Closinghostdisk//dev/nvme0n1'. grub-probe: info: opening hostdisk//dev/nvme0n1,gpt2. grub-core/kern/disk.c:196:disk: Opening hostdisk//dev/nvme0n1,gpt2'... grub-probe: info: drive = 0. grub-probe: info: the size of hostdisk//dev/nvme0n1 is 500118192. grub-core/partmap/gpt.c:93:gpt: Read a valid GPT header grub-core/partmap/gpt.c:115:gpt: GPT entry 0: start=4096, length=2093056 grub-core/partmap/gpt.c:115:gpt: GPT entry 1: start=2097152, length=8388608 grub-core/kern/fs.c:56:fs: Detecting zfs... grub-core/osdep/hostdisk.c:379:hostdisk: opening the device/dev/nvme0n1p2' in open_device() grub-core/fs/zfs/zfs.c:1199:zfs: label ok 0 grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1p2' grub-core/fs/zfs/zfs.c:1014:zfs: check 2 passed grub-core/fs/zfs/zfs.c:1025:zfs: check 3 passed grub-core/fs/zfs/zfs.c:1032:zfs: check 4 passed grub-core/fs/zfs/zfs.c:1042:zfs: check 6 passed grub-core/fs/zfs/zfs.c:1050:zfs: check 7 passed grub-core/fs/zfs/zfs.c:1061:zfs: check 8 passed grub-core/fs/zfs/zfs.c:1071:zfs: check 9 passed grub-core/fs/zfs/zfs.c:1093:zfs: check 11 passed grub-core/fs/zfs/zfs.c:1119:zfs: check 10 passed grub-core/fs/zfs/zfs.c:1144:zfs: check 12 passed (feature flags) grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 4096/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = -1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 1803d8 grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1p2' grub-core/fs/zfs/zfs.c:2694:zfs: endian = -1, blkid=0 grub-core/fs/zfs/zfs.c:2031:zfs: endian = -1 grub-core/fs/zfs/zfs.c:2062:zfs: endian = -1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 131072/8192 grub-core/fs/zfs/zfs.c:1907:zfs: endian = -1 grub-core/fs/zfs/zfs.c:597:zfs: dva=10, 6803c8 grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1p2' grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2057:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 16384/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 500318 grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1p2' grub-core/fs/zfs/zfs.c:2699:zfs: alive grub-core/fs/zfs/zfs.c:2505:zfs: looking for 'features_for_read' grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2057:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 16384/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, a8 grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1p2' grub-core/fs/zfs/zfs.c:2515:zfs: zap read grub-core/fs/zfs/zfs.c:2528:zfs: fat zap grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2057:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 16384/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, a0 grub-core/fs/zfs/zfs.c:2288:zfs: fzap: length 18 grub-core/fs/zfs/zfs.c:2532:zfs: returned 0 grub-core/fs/zfs/zfs.c:2694:zfs: endian = -1, blkid=1 grub-core/fs/zfs/zfs.c:2031:zfs: endian = -1 grub-core/fs/zfs/zfs.c:2062:zfs: endian = -1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 131072/8192 grub-core/fs/zfs/zfs.c:1907:zfs: endian = -1 grub-core/fs/zfs/zfs.c:597:zfs: dva=10, 6803c8 grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2057:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 16384/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 500348 grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1p2' grub-core/fs/zfs/zfs.c:2699:zfs: alive grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2057:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 512/512 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 100050 grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1p2' grub-core/fs/zfs/zfs.c:2118:zfs: zap: name = , value = 0, cd = 0 grub-core/fs/zfs/zfs.c:2118:zfs: zap: name = , value = 0, cd = 0 grub-core/fs/zfs/zfs.c:2118:zfs: zap: name = , value = 0, cd = 0 grub-core/fs/zfs/zfs.c:2118:zfs: zap: name = , value = 0, cd = 0 grub-core/fs/zfs/zfs.c:2118:zfs: zap: name = , value = 0, cd = 0 grub-core/fs/zfs/zfs.c:2118:zfs: zap: name = , value = 0, cd = 0 grub-core/fs/zfs/zfs.c:2118:zfs: zap: name = , value = 0, cd = 0 grub-core/fs/zfs/zfs.c:3293:zfs: alive grub-core/fs/zfs/zfs.c:3105:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2694:zfs: endian = 1, blkid=0 grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2062:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 131072/8192 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=10, 6803c8 grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2057:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 16384/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 500318 grub-core/fs/zfs/zfs.c:2699:zfs: alive grub-core/fs/zfs/zfs.c:3112:zfs: alive grub-core/fs/zfs/zfs.c:2505:zfs: looking for 'root_dataset' grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2057:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 16384/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, a8 grub-core/fs/zfs/zfs.c:2515:zfs: zap read grub-core/fs/zfs/zfs.c:2528:zfs: fat zap grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2057:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 16384/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, a0 grub-core/fs/zfs/zfs.c:2288:zfs: fzap: length 13 grub-core/fs/zfs/zfs.c:2532:zfs: returned 0 grub-core/fs/zfs/zfs.c:3118:zfs: alive grub-core/fs/zfs/zfs.c:2694:zfs: endian = 1, blkid=1 grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2062:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 131072/8192 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=10, 6803c8 grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2057:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 16384/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 500348 grub-core/fs/zfs/zfs.c:2699:zfs: alive grub-core/fs/zfs/zfs.c:3124:zfs: alive grub-core/fs/zfs/zfs.c:3302:zfs: alive grub-core/fs/zfs/zfs.c:3306:zfs: endian = 0 grub-core/fs/zfs/zfs.c:3315:zfs: endian = 1 grub-core/fs/zfs/zfs.c:3170:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 4096/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 100038 grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1p2' grub-core/fs/zfs/zfs.c:3395:zfs: endian = 1 grub-core/fs/zfs/zfs.c:3170:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 4096/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 100038 grub-core/fs/zfs/zfs.c:2694:zfs: endian = 1, blkid=0 grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2062:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 131072/8192 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=10, 100018 grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2062:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 131072/8192 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=10, 200008 grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1p2' grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2062:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 131072/8192 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=10, 300000 grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1p2' grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2062:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 131072/8192 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=10, 100008 grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2062:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 131072/8192 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=10, 38 grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1p2' grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1p2' grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2057:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 16384/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 28 grub-core/fs/zfs/zfs.c:2699:zfs: alive grub-core/fs/zfs/zfs.c:2505:zfs: looking for 'ROOT' grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2057:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 512/512 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 10 grub-core/fs/zfs/zfs.c:2515:zfs: zap read grub-core/fs/zfs/zfs.c:2519:zfs: micro zap grub-core/fs/zfs/zfs.c:2522:zfs: returned 0 grub-core/fs/zfs/zfs.c:2694:zfs: endian = 1, blkid=1 grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2062:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 131072/8192 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=10, 100018 grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2062:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 131072/8192 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=10, 200008 grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2062:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 131072/8192 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=10, 300000 grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2062:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 131072/8192 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=10, 100008 grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2062:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 131072/8192 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=10, 38 grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2057:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 16384/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 30 grub-core/fs/zfs/zfs.c:2699:zfs: alive zfs grub-core/kern/disk.c:299:disk: Closing `hostdisk//dev/nvme0n1'.

@.***:~]#

Has anyone been able to reproduce this problem when creating bpool on a separate disk partition as the guide suggests?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

colmbuckley commented 1 year ago

You might be right about the part of testing a file based pool. But now I actually use my own guides on my physical laptop, I can conclusively demonstrate that compat=legacy on a baremental machine fixes the "bpool snapshot breaks GRUB compatibility" issue

Can you provide the same output from zpool get all and grub-probe in the case where it fails - ie: with compatibility=grub2? The above is expected behavior.

ghost commented 1 year ago

Still red herrings. I currently do not have time to investigate this further. As you can see from above, compat=legacy does solve this issue.

[root#yinzhou:~]# zpool history bpool History for 'bpool': 2023-11-08.23:26:23 zpool create -o compatibility=grub2 -o ashift=12 -o autotrim=on -O acltype=posixacl -O canmount=off -O devices=off -O normalization=formD -O relatime=on -O xattr=sa -O mountpoint=/boot bpool /dev/disk/by-id/nvme-SAMSUNG_MZVLV256HCHP-000H1_S2CSNA0J547878-part2 2023-11-08.23:26:34 zfs create -o mountpoint=none bpool/nixos 2023-11-08.23:27:03 zfs receive bpool/nixos/root 2023-11-08.23:27:41 zfs set mountpoint=legacy bpool/nixos/root 2023-11-08.23:28:11 zfs snap bpool#test

[root#yinzhou:~]# grub-probe /boot grub-probe: error: compression algorithm inherit not supported

[root#yinzhou:~]# grub-probe -vvv /boot grub-probe: info: cannot open /boot/grub/device.map': No such file or directory. grub-probe: info: /dev/nvme0n1p2 is not present. grub-probe: info: Looking for /dev/nvme0n1p2. grub-probe: info: /dev/nvme0n1 is a parent of /dev/nvme0n1p2. grub-probe: info: /dev/nvme0n1p2 starts from 2097152. grub-probe: info: opening the device hostdisk//dev/nvme0n1. grub-core/kern/disk.c:196:disk: Openinghostdisk//dev/nvme0n1'... grub-probe: info: drive = 0. grub-probe: info: the size of hostdisk//dev/nvme0n1 is 500118192. grub-core/kern/disk.c:196:disk: Opening hostdisk//dev/nvme0n1'... grub-probe: info: drive = 0. grub-probe: info: the size of hostdisk//dev/nvme0n1 is 500118192. grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:379:hostdisk: opening the device/dev/nvme0n1' in open_device() grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1' grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1' grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1' grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1' grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-core/partmap/gpt.c:93:gpt: Read a valid GPT header grub-core/partmap/gpt.c:115:gpt: GPT entry 0: start=4096, length=2093056 grub-core/partmap/gpt.c:115:gpt: GPT entry 1: start=2097152, length=8388608 grub-core/partmap/gpt.c:115:gpt: GPT entry 2: start=10485760, length=479146672 grub-core/partmap/gpt.c:115:gpt: GPT entry 3: start=489632432, length=8388608 grub-core/partmap/gpt.c:115:gpt: GPT entry 4: start=2048, length=2048 grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/gpt.c:93:gpt: Read a valid GPT header grub-core/partmap/gpt.c:115:gpt: GPT entry 0: start=4096, length=2093056 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:379:hostdisk: opening the device /dev/nvme0n1p1' in open_device() grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1p1' grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1p1' grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/dfly.c:69:partition: bad magic (found 0x41615252; wanted 0xc4464c59) grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/apple.c:120:partition: bad magic (found 0xeb58; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 1: start=2097152, length=8388608 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:379:hostdisk: opening the device/dev/nvme0n1p2' in open_device() grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1p2' grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1p2' grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x0; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x0; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 2: start=10485760, length=479146672 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:379:hostdisk: opening the device /dev/nvme0n1p3' in open_device() grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1p3' grub-core/osdep/hostdisk.c:379:hostdisk: opening the device /dev/nvme0n1' in open_device() grub-core/osdep/hostdisk.c:379:hostdisk: opening the device/dev/nvme0n1p3' in open_device() grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1p3' grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x0; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x4c55; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 3: start=489632432, length=8388608 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:379:hostdisk: opening the device/dev/nvme0n1p4' in open_device() grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1p4' grub-core/osdep/hostdisk.c:379:hostdisk: opening the device/dev/nvme0n1' in open_device() grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:379:hostdisk: opening the device /dev/nvme0n1p4' in open_device() grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x1c5a8c1a; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x8bac; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 4: start=2048, length=2048 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:379:hostdisk: opening the device/dev/nvme0n1p5' in open_device() grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1p5' grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1p5' grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x821cea; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x5256; wanted 0x4552) grub-core/partmap/dfly.c:69:partition: bad magic (found 0x20494645; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0xeb63; wanted 0x4552) grub-core/kern/disk.c:299:disk: Closing hostdisk//dev/nvme0n1'. grub-core/partmap/gpt.c:93:gpt: Read a valid GPT header grub-core/partmap/gpt.c:115:gpt: GPT entry 0: start=4096, length=2093056 grub-probe: info: Partition 0 starts from 4096. grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/dfly.c:69:partition: bad magic (found 0x41615252; wanted 0xc4464c59) grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/apple.c:120:partition: bad magic (found 0xeb58; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 1: start=2097152, length=8388608 grub-probe: info: Partition 1 starts from 2097152. grub-core/kern/disk.c:299:disk: Closinghostdisk//dev/nvme0n1'. grub-probe: info: /dev/nvme0n1p2 is present. grub-probe: info: Looking for /dev/nvme0n1p2. grub-probe: info: /dev/nvme0n1 is a parent of /dev/nvme0n1p2. grub-probe: info: /dev/nvme0n1p2 starts from 2097152. grub-probe: info: opening the device hostdisk//dev/nvme0n1. grub-core/kern/disk.c:196:disk: Opening hostdisk//dev/nvme0n1'... grub-probe: info: drive = 0. grub-probe: info: the size of hostdisk//dev/nvme0n1 is 500118192. grub-core/kern/disk.c:196:disk: Openinghostdisk//dev/nvme0n1'... grub-probe: info: drive = 0. grub-probe: info: the size of hostdisk//dev/nvme0n1 is 500118192. grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:379:hostdisk: opening the device /dev/nvme0n1' in open_device() grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1' grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-core/partmap/gpt.c:93:gpt: Read a valid GPT header grub-core/partmap/gpt.c:115:gpt: GPT entry 0: start=4096, length=2093056 grub-core/partmap/gpt.c:115:gpt: GPT entry 1: start=2097152, length=8388608 grub-core/partmap/gpt.c:115:gpt: GPT entry 2: start=10485760, length=479146672 grub-core/partmap/gpt.c:115:gpt: GPT entry 3: start=489632432, length=8388608 grub-core/partmap/gpt.c:115:gpt: GPT entry 4: start=2048, length=2048 grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/gpt.c:93:gpt: Read a valid GPT header grub-core/partmap/gpt.c:115:gpt: GPT entry 0: start=4096, length=2093056 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/dfly.c:69:partition: bad magic (found 0x41615252; wanted 0xc4464c59) grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/apple.c:120:partition: bad magic (found 0xeb58; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 1: start=2097152, length=8388608 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x0; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x0; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 2: start=10485760, length=479146672 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x0; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x4c55; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 3: start=489632432, length=8388608 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x1c5a8c1a; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x8bac; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 4: start=2048, length=2048 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x821cea; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x5256; wanted 0x4552) grub-core/partmap/dfly.c:69:partition: bad magic (found 0x20494645; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0xeb63; wanted 0x4552) grub-core/kern/disk.c:299:disk: Closing hostdisk//dev/nvme0n1'. grub-core/partmap/gpt.c:93:gpt: Read a valid GPT header grub-core/partmap/gpt.c:115:gpt: GPT entry 0: start=4096, length=2093056 grub-probe: info: Partition 0 starts from 4096. grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/dfly.c:69:partition: bad magic (found 0x41615252; wanted 0xc4464c59) grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/apple.c:120:partition: bad magic (found 0xeb58; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 1: start=2097152, length=8388608 grub-probe: info: Partition 1 starts from 2097152. grub-core/kern/disk.c:299:disk: Closinghostdisk//dev/nvme0n1'. grub-probe: info: /dev/nvme0n1p2 is present. grub-probe: info: Looking for /dev/nvme0n1p2. grub-probe: info: /dev/nvme0n1 is a parent of /dev/nvme0n1p2. grub-probe: info: /dev/nvme0n1p2 starts from 2097152. grub-probe: info: opening the device hostdisk//dev/nvme0n1. grub-core/kern/disk.c:196:disk: Opening hostdisk//dev/nvme0n1'... grub-probe: info: drive = 0. grub-probe: info: the size of hostdisk//dev/nvme0n1 is 500118192. grub-core/kern/disk.c:196:disk: Openinghostdisk//dev/nvme0n1'... grub-probe: info: drive = 0. grub-probe: info: the size of hostdisk//dev/nvme0n1 is 500118192. grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:379:hostdisk: opening the device /dev/nvme0n1' in open_device() grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1' grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-core/partmap/gpt.c:93:gpt: Read a valid GPT header grub-core/partmap/gpt.c:115:gpt: GPT entry 0: start=4096, length=2093056 grub-core/partmap/gpt.c:115:gpt: GPT entry 1: start=2097152, length=8388608 grub-core/partmap/gpt.c:115:gpt: GPT entry 2: start=10485760, length=479146672 grub-core/partmap/gpt.c:115:gpt: GPT entry 3: start=489632432, length=8388608 grub-core/partmap/gpt.c:115:gpt: GPT entry 4: start=2048, length=2048 grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/gpt.c:93:gpt: Read a valid GPT header grub-core/partmap/gpt.c:115:gpt: GPT entry 0: start=4096, length=2093056 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/dfly.c:69:partition: bad magic (found 0x41615252; wanted 0xc4464c59) grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/apple.c:120:partition: bad magic (found 0xeb58; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 1: start=2097152, length=8388608 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x0; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x0; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 2: start=10485760, length=479146672 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x0; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x4c55; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 3: start=489632432, length=8388608 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x1c5a8c1a; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x8bac; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 4: start=2048, length=2048 grub-core/disk/diskfilter.c:136:diskfilter: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1 grub-probe: info: Scanning for DISKFILTER devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid1x devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09 devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for mdraid09_be devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for dmraid_nv devices on disk hostdisk//dev/nvme0n1. grub-probe: info: Scanning for ldm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: scanning hostdisk//dev/nvme0n1 for LDM. grub-probe: info: no LDM signature found. grub-probe: info: Scanning for lvm devices on disk hostdisk//dev/nvme0n1. grub-probe: info: no LVM signature found. grub-core/partmap/dfly.c:69:partition: bad magic (found 0x821cea; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0x5256; wanted 0x4552) grub-core/partmap/dfly.c:69:partition: bad magic (found 0x20494645; wanted 0xc4464c59) grub-core/partmap/apple.c:120:partition: bad magic (found 0xeb63; wanted 0x4552) grub-core/kern/disk.c:299:disk: Closing hostdisk//dev/nvme0n1'. grub-core/partmap/gpt.c:93:gpt: Read a valid GPT header grub-core/partmap/gpt.c:115:gpt: GPT entry 0: start=4096, length=2093056 grub-probe: info: Partition 0 starts from 4096. grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/dfly.c:69:partition: bad magic (found 0x41615252; wanted 0xc4464c59) grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 0: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 1: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 2: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/msdos.c:184:partition: partition 3: flag 0x0, type 0x0, start 0x0, len 0x0 grub-core/partmap/apple.c:120:partition: bad magic (found 0xeb58; wanted 0x4552) grub-core/partmap/gpt.c:115:gpt: GPT entry 1: start=2097152, length=8388608 grub-probe: info: Partition 1 starts from 2097152. grub-core/kern/disk.c:299:disk: Closinghostdisk//dev/nvme0n1'. grub-probe: info: opening hostdisk//dev/nvme0n1,gpt2. grub-core/kern/disk.c:196:disk: Opening hostdisk//dev/nvme0n1,gpt2'... grub-probe: info: drive = 0. grub-probe: info: the size of hostdisk//dev/nvme0n1 is 500118192. grub-core/partmap/gpt.c:93:gpt: Read a valid GPT header grub-core/partmap/gpt.c:115:gpt: GPT entry 0: start=4096, length=2093056 grub-core/partmap/gpt.c:115:gpt: GPT entry 1: start=2097152, length=8388608 grub-core/kern/fs.c:56:fs: Detecting zfs... grub-core/osdep/hostdisk.c:379:hostdisk: opening the device/dev/nvme0n1p2' in open_device() grub-core/fs/zfs/zfs.c:1199:zfs: label ok 0 grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1p2' grub-core/fs/zfs/zfs.c:1014:zfs: check 2 passed grub-core/fs/zfs/zfs.c:1025:zfs: check 3 passed grub-core/fs/zfs/zfs.c:1032:zfs: check 4 passed grub-core/fs/zfs/zfs.c:1042:zfs: check 6 passed grub-core/fs/zfs/zfs.c:1050:zfs: check 7 passed grub-core/fs/zfs/zfs.c:1061:zfs: check 8 passed grub-core/fs/zfs/zfs.c:1071:zfs: check 9 passed grub-core/fs/zfs/zfs.c:1093:zfs: check 11 passed grub-core/fs/zfs/zfs.c:1119:zfs: check 10 passed grub-core/fs/zfs/zfs.c:1135:zfs: str=com.delphix:hole_birth grub-core/fs/zfs/zfs.c:1135:zfs: str=com.delphix:embedded_data grub-core/fs/zfs/zfs.c:1144:zfs: check 12 passed (feature flags) grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 4096/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = -1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 100040 grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1p2' grub-core/fs/zfs/zfs.c:2694:zfs: endian = -1, blkid=0 grub-core/fs/zfs/zfs.c:2031:zfs: endian = -1 grub-core/fs/zfs/zfs.c:2062:zfs: endian = -1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 131072/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = -1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 15570 grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1p2' grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2057:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 16384/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 200098 grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1p2' grub-core/fs/zfs/zfs.c:2699:zfs: alive grub-core/fs/zfs/zfs.c:2505:zfs: looking for 'features_for_read' grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2057:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 16384/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, b8 grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1p2' grub-core/fs/zfs/zfs.c:2515:zfs: zap read grub-core/fs/zfs/zfs.c:2528:zfs: fat zap grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2057:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 16384/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, b0 grub-core/fs/zfs/zfs.c:2288:zfs: fzap: length 18 grub-core/fs/zfs/zfs.c:2532:zfs: returned 0 grub-core/fs/zfs/zfs.c:2694:zfs: endian = -1, blkid=1 grub-core/fs/zfs/zfs.c:2031:zfs: endian = -1 grub-core/fs/zfs/zfs.c:2062:zfs: endian = -1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 131072/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = -1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 15570 grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2057:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 16384/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 2057c8 grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1p2' grub-core/fs/zfs/zfs.c:2699:zfs: alive grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2057:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 512/512 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 48f960 grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device /dev/nvme0n1p2' grub-core/fs/zfs/zfs.c:2118:zfs: zap: name = org.illumos:lz4_compress, value = 1, cd = 0 grub-core/fs/zfs/zfs.c:2118:zfs: zap: name = com.delphix:hole_birth, value = 1, cd = 0 grub-core/fs/zfs/zfs.c:2118:zfs: zap: name = com.delphix:extensible_dataset, value = 2, cd = 0 grub-core/fs/zfs/zfs.c:2118:zfs: zap: name = com.delphix:embedded_data, value = 1, cd = 0 grub-core/fs/zfs/zfs.c:2118:zfs: zap: name = org.open-zfs:large_blocks, value = 0, cd = 0 grub-core/fs/zfs/zfs.c:2118:zfs: zap: name = , value = 0, cd = 0 grub-core/fs/zfs/zfs.c:2118:zfs: zap: name = , value = 0, cd = 0 grub-core/fs/zfs/zfs.c:3293:zfs: alive grub-core/fs/zfs/zfs.c:3105:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2694:zfs: endian = 1, blkid=0 grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2062:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 131072/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 15570 grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2057:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 16384/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 200098 grub-core/fs/zfs/zfs.c:2699:zfs: alive grub-core/fs/zfs/zfs.c:3112:zfs: alive grub-core/fs/zfs/zfs.c:2505:zfs: looking for 'root_dataset' grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2057:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 16384/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, b8 grub-core/fs/zfs/zfs.c:2515:zfs: zap read grub-core/fs/zfs/zfs.c:2528:zfs: fat zap grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2057:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 16384/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, b0 grub-core/fs/zfs/zfs.c:2288:zfs: fzap: length 13 grub-core/fs/zfs/zfs.c:2532:zfs: returned 0 grub-core/fs/zfs/zfs.c:3118:zfs: alive grub-core/fs/zfs/zfs.c:2694:zfs: endian = 1, blkid=1 grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2062:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 131072/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 15570 grub-core/fs/zfs/zfs.c:2031:zfs: endian = 1 grub-core/fs/zfs/zfs.c:2057:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 16384/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 2057c8 grub-core/fs/zfs/zfs.c:2699:zfs: alive grub-core/fs/zfs/zfs.c:3124:zfs: alive grub-core/fs/zfs/zfs.c:3302:zfs: alive grub-core/fs/zfs/zfs.c:3306:zfs: endian = 0 grub-core/fs/zfs/zfs.c:3315:zfs: endian = 1 grub-core/fs/zfs/zfs.c:3170:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 4096/4096 grub-core/fs/zfs/zfs.c:1907:zfs: endian = 1 grub-core/fs/zfs/zfs.c:597:zfs: dva=8, 100020 grub-core/osdep/hostdisk.c:358:hostdisk: reusing open device/dev/nvme0n1p2' grub-core/fs/zfs/zfs.c:3395:zfs: endian = 1 grub-core/fs/zfs/zfs.c:3170:zfs: endian = 1 grub-core/fs/zfs/zfs.c:1885:zfs: zio_read: E 0: size 0/512 grub-core/kern/fs.c:79:fs: error: compression algorithm inherit not supported . grub-core/kern/fs.c:80:fs: zfs detection failed. grub-probe: error: compression algorithm inherit not supported .

[root#yinzhou:~]#

Colm @.***> writes:

You might be right about the part of testing a file based pool. But now I actually use my own guides on my physical laptop, I can conclusively demonstrate that compat=legacy on a baremental machine fixes the "bpool snapshot breaks GRUB compatibility" issue

Can you provide the same output from zpool get all and grub-probe in the case where it fails - ie: with compatibility=grub2? The above is expected behavior.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

ghost commented 1 year ago

Here is the bpool recreated with compat=legacy

[root#yinzhou:~]# zpool history bpool History for 'bpool': 2023-11-08.23:32:14 zpool create -o compatibility=legacy -o ashift=12 -o autotrim=on -O acltype=posixacl -O canmount=off -O devices=off -O normalization=formD -O relatime=on -O xattr=sa -O mountpoint=/boot bpool /dev/disk/by-id/nvme-SAMSUNG_MZVLV256HCHP-000H1_S2CSNA0J547878-part2 2023-11-08.23:32:23 zfs create -o mountpoint=none bpool/nixos 2023-11-08.23:32:32 zfs receive bpool/nixos/root 2023-11-08.23:32:39 zfs set mountpoint=legacy bpool/nixos/root 2023-11-08.23:33:14 zpool export bpool 2023-11-08.23:33:18 zpool import bpool 2023-11-08.23:33:55 zfs snap bpool#test

[root#yinzhou:~]# grub-probe /boot zfs

colmbuckley commented 1 year ago

Here is the bpool recreated with compat=legacy

This is very mysterious; something in the feature set enabled by compatibility=grub2 is clearly triggering a GRUB bug (and it is a bug, because somehow it's misinterpreting an inherited zpool attribute as being the name of a compression algorithm and barfing), but none of the features in that compatibility set should be even vaguely related.

I have definitely installed dozens of Debian systems using a bpool with compatibility=grub2 and this has been stable through kernel upgrades (to 6.4 though - I haven't tried 6.5), ZFS upgrades (to at least 2.1.11), GRUB reinstalls, and snapshots both automatic and manual of bpool and rpool; I have not encountered this issue. It might be a recent reversion with 2.1.13 or kernel 6.5.

Regardless, the correct long-term fix is definitely to try to identify which feature(s) trigger the behavior and remove them from the grub2 compatibility set, rather than to avoid using it completely (this is literally its purpose). I will try to bisect the problem tomorrow on a suitable sacrificial VM.

ghost commented 1 year ago

I will try to bisect the problem tomorrow on a suitable sacrificial VM.

That would be great. However, I suggest you to not place too much importance on the "error: compression algorithm inherit not supported" message. I have observed GRUB emitting this message where the boot process succeeded.

In other words, this error message might be completely irrelevant to the cause itself. If you have knowledge of GRUB internals, which I don't, I suggest you to recompile GRUB with all debugging features enabled, then look at the backtraces. GRUB is quite complex -- it is essentially a mini-OS.

colmbuckley commented 1 year ago

I agree; GRUB is a total bear, and very difficult to understand. Like others, I've switched to ZFSBootMenu for most of my systems; although I'd like to see it get another year or two of solid production experience before making it a formal recommendation.

We'll see if there's any movement on the upstream bug also.

ghost commented 1 year ago

Colm @.***> writes:

I agree; GRUB is a total bear, and very difficult to understand.

Just my two cents: with the transition to UEFI, the situation seems to become the following:

BIOS (1980-era, very simplistic) + GRUB (small OS)

to

UEFI (overengineered, considered bloated by some, its specification spans thousands of pages, buggy implementation abound) + relative simple bootloader

https://news.ycombinator.com/item?id=20608437

ahesford commented 1 year ago

Just my two cents: with the transition to UEFI, the situation seems to become the following: BIOS (1980-era, very simplistic) + GRUB (small OS) to UEFI (overengineered, considered bloated by some, its specification spans thousands of pages, buggy implementation abound) + relative simple bootloader

The popular criticisms of UEFI are not unreasonable, but calling GRUB complex and ZFSBootMenu relatively simple is an inversion of reality. The problems people have with GRUB are caused because it is too simple, which pushes complexity and error handling onto the user. Half of the GRUB woes experienced by ZFS users would vanish if GRUB took on added complexity to properly support the file system. Likewise, configuration is a hassle because there are layers of abstraction between what the executable expects and what users are expected to write. Furthermore, GRUB needs static configuration for all of the bits that users want to be dynamic. If GRUB were capable of parsing more convenient formats and could be more intelligent about discovering what it offers to boot, configuration for the user could be made straightforward.

ZFSBootMenu, on the other hand, is literally a Linux operating system with a custom init and user shell (written in bash!) added on top. It's easy to interact with because there is complex behavior to walk ZFS systems and find what it presents you, and driver support that lets us push the configurable bits into ZFS properties where they feel kind of natural. It's simple to develop because we don't manage all of the complexity of the kernel, ZFS drivers and runtime; we just copy them into the image and run them.

On UEFI systems, ZBM is easy to install because the complexity of the firmware means you can just drop a single file somewhere and the system will bring it up. On BIOS systems, launching ZFSBootMenu requires an intermediate loader (syslinux is a convenient choice) that requires configuration and user interaction because the firmware lacks these niceties.

colmbuckley commented 1 year ago

I am unable to replicate this, unfortunately:

Linux kernel: 6.5.3-1~bpo12+1 (2023-10-08) x86_64 GNU/Linux
ZFS version: zfs-2.1.13-1~bpo12+1
GRUB version: 2.06-13+deb12u1

All of these are the most up to date versions from Debian bookworm plus backports.

This is a minimal session to create a disk containing the bpool pool plus GRUB's infrastructure. $DISK is a full virtual disk attached to a (Google Cloud) VM; I have used an identical setup to install numerous full systems.

root@bpooltest:/boot# sgdisk --zap-all $DISK
Creating new GPT entries in memory.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
root@bpooltest:/boot# sgdisk -a1 -n1:24K:+1000K -t1:EF02 $DISK
Creating new GPT entries in memory.
Warning: Setting alignment to a value that does not match the disk's
physical block size! Performance degradation may result!
Physical block size = 4096
Logical block size = 512
Optimal alignment = 8 or multiples thereof.
The operation has completed successfully.
root@bpooltest:/boot# sgdisk     -n3:0:+1G      -t3:BF01 $DISK
The operation has completed successfully.
root@bpooltest:/boot# zpool create \
    -o ashift=12 \
    -o autotrim=on \
    -o compatibility=grub2 \
    -o cachefile=/etc/zfs/zpool.cache \
    -O devices=off \
    -O acltype=posixacl -O xattr=sa \
    -O compression=lz4 \
    -O normalization=formD \
    -O relatime=on \
    -O canmount=off -O mountpoint=/boot -R /mnt \
    bpool ${DISK}-part3
root@bpooltest:/boot# zfs create -o canmount=off -o mountpoint=none bpool/BOOT
root@bpooltest:/boot# zfs create -o mountpoint=/boot bpool/BOOT/debian
root@bpooltest:/boot# mount | grep mnt
bpool/BOOT/debian on /mnt/boot type zfs (rw,nodev,relatime,xattr,posixacl)
root@bpooltest:/boot# cp /boot/*6.5* /mnt/boot
root@bpooltest:/boot# grub-probe /mnt/boot
zfs
root@bpooltest:/boot# grub-install --boot-directory=/mnt/boot $DISK
Installing for i386-pc platform.
Installation finished. No error reported.
root@bpooltest:/boot# zfs snap bpool@test
root@bpooltest:/boot# grub-probe /mnt/boot
zfs
root@bpooltest:/boot# grub-install --boot-directory=/mnt/boot $DISK
Installing for i386-pc platform.
Installation finished. No error reported.
root@bpooltest:/boot# zfs snap -r bpool@test2
root@bpooltest:/boot# grub-install  --boot-directory=/mnt/boot $DISK
Installing for i386-pc platform.
Installation finished. No error reported.
root@bpooltest:/boot# grub-probe /mnt/boot
zfs

Is anyone able to trigger the GRUB bug with a setup similar to this?

ghost commented 1 year ago

I have

6.1.60-hardened1 zfs-2.2.0-1 zfs-kmod-2.2.0-1 grub-2.12-rc1

colmbuckley commented 1 year ago

But can you replicate the error with a sequence similar to what I used? Your earlier replication seems to bring a lot more data into the bpool with the nixos root dataset on it?

gmelikov commented 1 year ago

Sidenote: looks like there may be differences in distro-specific grub2 builds, I've had problems even with root pool with Suse several years ago while tested one of our guides https://github.com/openzfs/openzfs-docs/issues/258

colmbuckley commented 1 year ago

I wonder if this is a regression in the newest versions of GRUB; Debian's version is fairly old.

J4gQBqqR commented 12 months ago

Hi, active zfs user here. Following exactly the documentation's installation method. And I snapshot the root of bpool and rpool the other day. I "reproduced/confirmed" this reported failure of GRUB not able to boot my bpool.

Now I ends up with a system which cannot be booted, and I spend hours today trying to fix it without success.

Is there any fix that I can carry out on my system to mitigate my mis-operation of snapshotting my bpool?

Any suggestion?

ghost commented 12 months ago

J4gQBqqR @.***> writes:

Is there any fix that I can carry out on my system to mitigate my mis-operation of snapshotting my bpool? Any suggestion?

Yes, there is. You basically need to follow the first part of this installation guide. That is,

download and boot into Alpine Linux Extended (it has ZFS preinstalled).
Import and mount rpool, load keys if encrypted.
Import and use zfs-send to back up bpool.
Destroy and recreate bpool with current instructions, pay extra attention to the compatibility=legacy part.
Set mountpoint=legacy on the restored boot dataset, it should be bpool/archlinux/root by default.

... mitigate my mis-operation of snapshotting my bpool ...

This is not your fault, taking snapshots is expected to work but it broke GRUB. I consider this unacceptable and is an oversight on our part -- but I currently have no time to investigate it further, other than using compat=legacy or suggest a migration to ZBM.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

colmbuckley commented 12 months ago

This is almost definitely due to a regression in GRUB; as -o compatibility=grub2 was confirmed to be working (snapshots and all) as of a couple of months ago. It's unfortunately very time-consuming to figure out which zpool feature is triggering the GRUB bug (it should be removed from compatibility.d/grub2), and I have not yet been able to do this.

In the meantime, yes - recreating bpool with -o compatibility=legacy or migrating to ZBM are suitable workarounds.

J4gQBqqR commented 12 months ago

Thank you for your help. I will employ both methods and keep one method as redundancy for the future.

openzfs / openzfs-docs

Switch to ZFSBootMenu for Arch #463

dd if=/dev/zero of=test.img bs=1M count=1024

zpool create tpool -O mountpoint=legacy -o compatibility=grub2 /root/test.img

mount -t zfs tpool /mnt

grub-probe /mnt

zfs snap @.***

grub-probe /mnt

zpool destroy tpool

zpool create tpool -O mountpoint=legacy -o compatibility=legacy /root/test.img

mount -t zfs tpool /mnt

grub-probe /mnt

zfs snap @.***

grub-probe /mnt