[After FreeBSD update] OpenZFS: load /boot/modules/openzfs.ko … bzf_open: not a file – can't find 'kernel'

[grahamperrin]

0.4.0 (OD26)

FreeBSD installed whilst booted from 0.4.0 (OD26) in live mode. OS and packages updated.

OpenZFS installed, /boot/loader.conf edited, system restarted:

I'll start a snapshot of the virtual machine from before the restart, attempt to understand the issue …

[grahamperrin]

From a snapshot:

date ; uptime ; uname -v
Sat Feb  6 11:39:09 EST 2021
11:39AM  up 9 mins, 0 users, load averages: 0.24, 0.25, 0.16
FreeBSD 12.2-RELEASE-p3 GENERIC 
mowa219-gjp4-vm-hellosystem-040-0d26% freebsd-version -kru
12.2-RELEASE-p3
12.2-RELEASE-p3
12.2-RELEASE-p3
mowa219-gjp4-vm-hellosystem-040-0d26% pkg query '%o %v %R' furybsd-settings-0 hello openzfs openzfs-kmod     
local 0 unknown-repository
helloSystem 0.4.0_0D26 unknown-repository
sysutils/openzfs 2021012500 FreeBSD
sysutils/openzfs-kmod 2021012500 FreeBSD
mowa219-gjp4-vm-hellosystem-040-0d26% kldstat | grep zfs
 2    1 0xffffffff8247b000   3bad38 zfs.ko
mowa219-gjp4-vm-hellosystem-040-0d26% cat /boot/loader.conf
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
opensolaris_load="YES"
zfs_load="NO"
openzfs_load="YES"
# TODO: Transfer settings from Live ISO /boot/loader.conf to installed one
# https://github.com/furybsd/furybsd-livecd/issues/190
boot_mute="YES"
beastie_disable="YES"
autoboot_delay="3"
hw.psm.elantech_support="1"
hw.psm.synaptics_support="1"
mowa219-gjp4-vm-hellosystem-040-0d26% beadm list
BE      Active Mountpoint  Space Created
default NR     /            4.4G 2021-02-04 14:49
mowa219-gjp4-vm-hellosystem-040-0d26% lsblk
DEVICE         MAJ:MIN SIZE TYPE                              LABEL MOUNT
ada0             0:87  256G GPT                                   - -
  ada0p1         0:88  200M efi                        gpt/efiboot0 -
  ada0p2         0:89  512K freebsd-boot               gpt/gptboot0 -
  <FREE>         -:-   492K -                                     - -
  ada0p3         0:90  256G freebsd-zfs                    gpt/zfs0 <ZFS>
  <FREE>         -:-   1.0M -                                     - -
mowa219-gjp4-vm-hellosystem-040-0d26% gpart show
=>       40  536870832  ada0  GPT  (256G)
         40     409600     1  efi  (200M)
     409640       1024     2  freebsd-boot  (512K)
     410664        984        - free -  (492K)
     411648  536457216     3  freebsd-zfs  (256G)
  536868864       2008        - free -  (1.0M)

mowa219-gjp4-vm-hellosystem-040-0d26% 

[grahamperrin]

Scrubbed, and gdisk(8) finds no problem with the GPT:

mowa219-gjp4-vm-hellosystem-040-0d26% zpool status
  pool: zroot
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 0 days 00:04:27 with 0 errors on Sat Feb  6 13:26:13 2021
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          ada0p3    ONLINE       0     0     0

errors: No known data errors
mowa219-gjp4-vm-hellosystem-040-0d26% zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zroot   254G  4.38G   250G        -         -     0%     1%  1.00x  ONLINE  -
mowa219-gjp4-vm-hellosystem-040-0d26% sudo pkg install gdisk
Password:
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        gdisk: 1.0.6

Number of packages to be installed: 1

141 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching gdisk-1.0.6.txz: 100%  141 KiB 144.3kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/1] Installing gdisk-1.0.6...
[1/1] Extracting gdisk-1.0.6: 100%
=====
Message from gdisk-1.0.6:

--
PLEASE, READ DOCUMENTATION BEFORE USE

See `man gdisk' , `man cgdisk', `man sgdisk', `man fixparts'.

Any errors in usage gdisk (cgdisk, sgdisk, fixparts) or
not understanding of that you do, can call loss of your data.
Make a backup copy of all data of a disk before you use
these programs for its processing.
mowa219-gjp4-vm-hellosystem-040-0d26% sudo gdisk /dev/ada0
GPT fdisk (gdisk) version 1.0.6

NOTE: Write test failed with error number 1. It will be impossible to save
changes to this disk's partition table!
You may be able to enable writes by exiting this program, typing
'sysctl kern.geom.debugflags=16' at a shell prompt, and re-running this
program.

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): p
Disk /dev/ada0: 536870912 sectors, 256.0 GiB
Sector size (logical): 512 bytes
Disk identifier (GUID): 20E8636C-6722-11EB-B495-080027DC743A
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 40, last usable sector is 536870871
Partitions will be aligned on 8-sector boundaries
Total free space is 2992 sectors (1.5 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              40          409639   200.0 MiB   EF00  efiboot0
   2          409640          410663   512.0 KiB   A501  gptboot0
   3          411648       536868863   255.8 GiB   A504  zfs0

Command (? for help): q
mowa219-gjp4-vm-hellosystem-040-0d26% 

[grahamperrin]

Removing the two packages, let's try installation from ports instead:

mowa219-gjp4-vm-hellosystem-040-0d26% cd /usr/ports/sysutils/openzfs
mowa219-gjp4-vm-hellosystem-040-0d26% sudo pkg remove openzfs openzfs-kmod
Password:
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 2 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
        openzfs: 2021012500
        openzfs-kmod: 2021012500

Number of packages to be removed: 2

The operation will free 22 MiB.

Proceed with deinstalling packages? [y/N]: y
[1/2] Deinstalling openzfs-2021012500...
[1/2] Deleting files for openzfs-2021012500: 100%
[2/2] Deinstalling openzfs-kmod-2021012500...
[2/2] Deleting files for openzfs-kmod-2021012500: 100%
mowa219-gjp4-vm-hellosystem-040-0d26% cd ../openzfs-kmod
mowa219-gjp4-vm-hellosystem-040-0d26% sudo pkg install --automatic gmake python37 autoconf automake libtool
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 6 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        autoconf: 2.69_3
        autoconf-wrapper: 20131203
        automake: 1.16.3
        gmake: 4.3_2
        libtool: 2.4.6_1
        m4: 1.4.18_1,1

Number of packages to be installed: 6

The process will require 10 MiB more space.
2 MiB to be downloaded.

Proceed with this action? [y/N]: y
[1/6] Fetching gmake-4.3_2.txz: 100%  416 KiB 425.9kB/s    00:01    
[2/6] Fetching autoconf-2.69_3.txz: 100%  543 KiB 555.6kB/s    00:01    
[3/6] Fetching automake-1.16.3.txz: 100%  549 KiB 561.7kB/s    00:01    
[4/6] Fetching libtool-2.4.6_1.txz: 100%  357 KiB 365.2kB/s    00:01    
[5/6] Fetching m4-1.4.18_1,1.txz: 100%  208 KiB 213.5kB/s    00:01    
[6/6] Fetching autoconf-wrapper-20131203.txz: 100%    3 KiB   2.6kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/6] Installing m4-1.4.18_1,1...
[1/6] Extracting m4-1.4.18_1,1: 100%
[2/6] Installing autoconf-wrapper-20131203...
[2/6] Extracting autoconf-wrapper-20131203: 100%
[3/6] Installing autoconf-2.69_3...
[3/6] Extracting autoconf-2.69_3: 100%
[4/6] Installing gmake-4.3_2...
[4/6] Extracting gmake-4.3_2: 100%
[5/6] Installing automake-1.16.3...
[5/6] Extracting automake-1.16.3: 100%
[6/6] Installing libtool-2.4.6_1...
[6/6] Extracting libtool-2.4.6_1: 100%
mowa219-gjp4-vm-hellosystem-040-0d26% pwd
/usr/ports/sysutils/openzfs-kmod
mowa219-gjp4-vm-hellosystem-040-0d26% make install clean
===>  openzfs-kmod-2021012500 requires kernel source files in
SRC_BASE=/usr/src.
*** Error code 1

Stop.
make: stopped in /usr/ports/sysutils/openzfs-kmod
mowa219-gjp4-vm-hellosystem-040-0d26% sudo git -C /usr clone -o freebsd -b stable/12 --depth 1 https://git.freebsd.org/src.git src
Password:
Cloning into 'src'...
…

The same:

Mounting from zfs:zroot/ROOT/default failed with error 2: unknown file system.

Investigating:

… bzf_open: not a file …

https://cgit.freebsd.org/src/tree/lib/libstand/bzipfs.c?h=stable/12&id=13b21828a8a6b2943944a05f9bc7e9a7432846fe#n172

[grahamperrin]

With end of life FreeBSD 12.1-RELEASE and the OpenZFS packages from latest, no problem:

[grahamperrin]

Comparing with a non-helloSystem 12.2-RELEASE-p3 system that boots from UFS, it seems that installations from ports is a must at this time:

Similarly, for helloSystem (0.3.0?) with packages (not installed from ports) before an upgrade from 12.1⋯ to 12.2-RELEASE-p3, there's this after restarting into 12.2-RELEASE-p3:

link_elf_obj: symbol vm_page_max_wired undefined
KLD file openzfs.ko - could not finalize loading

[probonopd]

Things like this are the reson why I am not in favor of using pkg as a way to update the base system and would rather like to see the base system be contained in one single image that gets (binary delta) updated.

Please keep in mind that updating helloSystem with pkg is not a supported aspect at this time (= possible but entirely untested and not actively debugged).

[grahamperrin]

updating helloSystem with pkg is not a supported aspect at this time

:+1:

I'm currently installing from ports in a clean-ish non-helloSystem FreeBSD 12.2 to tell wheether the issue is helloSystem-specific …

[grahamperrin]

https://github.com/helloSystem/ISO/issues/125#issuecomment-774528428

My bad:

sudo git -C /usr clone -o freebsd -b stable/12 --depth 1 https://git.freebsd.org/src.git src

Wrong branch! For 12.2-RELEASE-p3 it should have been:

sudo git -C /usr clone -o freebsd -b releng/12.2 --depth 1 https://git.freebsd.org/src.git src

Reference: https://matrix.to/#/!qZXgYtEIHagWHrEQQb:matrix.org/$1612705540454645SmOSn:matrix.org?via=matrix.org&via=t2bot.io

releng/12.2 or better yet, the specific commit on that which corresponds to -p3 (I don't know if RE have started doing proper tags on patch releases yet, but they need to start doing that)

[grahamperrin]

When I raised this incident, I did not realise that packages for 12.2 were built from 12.1 jails. That approach is ending.

From https://matrix.to/#/!qZXgYtEIHagWHrEQQb:matrix.org/$1612636778280808wSFLC:matrix.org?via=matrix.org&via=t2bot.io yesterday:

… the 12.x repo is built on 12.1 until after 12.1 is EOL, and then it's built on 12.2

… a full build takes nearly 4 days …

12.1 https://pkg-status.freebsd.org/builds/default:default:121amd64:563269:beefy6 began on Saturday 30th January, ran for around ninety-one hours:

• openzfs was not amongst the failures.

12.2 https://pkg-status.freebsd.org/builds/default:default:122amd64:563982:beefy6 began on Thursday 4th February, around eighty-seven hours elapsed:

• given the 12.1 success above and the 12.2 success below, it's reasonable to assume that this build will include sysutils/openzfs-kmod.

With sysutils/openzfs from the latest repo and sysutils/openzfs-kmod from ports:

• boot succeeds
• as far as I can tell, the opensolaris kernel module is not required.