ERROR: Failed to load the ${_module} module!

[Marietto2008]

Hello.

I tried to compile your code on a bhyve/VM running with FreeBSD 14.0-CURRENT. The first task I did has been to remove the old nvidia-driver with the command : pkg remove nvidia-driver. During the compilation this is what happened :

https://pastebin.ubuntu.com/p/NyV7VQn9qH/

within the guest/ VM I have passed thru the following gpus :

# lspci

00:04.0 VGA compatible controller: Device fb5d:40fb
00:07.0 Display controller: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] (rev 02)
00:08.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] (rev a1)
00:08.1 Audio device: NVIDIA Corporation TU102 High Definition Audio Controller (rev a1)
00:08.2 USB controller: NVIDIA Corporation TU102 USB 3.1 Host Controller (rev a1)
00:08.3 Serial bus controller: NVIDIA Corporation TU102 USB Type-C UCSI Controller (rev a1)

I've used the following bhyve parameters to start the guest :

bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-hd,/mnt/$vmdisk1'p2'/bhyve/img/BSD/FreeBSD/freebsd14.img,bootindex=1 \
-s 2,virtio-blk,/dev/$vmdisk5 \
-s 4,fbuf,tcp=0.0.0.0:5914,w=1600,h=950 \
-s 7:0,passthru,0/2/0 \
-s 8:0,passthru,2/0/0 \
-s 8:1,passthru,2/0/1 \
-s 8:2,passthru,2/0/2 \
-s 8:3,passthru,2/0/3 \
-s 9,virtio-net,tap14 \
-s 10,virtio-9p,sharename=/ \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm14 < /dev/null & sleep 2 && vncviewer 0:14

instead,the host is FreeBSD 13.1-RELEASE with the following hardware components :

# lspci

00:00.0 Host bridge: Intel Corporation 8th/9th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] (rev 0d)
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 0d)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) (rev 0d)
00:02.0 Display controller: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630] (rev 02)
00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #17 (rev f0)
00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0)
00:1f.0 ISA bridge: Intel Corporation Z390 Chipset LPC/eSPI Controller (rev 10)
00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10)
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
00:1f.5 Serial bus controller: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-V (rev 10)
01:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1)
02:00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] (rev a1)
02:00.1 Audio device: NVIDIA Corporation TU102 High Definition Audio Controller (rev a1)
02:00.2 USB controller: NVIDIA Corporation TU102 USB 3.1 Host Controller (rev a1)
02:00.3 Serial bus controller: NVIDIA Corporation TU102 USB Type-C UCSI Controller (rev a1)
03:00.0 Non-Volatile memory controller: Micron/Crucial Technology Device 5403 (rev 03)

you maybe interested to explore the reason why I get this error :

[amshafer]

This commit in 520.60.11 should fix the build issue you saw. Do you know what branch you are on? You should use 520.60.11. I updated the default branch to that a bit ago, so depending when you cloned the repo you may have gotten the old branch that doesn't contain this fix.

Regarding the image you sent, it's telling you that there is a version mismatch when trying to load. You need to have drm-kmod installed, either through pkg or building from source. If you use drm-kmod from pkg then make sure your copy of the source that you point nvidia-drm at (i.e. /root/git/drm-kmod) is the same version (probably the 5.10-lts branch)

[Marietto2008]

yes,I know. Today I've used the 520.60.11/default and the drm-kmod/master. I'd thought that the drm-kmod would have been installed by the make install command after having cloned your repo. Infact I saw that it created 3 nvidia libraries inside the directory /boot/modules,so the script almost worked,but right after that the error stopped the procedure.

[amshafer]

So with 520.60.11 the build is working fine correct?

I'd thought that the drm-kmod would have been installed by the make install command after having cloned your repo

The error you're getting at the end of make install is when it tries to kldload nvidia-modeset. Given the line I quoted above it sounds like you don't actually have drm-kmod installed which would explain the error message? If you want drm-kmod installed you'll have to do so by building/installing from its own repo, nvidia-drm isn't going to do that for you.

[Marietto2008]

It does not work. These are the commands issued :

root@marietto:/home/marietto/Desktop # pkg install drm-kmod

root@marietto:/home/marietto/Desktop # pw usermod marietto -G video

root@marietto:/home/marietto/Desktop # git clone https://github.com/freebsd/drm-kmod --branch master

root@marietto:/home/marietto/Desktop # cp -r drm-kmod /root/git

root@marietto:/home/marietto/Desktop # git clone https://github.com/amshafer/nvidia-driver.git

root@marietto:/home/marietto/Desktop # cd nvidia-driver

root@marietto:/home/marietto/Desktop/nvidia-driver # cd nvidia

root@marietto:/home/marietto/Desktop/nvidia-driver/nvidia # make

no errors

root@marietto:/home/marietto/Desktop/nvidia-driver/nvidia # make install

.....
===> x11 (install)
===> x11/driver (install)
===> x11/extension (install)
===> x11/bin (install)
===> x11/man (install)
===> doc (install)
===> nvml (install)
===> nvml/bin (install)
===> nvml/man (install)
ERROR: Failed to load the ${_module} module!
*** Error code 1

[Marietto2008]

root@marietto:/home/marietto/Desktop/nvidia-driver/nvidia # rm -r /boot/modules/nvidia*

root@marietto:/home/marietto/Desktop/nvidia-driver/nvidia # pkg install nvidia-driver
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        nvidia-driver: 510.60.02

Number of packages to be installed: 1

The process will require 229 MiB more space.
77 MiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching nvidia-driver-510.60.02.pkg: 100%   77 MiB   9.0MB/s    00:09    
Checking integrity... done (0 conflicting)
[1/1] Installing nvidia-driver-510.60.02...
[1/1] Extracting nvidia-driver-510.60.02: 100%
=====
Message from nvidia-driver-510.60.02:

--
To use these drivers, make sure that you have loaded the NVidia kernel
module, by running

        # kldload nvidia                (or nvidia-modeset, see below)

on the command line, or by putting ``nvidia'' (or ``nvidia-modeset'',
see below again) on ``kld_list'' variable in /etc/rc.conf, either manually
or by running

        # sysrc kld_list+=nvidia        (or nvidia-modeset, see below)

If you build this port with FreeBSD AGP GART driver, make sure you have
agp.ko kernel module installed and loaded, since nvidia.ko will depend
on it, or have your kernel compiled with "device agp".  Otherwise, the
NVidia kernel module will not load.  Also, please set correct value for
``Option "NvAGP"'' in ``Device'' section of your X11 configuration file.

When building with Linux compatibility support, make sure that linux.ko
module is available as well (or have it compiled in kernel).  It can be
loaded via /boot/loader.conf, or later in the boot process if you add

        linux_enable="YES"

to your /etc/rc.conf.

If X.org cannot start and reports

        (EE) NVIDIA(0): Failed to obtain a shared memory identifier.

in /var/log/Xorg.0.log while actually you have ``options SYSVSHM''
enabled in kernel, the sysctl ``kern.ipc.shmall'' should be increased.

Starting with version 358.09, some important functionality of the driver
has been broken out into a separate kernel module, nvidia-modeset.ko.
Users that experience hangs when starting X11 server, or observe

        (II) NVIDIA(0): Validated MetaModes:
        (II) NVIDIA(0):     "NULL"

in /var/log/Xorg.0.log should replace ``nvidia'' with ``nvidia-modeset''
in their /boot/loader.conf or /etc/rc.conf configuration files, depending
on how they prefer to load NVidia driver kernel module.

See /usr/local/share/doc/NVIDIA_GLX-1.0/README for more information.

root@marietto:/home/marietto/Desktop/nvidia-driver/nvidia # ls /boot/modules/nv*
/boot/modules/nvidia-modeset.ko /boot/modules/nvidia.ko

root@marietto:/home/marietto/Desktop/nvidia-driver/nvidia # kldload nvidia-modeset

kldload: an error occurred while loading module nvidia-modeset. Please check dmesg(8) for more details.

root@marietto:/home/marietto/Desktop/nvidia-driver/nvidia # dmesg

KLD drm.ko: depends on kernel - not available or version mismatch
linker_load_file: /boot/modules/drm.ko - unsupported file type
KLD nvidia.ko: depends on drmn - not available or version mismatch
linker_load_file: /boot/modules/nvidia.ko - unsupported file type
KLD drm.ko: depends on kernel - not available or version mismatch
linker_load_file: /boot/modules/drm.ko - unsupported file type
KLD nvidia.ko: depends on drmn - not available or version mismatch
linker_load_file: /boot/modules/nvidia.ko - unsupported file type
KLD nvidia-modeset.ko: depends on nvidia - not available or version mismatch
linker_load_file: /boot/modules/nvidia-modeset.ko - unsupported file type
KLD nvidia-modeset.ko: depends on kernel - not available or version mismatch
linker_load_file: /boot/modules/nvidia-modeset.ko - unsupported file type

[amshafer]

ERROR: Failed to load the ${_module} module!

This line means that the nvidia driver built correctly but could not be loaded due to the version mismatch or missing drm.ko module as we discussed.

KLD drm.ko: depends on kernel - not available or version mismatch

It looks to me like you're not building things to properly target your exact version. In a 14.0-CURRENT installation the packages are going to not be perfectly in sync with what you have installed since the packages are built on whatever the "latest" CURRENT is (and that gets updated with each commit). You should use uname to find the commit hash for your kernel, check out /usr/src at that commit, then build nvidia-drm and drm-kmod to target that version. Then everything should load and have compatible versions. You can probably find more in-detail guides on running CURRENT elsewhere.

[Marietto2008]

Man,I'm using the latest version of packages and ports (I've just upgraded both "portsnap fetch extract and portsnap fetch update for the ports and pkg update ; pkg upgrade" for the packages. And yes,I'm using "pkg+http://pkg.FreeBSD.org/${ABI}/latest") and I saw that the nvidia driver is at version : "510.60.02". This is the version that is not accepted. So,can you tell me where I can get the version which works ? thanks.

[Marietto2008]

# uname -a

FreeBSD marietto 14.0-CURRENT FreeBSD 14.0-CURRENT #0 main-n259276-5d42ef55dec1: Thu Nov 17 09:06:29 UTC 2022     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64

[Marietto2008]

I've upgraded the source code with the following commands :

# cd usr
# mv src src-old
# git clone https://git.FreeBSD.org/src.git /usr/src
# cd /usr/src
# make -j4 buildworld
# make -j4 kernel
# shutdown -r now
# cd /usr/src
# make installworld
# shutdown -r now

and voila'. it worked :

Installation of the NVIDIA Accelerated Graphics Driver 525.60.11 for FreeBSD is now complete. You can now run the nvidia-xconfig utility to automatically update your X server configuration file. Please see the README for details if you wish to update your X configuration file manually.

Doing like that there is no need to remove and / or reinstall the nvidia kernel modules installed with the previous installation of FreeBSD 14-CURRENT. So,something changes in the source code of the OS,but not on the source code of the nvidia kernel modules ? . Anyway,I don't know if actually it is running the nvidia driver 510 or 525. Because your script says 525,but I have installed the nvidia driver 510. It seems to me that your script does not kldload the nvidia kernel module 525,but the 510. So,I don't understand what's the goal of your project,if the system can't load the kernel module that it compiles.

I suggest to you to change your script. You could force the upgrade of the source code of the 14-CURRENT to the latest version,so that the code is in sync with the correct nvidia modules,otherwise a confusion arises.

[amshafer]

It sounds like you have successfully installed 520.60.11, replacing the existing 510 driver. If you run nvidia-smi I think it should show that you have the 520 driver loaded. You'll want to unload nvidia.ko and nvidia-modeset.ko and run kldload nvidia-drm. You have have installed 510 through pkg but installing the DRM-aware driver from this github repository will overwrite all the 510 files and your actual installed version will be 520 (despite what pkg says). Mixing packages and custom versions usually involves pkg lock nvidia-driver to keep pkg from updating it again.

[Marietto2008]

If I have understood well,I have to add the entry "nvidia-drm" to the file /etc/rc.conf,like this ?

/etc/rc.conf :

kld_list="nvidia-drm"
#kld_list="nvidia nvidia-modeset"

It didn't work :

[Marietto2008]

[amshafer]

Interesting, haven't seen that one yet. I tested CURRENT earlier this week without issue but haven't tried with VM passthrough.

Can you please try to view that in kgdb to see some more info? Basically replicate the panic, type the dump command (Or panic works too iirc), reboot without loading the modules, then do kgdb -n last. Should give some more details about the call stack. I assume your VM set up dumping to swap out of the box so hopefully it shouldn't give you any trouble.

[Marietto2008]

# kgdb -n last 
-sh: kgdb: not found

[amshafer]

pkg install gdb is probably what you're missing

[Marietto2008]

root@marietto:/home/marietto # kgdb -n last

GNU gdb (GDB) 12.1 [GDB v12.1 for FreeBSD]
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd14.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/obj/usr/src/amd64.amd64/sys/GENERIC/kernel.full...

Unread portion of the kernel message buffer:

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0xd8
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff86f63d47
stack pointer           = 0x28:0xfffffe009f392758
frame pointer           = 0x28:0xfffffe009f392770
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 1167 (kldload)
rdi: fffff800016e9000 rsi:                2 rdx: ffffffff812a4bf9
rcx:                1  r8:                0  r9:                0
rax:                0 rbx: fffff800016e9000 rbp: fffffe009f392770
r10:                0 r11:                0 r12: fffff800011ea200
r13: fffffe009f392da0 r14:                0 r15: ffffffff86f72130
trap number             = 12
panic: page fault
cpuid = 0
time = 1672692345
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe009f392510
vpanic() at vpanic+0x151/frame 0xfffffe009f392560
panic() at panic+0x43/frame 0xfffffe009f3925c0
trap_fatal() at trap_fatal+0x409/frame 0xfffffe009f392620
trap_pfault() at trap_pfault+0xab/frame 0xfffffe009f392680
calltrap() at calltrap+0x8/frame 0xfffffe009f392680
--- trap 0xc, rip = 0xffffffff86f63d47, rsp = 0xfffffe009f392758, rbp = 0xfffffe009f392770 ---
drm_core_check_all_features() at drm_core_check_all_features+0x17/frame 0xfffffe009f392770
drm_core_check_feature() at drm_core_check_feature+0x1b/frame 0xfffffe009f392790
nv_drm_load() at nv_drm_load+0x5f/frame 0xfffffe009f392930
drm_dev_register() at drm_dev_register+0x148/frame 0xfffffe009f392960
nv_drm_register_drm_device() at nv_drm_register_drm_device+0x156/frame 0xfffffe009f3929a0
nv_drm_probe_devices() at nv_drm_probe_devices+0xc9/frame 0xfffffe009f3929f0
nv_drm_init() at nv_drm_init+0x50/frame 0xfffffe009f392a10
nvidia_drm_evh() at nvidia_drm_evh+0x3c/frame 0xfffffe009f392a40
module_register_init() at module_register_init+0xbd/frame 0xfffffe009f392a70
linker_load_module() at linker_load_module+0xbbf/frame 0xfffffe009f392d80
kern_kldload() at kern_kldload+0x16f/frame 0xfffffe009f392dd0
sys_kldload() at sys_kldload+0x5b/frame 0xfffffe009f392e00
amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe009f392f30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe009f392f30
--- syscall (304, FreeBSD ELF64, kldload), rip = 0x292ad730d82a, rsp = 0x292ad526c2a8, rbp = 0x292ad526c820 ---
KDB: enter: panic

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:59
59              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,
(kgdb) 

[amshafer]

Thanks, although it looks like you're missing debug symbols. You'll want to rebuild and reinstall nvidia-drm and drm-kmod with something like this:

$ CFLAGS='-g -O0' DRM_DEBUG_LOG_ALL=1 DEBUG_FLAGS='-g -O0' WITH_DEBUG=1 WITH_CTF=1 make
$ CFLAGS='-g -O0' DRM_DEBUG_LOG_ALL=1 DEBUG_FLAGS='-g -O0' WITH_DEBUG=1 WITH_CTF=1 make install

This is the "all debug symbols, all logging, dtrace enabled" env vars I use.

[Marietto2008]

root@marietto:/home/marietto # kgdb -n last

GNU gdb (GDB) 12.1 [GDB v12.1 for FreeBSD]
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd14.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/obj/usr/src/amd64.amd64/sys/GENERIC/kernel.full...

Unread portion of the kernel message buffer:

Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 03
fault virtual address   = 0xd8
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff83bc8d47
stack pointer           = 0x28:0xfffffe009f3ac758
frame pointer           = 0x28:0xfffffe009f3ac770
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 5750 (kldload)
rdi: fffff800036ee000 rsi:                2 rdx: ffffffff812a4bf9
rcx:                1  r8:                0  r9:                0
rax:                0 rbx: fffff800036ee000 rbp: fffffe009f3ac770
r10:                0 r11:                0 r12: fffff8000370e180
r13: fffffe009f3acda0 r14:                0 r15: ffffffff83bd7130
trap number             = 12
panic: page fault
cpuid = 3
time = 1672779095
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe009f3ac510
vpanic() at vpanic+0x151/frame 0xfffffe009f3ac560
panic() at panic+0x43/frame 0xfffffe009f3ac5c0
trap_fatal() at trap_fatal+0x409/frame 0xfffffe009f3ac620
trap_pfault() at trap_pfault+0xab/frame 0xfffffe009f3ac680
calltrap() at calltrap+0x8/frame 0xfffffe009f3ac680
--- trap 0xc, rip = 0xffffffff83bc8d47, rsp = 0xfffffe009f3ac758, rbp = 0xfffffe009f3ac770 ---
drm_core_check_all_features() at drm_core_check_all_features+0x17/frame 0xfffffe009f3ac770
drm_core_check_feature() at drm_core_check_feature+0x1b/frame 0xfffffe009f3ac790
nv_drm_load() at nv_drm_load+0x5f/frame 0xfffffe009f3ac930
drm_dev_register() at drm_dev_register+0x148/frame 0xfffffe009f3ac960
nv_drm_register_drm_device() at nv_drm_register_drm_device+0x156/frame 0xfffffe009f3ac9a0
nv_drm_probe_devices() at nv_drm_probe_devices+0xc9/frame 0xfffffe009f3ac9f0
nv_drm_init() at nv_drm_init+0x50/frame 0xfffffe009f3aca10
nvidia_drm_evh() at nvidia_drm_evh+0x3c/frame 0xfffffe009f3aca40
module_register_init() at module_register_init+0xbd/frame 0xfffffe009f3aca70
linker_load_module() at linker_load_module+0xbbf/frame 0xfffffe009f3acd80
kern_kldload() at kern_kldload+0x16f/frame 0xfffffe009f3acdd0
sys_kldload() at sys_kldload+0x5b/frame 0xfffffe009f3ace00
amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe009f3acf30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe009f3acf30
--- syscall (304, FreeBSD ELF64, kldload), rip = 0x181eac3b982a, rsp = 0x181ea9cb38f8, rbp = 0x181ea9cb3e70 ---
KDB: enter: panic

__curthread () at /usr/src/sys/amd64/include/pcpu_aux.h:59
59              __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (offsetof(struct pcpu,

[Marietto2008]

its the same as before. Anyway I have also a crash dump log file. Send me an email here and I will send it to you : marietto2008@gmail.com