vm-bhyve does not allow to passthru my GPU to a Windows vm.

[Marietto2008]

Hello to everyone. I'm trying to passthru my GPU (RTX 2080 ti) from the host OS (FreeBSD 14.0 to my Windows 10 guest os). This is how I have configured it :

nano /boot/loader.conf :

pptdevs="2/0/0 2/0/1 2/0/2 2/0/3"

nano /etc/rc.conf :

vm_enable="YES"
vm_dir="/mnt/da3p2/bhyve/img/Windows"

nano /mnt/da3p2/bhyve/img/Windows/Windows10/Windows10.conf

passthru0="2/0/0=8:0"
passthru1="2/0/1=8:1"
passthru2="2/0/2=8:2"
passthru3="2/0/3=8:3"

[mario@marietto /mnt/da3p2/bhyve/img/Windows]==> vm passthru             

DEVICE     BHYVE ID     READY        DESCRIPTION

ppt0       2/0/0        Yes          TU102 [GeForce RTX 2080 Ti]
ppt1       2/0/1        Yes          TU102 High Definition Audio Controller
ppt2       2/0/2        Yes          TU102 USB 3.1 Host Controller
ppt3       2/0/3        Yes          TU102 USB Type-C UCSI Controller

this is my Windows10.conf file :

nano /mnt/da3p2/bhyve/img/Windows/Windows10/Windows10.conf

loader="uefi"

# put up to 8 disks on a single ahci controller.
# without this, adding a disk pushes the following network devices onto higher slot numbers,
# which causes windows to see them as a new interface
ahci_device_limit="8"

# windows expects the host to expose localtime by default, not UTC
utctime="no"

cpu="4"
cpu_sockets="1"
cpu_cores="2"
cpu_threads="2"
memory="4G"

# ideally this should be changed to virtio-net and drivers installed in the guest
# e1000 works out-of-the-box
network0_type="e1000"
network0_switch="public"

disk0_type="ahci-hd"
disk0_name="Windows10-ltsc2.img"
disk1_dev="custom"
disk1_opts="sectorsize=512"
disk1_type="ahci-hd"
disk1_name="/dev/da5"

graphics="yes"
graphics_listen="127.0.0.1"
graphics_res="1280x720"
graphics_wait="yes"
xhci_mouse="yes"

passthru0="2/0/0=8:0"
passthru1="2/0/1=8:1"
passthru2="2/0/2=8:2"
passthru3="2/0/3=8:3"
uuid="46ee2503-a0c7-418b-95c3-0a1a043e21ce"
network0_mac="58:9c:fc:09:af:c7"

This is what happens :

[mario@marietto /mnt/da3p2/bhyve/img/Windows]==> vm start Windows10
Starting Windows10
  * found guest in /mnt/da3p2/bhyve/img/Windows/Windows10
  * booting...

[root@marietto /mnt/da3p2/bhyve/img/Windows]==> vm list
NAME       DATASTORE  LOADER  CPU  MEMORY  VNC  AUTO  STATE
Windows10  default    uefi    4    4G      -    No    Stopped

it does not boot. But when I remove these args :

passthru0="2/0/0=8:0"
passthru1="2/0/1=8:1"
passthru2="2/0/2=8:2"
passthru3="2/0/3=8:3"

it boot regularly. So,there is something wrong with the passthru arguments. But I can pass thru the GPU if I don't use vm-bhyve,so the problem is with vm-bhyve. I mean,if I use the raw bhyve parameters,like these :

bhyve -S -c sockets=2,cores=2,threads=2 -m 8G -w -H -A \
-s 0,hostbridge \
-s 1,ahci-hd,/mnt/$vmdisk0'p2'/bhyve/img/Windows/Windows10-ltsc.img \
-s 4,ahci-hd,/dev/$vmdisk5 \
-s 8:0,passthru,2/0/0 \
-s 8:1,passthru,2/0/1 \
-s 8:2,passthru,2/0/2 \
-s 8:3,passthru,2/0/3 \
-s 13,virtio-net,tap17 \
-s 29,fbuf,tcp=0.0.0.0:5917,w=1600,h=950,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
vm0:17 < /dev/null & sleep 2 && vncviewer 0:17

I see the gpu on Windows 10 guest.

[GogoFC]

At some point I watched this video, he explained how he does it, same as you without vm-bhyve, just bhyve. He also says Nvidia isn't supported. https://youtu.be/eurBCPj65oI?si=9DXc26rsmyA8LCj8&t=441

[Marietto2008]

Nvidia gpu passthru is partial supported on FreeBSD 14,using Linux and Windows. I think that until it is supported for everyone and in any condition,he wants to use some caution. I think that you should be lucky at this point. So,try and see. Regarding me,I can passthru my nvidia gpu (RTX 2080 ti) on a Linux and Windows vm without getting errors,BUT I can't use the vm-bhyve,otherwise,as you can see on this bug ticket,it will not work.