wifibox on a Dell XPS with Atheros not starting

[orbitz]

Hello,

I saw wifibox in the latest FreeBSD Quarterly Report and and excited to use it!

I'm on:

FreeBSD bsdell 14.0-CURRENT FreeBSD 14.0-CURRENT #67 main-n255572-bf46c0a9ae4e: Tue May 10 21:23:49 EDT 2022 root@bsdell:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64

With:

none3@pci0:2:0:0:       class=0x028000 rev=0x32 hdr=0x00 vendor=0x168c device=0x003e subvendor=0x1a56 subdevice=0x143a
    vendor     = 'Qualcomm Atheros'
    device     = 'QCA6174 802.11ac Wireless Network Adapter'
    class      = network

And I installed wifibox-alpine-ath10k-20220430 (I'm pretty sure it's an ath10k)

I configured bhyve.confg with:

passthru=2/0/0

I then did sudo service wifibox onestart.

I see the following every few seconds:

2022-06-11T18:23:56-0400 WARN The PPT device could not be destroyed

I'm unable to ping anything (not sure if I should be able to ping anything).

I looked in /var/run/wifibox/appliance/log and there are no logs. I also did ps auxww | grep bhy and there are not running bhyve instances.

In /var/log/messages I see:

Jun 11 18:23:41 bsdell kernel: ppt0 mem 0xdc000000-0xdc1fffff irq 18 at device 0.0 on pci2
Jun 11 18:23:42 bsdell kernel: ppt0: detached
Jun 11 18:23:42 bsdell kernel: pci2: <network> at device 0.0 (no driver attached)
Jun 11 18:23:48 bsdell kernel: ppt0 mem 0xdc000000-0xdc1fffff irq 18 at device 0.0 on pci2
Jun 11 18:23:49 bsdell kernel: ppt0: detached
Jun 11 18:23:49 bsdell kernel: pci2: <network> at device 0.0 (no driver attached)
Jun 11 18:23:55 bsdell kernel: ppt0 mem 0xdc000000-0xdc1fffff irq 18 at device 0.0 on pci2
Jun 11 18:23:56 bsdell kernel: ppt0: detached
Jun 11 18:23:56 bsdell kernel: pci2: <network> at device 0.0 (no driver attached)

One thing to note is that I currently use a USB dongle for my wifi, and I started the machine with the dongle installed and then unplugged it before starting wifibox.

Any thoughts on what's going on? Can I acquire any more details for you or try anything?

Thank you.

[thesunexpress]

Which model XPS do you have? I own a 9360 (Intel NIC, swapped to Atheros) & 9380 (Atheros NIC) and wifibox works beautifully with both. Are your ports & binaries sync'ed to compatible versions?

Dell XPS 9380 --- WiFi NIC = QCA6174, and printed onto the actual device is "Killer 1435-S" & "QCNFA324"

[pgj]

Hi @orbitz and thanks for giving this project a try! Could you please set logging to debug level in $ETCDIR/wifibox/core.conf and show us what is logged in /var/log/wifibox.log ? I do not think that the USB dongle would cause any problem, but it is indeed better to be removed first.

[orbitz]

Here is the output with debug:

2022-06-12T11:31:07-0400 INFO  VM manager launched
2022-06-12T11:31:07-0400 INFO  Gathering necessary configuration files for launching the guest
2022-06-12T11:31:07-0400 INFO  Pulling bhyve options from guest-specific configuration file
2022-06-12T11:31:07-0400 INFO  Pulling bhyve options from configuration file
2022-06-12T11:31:07-0400 DEBUG cpus=1, memory=128M, passthru=2/0/0, console=no
2022-06-12T11:31:07-0400 INFO  Guest console is not configured to use
2022-06-12T11:31:07-0400 INFO  Passthru device is configured: 2/0/0
2022-06-12T11:31:07-0400 INFO  PPT driver is configured for pci2:0:0 device
2022-06-12T11:31:07-0400 INFO  tap interface is configured: tap2
2022-06-12T11:31:07-0400 INFO  Pulling WPA Supplicant control options from configuration file
2022-06-12T11:31:07-0400 DEBUG wpa_ctrl.enabled=yes
2022-06-12T11:31:07-0400 INFO  wpa_supplicant.conf will be mounted writable
2022-06-12T11:31:07-0400 INFO  Launching guest wifibox from /usr/local/share/wifibox with grub-bhyve
2022-06-12T11:31:07-0400 DEBUG Arguments:  -S -M 128M -r host  -d /usr/local/share/wifibox wifibox
2022-06-12T11:31:07-0400 DEBUG [grub-bhyve] Error opening terminal: unknown.
2022-06-12T11:31:07-0400 INFO  Destroying guest wifibox
2022-06-12T11:31:08-0400 INFO  Destroying bhyve PPT device: pci2:0:0
2022-06-12T11:31:08-0400 DEBUG [devctl] devctl: Failed to clear pci2:0:0 driver: Device not configured
2022-06-12T11:31:08-0400 WARN  The PPT device could not be destroyed
2022-06-12T11:31:08-0400 DEBUG exit_code=1
2022-06-12T11:31:08-0400 INFO  grub-bhyve failed to start, signaling restart after 5 seconds
2022-06-12T11:31:14-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: stop

[orbitz]

I'm not sure if this matters but I have not enabled wifibox to start on boot. I've been testing it where I manually start it. Is that OK? The FreeBSD wiki page implies that for passthru to work maybe it needs to be a configuration at boot?

https://wiki.freebsd.org/bhyve/pci_passthru

[orbitz]

@thesunexpress I have a Dell XPS 9370. I'm not sure exactly how to tell if everything is synced properly. I am on the kernel commit I mentioned, which is a month or so old at this point.

[pgj]

Thanks! It seems that grub-bhyve could not start up for some reason, that is why there is no bhyve process as well. Let me try to understand what is going on there, I have not seen such errors before.

[pgj]

Looks like this has something to do with the TERM environment variable. What is the value of that in your case? That is, could you please post the output for the echo $TERM command? That is xterm for me, and it is working.

[pgj]

I'm not sure if this matters but I have not enabled wifibox to start on boot. I've been testing it where I manually start it. Is that OK? The FreeBSD wiki page implies that for passthru to work maybe it needs to be a configuration at boot?

Ideally, it does not require you to configure the start on boot. Everything should be handled automatically unless stated otherwise in the manual page. For example, AMD-Vi has to be configured before trying to run Wifibox but that is documented.

[orbitz]

My $TERM is screen

[pgj]

Okay, let me summarize the situation here. You launch wifibox via screen through sudo locally. Is there any non-standard configuration setting present on your system?

[orbitz]

Technically I'm using tmux, and my terminal emulator is st as well. I don't know if I have any non-standard configuration. Anything specific I should look for?

[pgj]

Based on the grub-bhyve error message, this is an ncurses initialization problem. Thus I believe that anything that has any impact on the terminal definitions (termcap, terminfo, etc.) or the version of ncurses used might cause this. Which ncurses version do you use? Is it from the base or from the Ports Collection? It would be good to see which libncurses.so the grub-bhyve binary is linked to. For example, the following command could tell this:

$ ldd $(whereis grub-bhyve | cut -d ':' -f2) | fgrep ncurses
        libncurses.so.8 => /lib/libncurses.so.8 (0x800742000)

I am currently trying to reproduce your problem but without success so far.

[pgj]

I was able to reproduce the problem to some degree by moving away all the termcap files. I am inclined to think that the same could also happen if the termcap database is corrupted. I booted a 14.0-CURRENT-20220610-e50e40684aa-256063 snapshot and Wifibox just worked as expected. Based on that, it should not be specific to 14-CURRENT, however, the specific version you showed in the description might have problems with termcap.

[orbitz]

So my kernel was out of alignment with my pkgs database, so I fixed that, but it dosen't appear to have resolved the issue. I can confirm that the ncurses lib that grub-bhyve uses (libncursesw.so.9) exists. I can also run grub-bhyve by hand (I get a help screen) so at least that suggests to me that it can find its libraries. I'm now on:

FreeBSD bsdell 14.0-CURRENT FreeBSD 14.0-CURRENT #68 main-n256081-987d26ee3f7a:

I don't know how to verify the termcap situation. I'm compiling my own kernel, installing world, etcupdate yadda. Everything else works. I'll play with it some more tomorrow.

Thank you for the quick responses. Let me know if there is anything else you'd like me to do to test.

[orbitz]

Also, I do not know if this matters, but I have my own ubuntu and alpine bhyve VMs (which I have not been running during this due to a vmm conflict), but they run fine for me, I imagine they are using grub-bhyve as well.

[pgj]

I tried to analyze the behavior of grub-bhyve with the help of truss(1). Looks like ncurses is not utilized all the time only if tmux is in use. Have you tried to launch wifibox without using tmux? Also, it would be interesting to see the truss(1) output of the problematic command itself. Maybe only the way how grub-bhyve is used with wifibox causes the trouble.

# truss -o /tmp/grub-bhyve.truss.log /usr/local/sbin/grub-bhyve -S -M 128M -r host -d /usr/local/share/wifibox wifibox
# bhyvectl --vm=wifibox --destroy

When launched in a tmux-hosted terminal, the following could be noticed:

# grep -F termcap /tmp/grub-bhyve.truss.log
open("/root/.termcap.db",O_RDONLY|O_CLOEXEC,00)  ERR#2 'No such file or directory'
open("/root/.termcap",O_RDONLY|O_CLOEXEC,00)     ERR#2 'No such file or directory'
open("/usr/share/misc/termcap.db",O_RDONLY|O_CLOEXEC,00) = 4 (0x4)

[thesunexpress]

@thesunexpress I have a Dell XPS 9370. I'm not sure exactly how to tell if everything is synced properly. I am on the kernel commit I mentioned, which is a month or so old at this point.

Neat, you've got the model right in between my 2 XPS 93xx. Looks like it has the same soldered NIC on the motherboard. The reason I ask is this: Whilst testing between FreeBSD 13-STABLE & 14-CURRENT, along with wifibox versions 0.5 and 0.6, specifically testing functionality, stress & intentionally breaking it, I came across situations where console logs like yours: kernel: ppt0 mem 0xdc000000-0xdc1fffff irq 18 at device 0.0 on pci2 kernel: ppt0: detached kernel: pci2: <network> at device 0.0 (no driver attached) What became apparent is that any time the kernel sources & ports were far enough out of sync, vmm.ko (at least so far as what ends up copied into /boot/modules/) and would break and dump. Console would be spammed with a attach/detach loop that looked very similar to what you've got going on. I'm annoyed with myself that I didn't save those logs for comparison now. Using -STABLE / -CURRENT meant I'd be commonly using the most recent source & ports (pretty much 0-day).

Curious you are having some problems, for me it works out of the box without so much as a peep other than initial bhyve.conf & wpa_supplicant.conf configuration; your NIC is effectively identical to mine.

[orbitz]

@pgj

orbitz@bsdell ~> grep -F termcap /tmp/grub-bhyve.truss.log
fstatat(AT_FDCWD,"/etc/termcap",{ mode=-r--r--r-- ,inode=930947,size=212544,blksize=131072 },0x0) = 0 (0x0)
fstatat(AT_FDCWD,"/usr/share/misc/termcap",{ mode=-r--r--r-- ,inode=930947,size=212544,blksize=131072 },0x0) = 0 (0x0)

I also tried running outside of tmux in two ways:

1. Just in an st window
2. Via dmenu.

No luck.

[pgj]

@orbitz would it be possible to share the complete log file somehow? It would be interesting to see what else grub-byhve tries to do besides accessing the termcap files which eventually leads to the failure. By the way, when running the command by itself, do you get the Error opening terminal: unknown. error message?

[pgj]

@thesunexpress I think this problem is not specific to the actual hardware, it appears to be a software configuration issue or bug. We did not even get to the point to try to run the VM due to the failure of grub-bhyve.

[orbitz]

I switched from using the service to running wifibox by hand. Good news: It started! Bad news, no luck on connecting to wifi and after a bunch of starts and stops, my system became unstable and stopped responding and I had to reboot.

I looked at the dmesg produced and see:

[    0.988180] ath10k_pci 0000:00:06.0: can't derive routing for PCI INT A
[    0.988183] ath10k_pci 0000:00:06.0: PCI INT A: no GSI
[    0.990875] ath10k_pci 0000:00:06.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0
[    1.235293] ath10k_pci 0000:00:06.0: qca6174 hw3.2 target 0x05030000 chip_id 0x00340aff sub 1a56:143a
[    1.235297] ath10k_pci 0000:00:06.0: kconfig debug 0 debugfs 0 tracing 0 dfs 0 testmode 0
[    1.235641] ath10k_pci 0000:00:06.0: firmware ver WLAN.RM.4.4.1-00157-QCARMSWPZ-1 api 6 features wowlan,ignore-otp,mfp crc32 90eebefb
[    1.309619] ath10k_pci 0000:00:06.0: board_file api 2 bmi_id N/A crc32 706c395e

Also looking at the messages:

Jun 14 00:30:09 wifibox syslog.info syslogd started: BusyBox v1.34.1
Jun 14 00:30:09 wifibox daemon.err /etc/init.d/wpa_supplicant[760]: Could not find a wireless interface
Jun 14 00:30:09 wifibox daemon.info supervise-daemon[763]: Supervisor command line: supervise-daemon wpa_supplicant --start /sbin/wpa_supplicant -- -c/media/wpa/wpa_supplicant.conf
Jun 14 00:30:09 wifibox daemon.info supervise-daemon[772]: Child command line: /sbin/wpa_supplicant -c/media/wpa/wpa_supplicant.conf
Jun 14 00:30:09 wifibox daemon.warn supervise-daemon[770]: /sbin/wpa_supplicant, pid 772, exited with return code 255
Jun 14 00:30:09 wifibox daemon.info supervise-daemon[811]: Child command line: /sbin/wpa_supplicant -c/media/wpa/wpa_supplicant.conf
Jun 14 00:30:09 wifibox daemon.warn supervise-daemon[770]: /sbin/wpa_supplicant, pid 811, exited with return code 255
Jun 14 00:30:09 wifibox daemon.info supervise-daemon[813]: Child command line: /sbin/wpa_supplicant -c/media/wpa/wpa_supplicant.conf
Jun 14 00:30:09 wifibox daemon.warn supervise-daemon[770]: /sbin/wpa_supplicant, pid 813, exited with return code 255
Jun 14 00:30:09 wifibox daemon.info supervise-daemon[816]: Child command line: /sbin/wpa_supplicant -c/media/wpa/wpa_supplicant.conf
Jun 14 00:30:09 wifibox daemon.warn supervise-daemon[770]: /sbin/wpa_supplicant, pid 816, exited with return code 255
Jun 14 00:30:09 wifibox daemon.info supervise-daemon[818]: Child command line: /sbin/wpa_supplicant -c/media/wpa/wpa_supplicant.conf
Jun 14 00:30:09 wifibox daemon.warn supervise-daemon[770]: /sbin/wpa_supplicant, pid 818, exited with return code 255
Jun 14 00:30:09 wifibox daemon.info supervise-daemon[820]: Child command line: /sbin/wpa_supplicant -c/media/wpa/wpa_supplicant.conf
Jun 14 00:30:09 wifibox daemon.warn supervise-daemon[770]: /sbin/wpa_supplicant, pid 820, exited with return code 255
Jun 14 00:30:09 wifibox daemon.info supervise-daemon[822]: Child command line: /sbin/wpa_supplicant -c/media/wpa/wpa_supplicant.conf
Jun 14 00:30:09 wifibox daemon.warn supervise-daemon[770]: /sbin/wpa_supplicant, pid 822, exited with return code 255
Jun 14 00:30:09 wifibox daemon.info supervise-daemon[825]: Child command line: /sbin/wpa_supplicant -c/media/wpa/wpa_supplicant.conf
Jun 14 00:30:09 wifibox daemon.warn supervise-daemon[770]: /sbin/wpa_supplicant, pid 825, exited with return code 255
Jun 14 00:30:09 wifibox daemon.info supervise-daemon[828]: Child command line: /sbin/wpa_supplicant -c/media/wpa/wpa_supplicant.conf
Jun 14 00:30:09 wifibox daemon.warn supervise-daemon[770]: /sbin/wpa_supplicant, pid 828, exited with return code 255
Jun 14 00:30:09 wifibox daemon.info supervise-daemon[830]: Child command line: /sbin/wpa_supplicant -c/media/wpa/wpa_supplicant.conf
Jun 14 00:30:09 wifibox daemon.warn supervise-daemon[770]: /sbin/wpa_supplicant, pid 830, exited with return code 255
Jun 14 00:30:09 wifibox daemon.info supervise-daemon[832]: Child command line: /sbin/wpa_supplicant -c/media/wpa/wpa_supplicant.conf
Jun 14 00:30:09 wifibox daemon.warn supervise-daemon[770]: /sbin/wpa_supplicant, pid 832, exited with return code 255
Jun 14 00:30:09 wifibox daemon.warn supervise-daemon[770]: respawned "/sbin/wpa_supplicant" too many times, exiting
Jun 14 00:30:24 wifibox daemon.err /etc/init.d/wpa_passthru[922]: ERROR: cannot start wpa_passthru as wpa_supplicant would not start
Jun 14 00:30:24 wifibox daemon.info : starting pid 923, tty '': '/sbin/openrc default'
Jun 14 00:30:24 wifibox daemon.info supervise-daemon[943]: Supervisor command line: supervise-daemon wpa_supplicant --start /sbin/wpa_supplicant -- -iwlan0 -c/media/wpa/wpa_supplicant.conf
Jun 14 00:30:24 wifibox daemon.info supervise-daemon[947]: Child command line: /sbin/wpa_supplicant -iwlan0 -c/media/wpa/wpa_supplicant.conf
Jun 14 00:30:24 wifibox daemon.info supervise-daemon[967]: Supervisor command line: supervise-daemon wpa_passthru --start /sbin/wpa_passthru --
Jun 14 00:30:24 wifibox daemon.info supervise-daemon[971]: Child command line: /sbin/wpa_passthru
Jun 14 00:30:24 wifibox daemon.debug wpa_passthru[971]: Configuration: enabled=yes, network=10.0.0.1:255.255.255.0, wlan devs=[p2p_dev_wlan0 wlan0]
Jun 14 00:30:24 wifibox daemon.warn wpa_passthru[971]: /var/run/wpa_supplicant/p2p-dev-wlan0 is not available, skipping.
Jun 14 00:30:24 wifibox daemon.warn wpa_passthru[971]: /var/run/wpa_supplicant/wlan0 is not available, skipping.
Jun 14 00:30:24 wifibox daemon.info wpa_passthru[971]: Waiting for PID 999 to stop.

So clearly there is a problem there, not sure what. (does the /usr/local/etc/wifibox/wpa_supplicant/wpa_supplicant.conf file need any specific privs?)

[orbitz]

These logs are after I restarted and ran start again. I still have my wifi dongle plugged in so could be conflicting there

[orbitz]

I rebooted without the dongle and started wifibox. Still no luck, but here are latest:

messages from VM:

Jun 14 00:39:21 wifibox syslog.info syslogd started: BusyBox v1.34.1
Jun 14 00:39:21 wifibox daemon.info supervise-daemon[768]: Supervisor command line: supervise-daemon wpa_supplicant --start /sbin/wpa_supplicant -- -iwlan0 -c/media/wpa/wpa_supplicant.conf
Jun 14 00:39:21 wifibox daemon.info supervise-daemon[772]: Child command line: /sbin/wpa_supplicant -iwlan0 -c/media/wpa/wpa_supplicant.conf
Jun 14 00:39:36 wifibox daemon.info supervise-daemon[930]: Supervisor command line: supervise-daemon wpa_passthru --start /sbin/wpa_passthru --
Jun 14 00:39:36 wifibox daemon.info supervise-daemon[934]: Child command line: /sbin/wpa_passthru
Jun 14 00:39:36 wifibox daemon.info : starting pid 937, tty '': '/sbin/openrc default'
Jun 14 00:39:36 wifibox daemon.debug wpa_passthru[934]: Configuration: enabled=yes, network=10.0.0.1:255.255.255.0, wlan devs=[p2p_dev_wlan0 wlan0]
Jun 14 00:39:36 wifibox daemon.info wpa_passthru[934]: Associating 10.0.0.1:1201 (10.0.0.1:255.255.255.0) with /var/run/wpa_supplicant/p2p-dev-wlan0 (/tmp/tmp.CFLeGc/p2p-dev-wlan0)
Jun 14 00:39:36 wifibox daemon.info wpa_passthru[934]: Associating 10.0.0.1:1200 (10.0.0.1:255.255.255.0) with /var/run/wpa_supplicant/wlan0 (/tmp/tmp.CFLeGc/wlan0)
Jun 14 00:39:36 wifibox daemon.info wpa_passthru[934]: Waiting for PID 961 to stop.
Jun 14 00:39:36 wifibox cron.info crond[1002]: crond (busybox 1.34.1) started, log level 8
Jun 14 00:39:36 wifibox daemon.info : starting pid 1062, tty '/dev/ttyS0': '/sbin/getty -L ttyS0 115200 vt100'

wifibox.log

2022-06-13T20:39:19-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: start
2022-06-13T20:39:19-0400 INFO  Begin: wifibox start
2022-06-13T20:39:19-0400 DEBUG assert loaded: kmod=vmm, kmod_file=vmm.ko
2022-06-13T20:39:19-0400 DEBUG Kernel module vmm is not loaded
2022-06-13T20:39:19-0400 DEBUG Kernel module vmm was loaded successfully
2022-06-13T20:39:19-0400 DEBUG assert hardware support present: iommu=1, amdvi=0
2022-06-13T20:39:19-0400 DEBUG Backends reported by bhyve:
2022-06-13T20:39:19-0400 DEBUG [bhyve] ahci
2022-06-13T20:39:19-0400 DEBUG [bhyve] ahci-hd
2022-06-13T20:39:19-0400 DEBUG [bhyve] ahci-cd
2022-06-13T20:39:19-0400 DEBUG [bhyve] e1000
2022-06-13T20:39:19-0400 DEBUG [bhyve] dummy
2022-06-13T20:39:19-0400 DEBUG [bhyve] hda
2022-06-13T20:39:19-0400 DEBUG [bhyve] fbuf
2022-06-13T20:39:19-0400 DEBUG [bhyve] amd_hostbridge
2022-06-13T20:39:19-0400 DEBUG [bhyve] hostbridge
2022-06-13T20:39:19-0400 DEBUG [bhyve] lpc
2022-06-13T20:39:19-0400 DEBUG [bhyve] nvme
2022-06-13T20:39:19-0400 DEBUG [bhyve] passthru
2022-06-13T20:39:19-0400 DEBUG [bhyve] virtio-9p
2022-06-13T20:39:19-0400 DEBUG [bhyve] virtio-blk
2022-06-13T20:39:19-0400 DEBUG [bhyve] virtio-console
2022-06-13T20:39:19-0400 DEBUG [bhyve] virtio-input
2022-06-13T20:39:19-0400 DEBUG [bhyve] virtio-net
2022-06-13T20:39:19-0400 DEBUG [bhyve] virtio-rnd
2022-06-13T20:39:19-0400 DEBUG [bhyve] virtio-scsi
2022-06-13T20:39:19-0400 DEBUG [bhyve] uart
2022-06-13T20:39:19-0400 DEBUG [bhyve] xhci
2022-06-13T20:39:19-0400 INFO  Creating bridge interface: wifibox0
2022-06-13T20:39:19-0400 DEBUG [ifconfig] wifibox0
2022-06-13T20:39:19-0400 INFO  Linking tap interface to wifibox0: tap2
2022-06-13T20:39:19-0400 INFO  Waiting for bhyve to start up
2022-06-13T20:39:19-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: _manage_vm
2022-06-13T20:39:19-0400 DEBUG assert daemonized: parent=daemon
2022-06-13T20:39:19-0400 INFO  VM manager launched
2022-06-13T20:39:19-0400 INFO  Gathering necessary configuration files for launching the guest
2022-06-13T20:39:19-0400 INFO  Pulling bhyve options from guest-specific configuration file
2022-06-13T20:39:19-0400 INFO  Pulling bhyve options from configuration file
2022-06-13T20:39:19-0400 DEBUG cpus=1, memory=128M, passthru=2/0/0, console=no
2022-06-13T20:39:19-0400 INFO  Guest console is not configured to use
2022-06-13T20:39:19-0400 INFO  Passthru device is configured: 2/0/0
2022-06-13T20:39:19-0400 INFO  PPT driver is configured for pci2:0:0 device
2022-06-13T20:39:20-0400 INFO  tap interface is configured: tap2
2022-06-13T20:39:20-0400 INFO  Pulling WPA Supplicant control options from configuration file
2022-06-13T20:39:20-0400 DEBUG wpa_ctrl.enabled=yes
2022-06-13T20:39:20-0400 INFO  wpa_supplicant.conf will be mounted writable
2022-06-13T20:39:20-0400 INFO  Launching guest wifibox from /usr/local/share/wifibox with grub-bhyve
2022-06-13T20:39:20-0400 DEBUG Arguments:  -S -M 128M -r host  -d /usr/local/share/wifibox wifibox
2022-06-13T20:39:20-0400 INFO  Launching guest wifibox from /usr/local/share/wifibox with bhyve
2022-06-13T20:39:20-0400 DEBUG Arguments:  -c 1 -m 128M -AHP -u -S  -s 0,hostbridge -s 31,lpc -s 4:0,virtio-blk,/usr/local/share/wifibox/disk.img -s 4:1,virtio-9p,config=/usr/local/etc/wifibox,ro -s 4:2,virtio-9p,wpa_config=/usr/local/etc/wifibox/wpa_supplicant -s 4:3,virtio-9p,var=/var/run/wifibox/appliance -s 5:0,e1000,tap2 -s 6:0,passthru,2/0/0 wifibox
2022-06-13T20:39:20-0400 DEBUG [bhyve] rdmsr to register 0x140 on vcpu 0
2022-06-13T20:39:20-0400 DEBUG [bhyve] rdmsr to register 0x64d on vcpu 0
2022-06-13T20:39:20-0400 DEBUG [bhyve] rdmsr to register 0x64e on vcpu 0
2022-06-13T20:39:20-0400 DEBUG [bhyve] rdmsr to register 0x34 on vcpu 0
2022-06-13T20:39:20-0400 INFO  Guest wifibox has started up
2022-06-13T20:39:20-0400 INFO  Guest is managed by PID 2615
2022-06-13T20:39:20-0400 INFO  Bringing up WPA pass-through
2022-06-13T20:39:20-0400 INFO  Control interface: /var/run/wpa_supplicant
2022-06-13T20:39:20-0400 INFO  Found guest IP address: 10.0.0.1
2022-06-13T20:39:21-0400 INFO  Control interface group: 0
2022-06-13T20:39:21-0400 INFO  WLAN devices: [p2p_dev_wlan0
wlan0]
2022-06-13T20:39:21-0400 INFO  Hooking up 10.0.0.1:1201 as /var/run/wpa_supplicant/p2p-dev-wlan0
2022-06-13T20:39:21-0400 INFO  Hooking up 10.0.0.1:1200 as /var/run/wpa_supplicant/wlan0
2022-06-13T20:39:21-0400 INFO  End: wifibox start

When running, I can see that the /var/run/wpa_supplicant files exist, however running wpa_cli status times out

orbitz@bsdell ~> wpa_cli status
Selected interface 'wlan0'
'STATUS' command timed out.
orbitz@bsdell ~> sudo wpa_cli status
Selected interface 'wlan0'
'STATUS' command timed out.

[pgj]

I think first we should make sure that the wireless device is created on the VM. To check this, I would recommend you to enable the console in bhyve.conf and try the wifibox console command (you can login as root there with a blank password). Then post the output of the lsmod and ifconfig commands to see what is going on inside. You can even try to launch wpa_supplicant in the foreground.

[pgj]

Is the wifibox0 interface configured properly on the host? Can you ping the guest on 10.0.0.1?

[orbitz]

https://paste.sr.ht/~mmatalka/5048412c2bd093c3d7b0abccddbe50950b7b6929

Here is the information you requested.

I cannot ping 10.0.0.1.

[pgj]

Thanks for the logs. You will have to assign an IP address for the wifibox0 interface and ensure that it can be routed. This could either be done via DHCP or statically, that is up to your local configuration. For example, dhclient wifibox0 could be used to try DHCP.

[orbitz]

Is that the case even if I can't ping 10.0.0.1?

[pgj]

Yes, I believe so. The guest has the "virtual" 10.0.0.1 IP address which is not visible on the host. On the host there shall be an IP address on the same network (10.0.0.0/24) to communicate with the guest. dhclient can get an IP address through the wifibox0 interface from udhcpd running on the guest, as well as configure the default gateway to point to 10.0.0.1.

[orbitz]

IT'S WOOOORKIIIIIIIIIIIIIIIIING! Wooo, thank you!

I'm still not sure why I need to run wifibox rather than via the service.

[orbitz]

Question, what is the relationship between my existing wpa_supplicant.conf file and the one for wifibox. Is it fine to just copy my host wpa_supplication.conf file to the /usr/local/etc/wifibox configuration?

[orbitz]

Another question: I use bhyve to run VMs for work and I have the following pf.conf:

nat on wlan0 from 127.0.0.0/8 to any -> (wlan0)
nat on wlan0 from 10.10.123.0/24 to !10.10.123.0/24 -> (wlan0)

Should there be any issue with running wifibox and my other linux vm's at the same time? I know wifibox doesn't like it if I start my linux VMs first because of the vmm kernel module. And I suppose I can just turn wlan0 into wifibox0?

Thank you

[pgj]

Question, what is the relationship between my existing wpa_supplicant.conf file and the one for wifibox. Is it fine to just copy my host wpa_supplicant.conf file to the /usr/local/etc/wifibox configuration?

Yes, it should be just fine to use the same file -- it will be actually processed by the same program but on a different platform.

[pgj]

Should there be any issue with running wifibox and my other linux vm's at the same time? I know wifibox doesn't like it if I start my linux VMs first because of the vmm kernel module.

This should work, I also have other bhyve-based VMs besides wifibox locally. What Wifibox does not like is when it needs to revive the PCI pass-through after the ACPI resume event. That is when vmm(4) has to be unloaded and then re-loaded to restore the functionality. When other guests use vmm(4) at the same time the module cannot and will not be unloaded. But Wifibox can safely co-exist with others.

I suppose I can just turn wlan0 into wifibox0?

Yes, that is right.

[orbitz]

I still need to confirm, but I think it's turning the console on that allows running via service.

[pgj]

@orbitz I have seen that you closed the issue: did you eventually solve the problem with grub-bhyve? It would be still nice to understand what could have caused it but it may show up at other users.

[orbitz]

@pgj I closed the issue for the main problem: getting it working. I'll do some testing with turning console back off and see if it has trouble starting again. I'm not sure why the console setting change that behaviour, though. Note: this happened even during bootup, now that I have wifibox enabled on startup.

Will get back to you with what happens when I flip console back off.

[pgj]

By the way, just to clarify (I am still looking for ways to reproduce the problem): how do you turn on and off the console? Is this about the settings in /etc/ttys or is it something else?

[orbitz]

The option in bhyve.conf. Again, I still need to confirm, but so far it seems like if console is no then I hit this issue, and if it's yes, I don't.

[pgj]

Ah, now it makes sense (to some extent). On your system, grub-bhyve does not have a tty allocated when Wifibox is launched as a service and it fails. When the Wifibox console is turned on, a null-modem device is assigned as the tty so ncurses can work happily.

[orbitz]

Why would that matter when I try running with service outside of tmux, or even on system boot?

[pgj]

I do not know yet. That is something to explore.