search

pgj / freebsd-wifibox

wifibox: Use Linux to drive your wireless card on FreeBSD
BSD 2-Clause "Simplified" License
159 stars 12 forks source link

Caveats for FreeBSD 14.0-CURRENT #11

Closed realSaltyFish closed 2 years ago

realSaltyFish commented 2 years ago

There are two major problems with wifibox on FreeBSD 14.0-CURRENT. I spent several hours debugging and resolved them both. Please fix these to help others avoid issues.

wifibox service is enabled, but it is not starting at boot

Before upgrading to 14.0, wifibox worked perfectly, but after the upgrade it won't start automatically and I have to start it manually. Even on a fresh 14.0 install this issue exists. I read the boot log and found this error message:

ld-elf.so.1: Shared object "lib9p.so.1.1" not found, required by "bhyve"
ERROR: The Virtio 9p (VirtFS) bhyve interface is not available.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/alsa-lib /usr/local/lib/compat/pkg /usr/local/lib/e2fsprogs /usr/local/lib/gcc10 /usr/local/lib/perl5/5.32/mach/CORE /usr/local/lib/qt5 /usr/local/lib/samba4 /usr/local/llvm13/lib
32-bit compatibility ldconfig path: /usr/lib32
... (irrelevant logs)

It appears that we need to have the ldconfig service finish before launching bhyve. Adding it to the rc script fixes the issue. After modifying the rc script:

ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/alsa-lib /usr/local/lib/compat/pkg /usr/local/lib/e2fsprogs /usr/local/lib/gcc10 /usr/local/lib/perl5/5.32/mach/CORE /usr/local/lib/qt5 /usr/local/lib/samba4 /usr/local/llvm13/lib
32-bit compatibility ldconfig path: /usr/lib32
...(irrelevant logs)
ppt0: detached
pci3: <network> at device 0.0 (no driver attached)
ppt0 mem 0xb4300000-0xb4301fff at device 0.0 on pci3
Launching wifibox

~Kernel panic when resuming from suspension~

UPDATE. Ignore this and do not try my setup in 14.0-CURRENT. It does not work, and will hang the system on resume. I used this complicated setup because I ran into the same issue as #2. Basically I had to use wifibox resume, followed by wifibox restart, then restart netif and routing services (since I was using static IP). I did some "clever" tricks to avoid doing all these work on resume. This worked on 13.0-RELEASE-p6, but will not work on 14.0!

~Comment. I am not using devd for resume. I am using rc.suspend to stop wifibox and unload vmm.ko, and rc.resume to start wifibox. This saves me a lot of time when waking my laptop up. However, I did test the devd approach and the problem is the same. (Please consider adding this to the man page, thanks.)~

~When I resume my laptop, the system works at first, but hangs after a few seconds. I found that the problem arises when the vmm kernel module is unloaded. I did a ton of experiments and found that vmm will cause a KP when unloaded if there are active ppt devices. The solution is to use devctl detach to detach the ppt0 device before unloading vmm. Please consider adding this to the resume command. This one really got me a lot of headache...~

bhyve+ on 14.0 is not needed

During my debugging process I tried (force) uninstalling bhyve+ and switching to the system bhyve instead. It works perfectly. I guess the whole point of bhyve+ is to port the 14.0 bhyve to 13 and 12, so it isn't really necessary for 14. Maybe it is possible to make a separate package for 14 that does not require bhyve+ as dependency?

pgj commented 2 years ago

The use of bhyve+ is not mandatory on FreeBSD 13 and later as it is mentioned in the README, and by default, it uses the BHyVe implementation provided by the base system. It is the freebsd-wifibox-port where the packaging is implemented, therefore I would move such requests there.

A similar problem has been reported as https://github.com/pgj/freebsd-bhyve-plus-port/issues/1, which indicates that the way the kernel module is built by sysctl/bhyve+ is not fully reliable. I am a bit short on time to track down this problem, but I can make the bhyve+ dependency optional for the net/wifibox port on the newer systems. The reason why I wanted to force it to be there is to provide a stable foundation for the application to run, but it has turned out to be rather unstable in the meantime.

pgj commented 2 years ago

I have changed the port to make the dependency on bhyve+ optional. Are you fine with this resolution, together my comment above, may I close this ticket?

realSaltyFish commented 2 years ago

After switching to system bhyve the boot time autostart issue is also gone. Looks all good for now.