Closed venoom27 closed 1 year ago
pgj
commented
1 year ago Thanks for the report. Just to double check: How did you configure the suspend to RAM for the FreeBSD host? It may happen that you will have to stop wifibox completely before suspending and restart it after resume. The net/wifibox-core port lets you configure the way how suspend/resume situations should be handled, it might be worth trying.
venoom27
commented
1 year ago I was able to suspend and wake back up by putting the following lines in the /usr/local/etc/devd/wifibox.conf . I got this from https://xyinn.org/md/freebsd/wifibox which is looks like he got from you.
notify 11 {
match "system" "ACPI";
match "subsystem" "Suspend";
action "logger 'Stopping wifibox before suspend' && /usr/local/sbin/wifibox stop && /etc/rc.suspend acpi $notify";
};
notify 11 {
match "system" "ACPI";
match "subsystem" "Resume";
action "/etc/rc.resume acpi $notify && logger 'Starting wifibox after resume and getting IP via DHCP' && /usr/local/sbin/wifibox start && /sbin/dhclient wifibox0";
};Then I put the system to sleep by issuing acpiconf -s 3
Its takes longer to wake up which I am assuming is the wifibox starting back up. But once it back up the wlan0 within wifibox will not ping google. I can ping the router but when I ping google I get ping: bad address 'www.google.com'
Here is the ifconfig within wifibox console.
wlan0 Link encap:Ethernet HWaddr A8:BB:CF:22:EA:A6
inet6 addr: fe80::aabb:cfff:fe22:eaa6/64 Scope:Link
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:3533
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
Interrupt:40I also tried
service netif stop
service wifibox restart
service netif startStill will not ping google. I then have to reboot the computer to get the wifibox to ping google again.
pgj
commented
1 year ago Using wifibox stop and then wifibox start destroys the wifibox0 networking interface so it should be reconfigured by service netif restart wifibox0 in order to make it work again (there is no need for a complete reboot). But that is why I implemented wifibox stop guest and wifibox start guest extension where the networking interface is left intact and does not require a reconfiguration upon restart. Please try that version.
venoom27
commented
1 year ago I tried service netif restart after resuming and the wifbox console still comes back with
wifibox:~# ping www.google.com
ping: bad address 'www.google.com'I added
/etc/rc.d/netif restart to /usr/local/etc/devd/wifibox.conf so it looks like this
notify 11 {
match "system" "ACPI";
match "subsystem" "Suspend";
action "logger 'Stopping wifibox before suspend' && /usr/local/sbin/wifibox stop && /etc/rc.suspend acpi $notify";
};
notify 11 {
match "system" "ACPI";
match "subsystem" "Resume";
action "/etc/rc.resume acpi $notify && logger 'Starting wifibox after resume and getting IP via DHCP' && /usr/local/sbin/wifibox start && /etc/rc.d/netif restart && /sbin/dhclient wifibox0";
};
I still get
wifibox:~# ping www.google.com
ping: bad address 'www.google.com'after resuming and waking up.
pgj
commented
1 year ago The service netif restart wifibox0 command has to be issued on the FreeBSD host. The guest does not have a netif service but networking. Independently of that, if the guest cannot communicate with the Internet after resume, a reason could be that you will have to stop the guest, reload (unload and then load again) the vmm kernel module on host and restart the guest. This is what wifibox restart vmm can do for you while leaving the wifibox0 interface intact. (That is what I have also been using on my notebook.)
pgj
commented
1 year ago Note that if you have configured the net/wifibox-core port properly, that is all you need in /usr/local/etc/devd/wifibox.conf, the rest is handed by the wifibox service:
notify 11 {
match "system" "ACPI";
match "subsystem" "Suspend";
action "/etc/rc.suspend acpi $notify && service wifibox suspend";
};I original had
notify 11 {
match "system" "ACPI";
match "subsystem" "Suspend";
action "/etc/rc.suspend acpi $notify && service wifibox suspend";
};in my /usr/local/etc/devd/wifibox.conf when I opened the ticket.
When I make config the net/wifibox-core I have the
RECOVER_SUSPEND_GUEST selected. Should I pick the RECOVER_RESTART_VMM instead?
pgj
commented
1 year ago Yes, please try RECOVER_RESTART_VMM.
venoom27
commented
1 year ago I complied wifibox-core with the RECOVER_RESTART_VMM flag and change the
/usr/local/etc/devd/wifibox.conf to
notify 11 {
match "system" "ACPI";
match "subsystem" "Suspend";
action "/etc/rc.suspend acpi $notify && service wifibox suspend";
};I then suspended the computer with acpiconf -s 3 which suspended it. Then when I wake it the screen freezes on on console text and I have to force shutdown.
pgj
commented
1 year ago Okay, thanks for testing. Based on what you wrote, I believe the problem is that the kernel crashes on reloading the vmm kernel module. We could try to investigate the details (and validate this theory) in the following way.
wifibox service so that wifibox will not be tried for resume: service wifibox stop.wifibox manually: wifibox start && service netif restart wifibox0.wifibox: wifibox stop.vmm kernel module: kldunload vmm.vmm kernel module: kldload vmm.wifibox again manually: wifibox start && service netif restart wifibox0.It is expected that the whole procedure will not go through. Try to identify in which step the computer crashes and collect logs and data about the details of the crash.
pgj
commented
1 year ago If I recall correctly, this does not matter. The procedure tries to model how RECOVER_RESTART_VMM works and the goal is to walk it through step-by-step and troubleshoot.
venoom27
commented
1 year ago service wifibox stop
# service wifibox stop
Stopping wifibox.......OK
# wifibox start && service netif restart wifibox0
Starting wifibox.......OK
Stopping Network: wifibox0.
wifibox0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 58:9c:fc:00:34:21
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 2000000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
Starting Network: wifibox0.
wifibox0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 58:9c:fc:00:34:21
inet 10.0.0.165 netmask 0xffffff00 broadcast 10.0.0.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 2000000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
#acpiconf -s 3It freezes on resume and looks like the image below.
After hard shutdown
# wifibox stop
Stopping wifibox.......OK
#kldunload vmm
#kldload vmm
# wifibox start && service netif restart wifibox0
Starting wifibox.......OK
Stopping Network: wifibox0.
wifibox0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 58:9c:fc:00:34:21
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 2000000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
Starting Network: wifibox0.
wifibox0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 58:9c:fc:00:34:21
inet 10.0.0.165 netmask 0xffffff00 broadcast 10.0.0.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 4 priority 128 path cost 2000000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
Thanks for providing these details. The part after the freeze and the hard shutdown is not relevant. We would like to avoid the freeze so what is important here is to see what we can do before it happens.
If the machine freezes on resume, it might make sense to stop Wifibox before suspending the machine. Even unload the vmm kernel module. That is, the updated procedure to test is now as follows.
wifibox service so that wifibox will not be tried for resume: service wifibox stop.wifibox manually: wifibox start && service netif restart wifibox0.wifibox: wifibox stop guest.vmm: kldunload vmm && kldload vmm.wifibox: wifibox start guest.Alternatively, step 3 could be adjusted like this:
3/a. Stop wifibox: wifibox stop guest.
3/b. Unload the vmm kernel module: kldunload vmm.
And step 5 would change accordingly:
vmm: kldload vmm.venoom27
commented
1 year ago service wifibox stop
Stopping wifibox .......OKwifibox start && service netif restart wifibox0
Starting wifibox.......OK
Stopping Network: wifibox0
Starting Network wifibox0wifibox stop guest
Stopping wifibox .......OKacpiconf -s 3goes to sleep resume it wakes back up
kldunload vmm && kldload vmmkldunload vmm && kldload vmmSo now the only issue I am having after those chain of commands is that wifibox console can ping google but the host can't ping google. It says ping: Unknown host.
pgj
commented
1 year ago It would be useful to see the dmesg output from the guest after the resume to understand why the network connection is dead. It is very likely that the card is not accessible for bhyve any more but it would be good to have a confirmation about that.
How does it change if you run kldunload vmm before suspending the computer?
By the way, did you run the combination of the kldunload and kldload commands twice?
venoom27
commented
1 year ago It seems if I run wifibox stop then kldunload before acpiconf -s 3 and then once it is resumed running kldload vmm then wifibox start it brings it back correctly.
pgj
commented
1 year ago All right. In the dev branch of the freebsd-wifibox-ports repository, I have updated the net/wifibox-core port to support this configuration: please try it by selecting the RECOVER_SUSPEND_VMM option.
venoom27
commented
1 year ago I downloaded the new update to the dev branch and selected the RECOVER_SUSPEND_VMM flag in wifibox-core and installed it. When I suspend the computer by running acpiconf -s 3 it suspends but when I resume it it freezes on the console display as before.
pgj
commented
1 year ago Could you please include logs from /var/log/wifibox.log? The service wifibox suspend command may not be called on the suspend event. That is because it needs the wifibox.conf under /usr/local/etc/devd. Unfortunately, the FreeBSD base system only supports the resume event for services.
pgj
commented
1 year ago For testing individually, you can also try to simulate the behavior by calling service wifibox suspend before acpiconf -s 3 and then service wifibox resume after resume.
venoom27
commented
1 year ago Running service wifibox suspend first before running acpiconf -s 3 works. It resumes correctly. I guess I could just write small script that runs those two commands to suspend it. Thanks for your help.
pgj
commented
1 year ago Ideally, you should not have to. The wifibox.conf for devd shall do that for you exactly. Maybe it would be worth to investigate why devd does not call service wifibox suspend? The contents should be like that:
notify 11 {
match "system" "ACPI";
match "subsystem" "Suspend";
action "/etc/rc.suspend acpi $notify && service wifibox suspend";
};My wifibox.conf is just like you recommend when I did the test before. Are there any logs you would like me to pull?
pgj
commented
1 year ago The /var/log/wifibox.log could actually show (with DEBUG verbosity level) what commands were (or were not) run on suspend:
grep -F "with arguments" /var/log/wifibox.logDo you have devd enabled?
service devd statusIt is worth to check if /etc/devd.conf has the following lines (uncommented):
options {
# Each "directory" directive adds a directory to the list of
# directories that we scan for files. Files are loaded in the order
# that they are returned from readdir(3). The rule-sets are combined
# to create a DFA that's used to match events to actions.
directory "/etc/devd";
directory "/usr/local/etc/devd";
pid-file "/var/run/devd.pid";
...
}This default setting assumes that you have the third-party software and its configuration installed under /usr/local but I guess that holds for you.
venoom27
commented
1 year ago My /etc/devd.conf is just like you posted.
options {
# Each "directory" directive adds a directory to the list of
# directories that we scan for files. Files are loaded in the order
# that they are returned from readdir(3). The rule-sets are combined
# to create a DFA that's used to match events to actions.
directory "/etc/devd";
directory "/usr/local/etc/devd";
pid-file "/var/run/devd.pid";Could you please answer my other comment above? Without that, I cannot help. Currently, I have no idea why the suspend event is not handled properly on your system.
venoom27
commented
1 year ago Sorry I didn't see that response. Here is the info. Thanks again for your help.
grep -F "with arguments" /var/log/wifibox.log# grep -F "with arguments" /var/log/wifibox.log
2023-08-12T21:18:30-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: start
2023-08-12T21:18:30-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: _manage_vm
2023-08-12T21:20:28-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: stop
2023-08-12T21:20:32-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: start
2023-08-12T21:20:33-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: _manage_vm
2023-08-12T21:23:09-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: start
2023-08-12T21:23:09-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: _manage_vm
2023-08-13T20:45:20-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: start
2023-08-13T20:45:20-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: _manage_vm
2023-08-13T20:48:45-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: stop vmm
2023-08-13T20:49:15-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: stop vmm
2023-08-13T20:49:15-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: start vmm
2023-08-13T20:49:16-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: _manage_vm
2023-08-13T20:49:32-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: start vmm
2023-08-13T20:54:16-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: stop
2023-08-15T19:29:42-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: start
2023-08-15T19:29:42-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: _manage_vm
2023-08-15T19:37:34-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: stop
2023-08-17T21:05:51-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: start
2023-08-17T21:05:51-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: _manage_vm
2023-08-17T21:12:27-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: stop
2023-08-17T21:14:43-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: start
2023-08-17T21:14:43-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: _manage_vm
2023-08-19T07:44:23-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: start
2023-08-19T07:44:23-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: _manage_vm
2023-08-19T07:53:04-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: start
2023-08-19T07:53:04-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: _manage_vm
# service devd status
devd is running as pid 1699.
In wifibox.log, I can see stop vmm and start vmm commands although only once and in a very quick succession. They correspond to calling service wifibox suspend and then service wifibox resume so that part seems to be working. But we should align the timestamps with the invocations of acpiconf -s 3 and the subsequent wake-up.
What does the command grep -F "acpi[" /var/log/messages say? It should be displaying the suspend and resume times as handled by the acpi process. I have the following output but this might just be due to some local configuration:
Aug 20 01:18:58 XXX acpi[43580]: suspend at 20230820 01:18:58
Aug 20 01:19:18 XXX acpi[43768]: resumed at 20230820 01:19:18which have the corresponding matching lines in wifibox.log:
2023-08-20T01:19:13+0200 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: stop vmm
2023-08-20T01:19:18+0200 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: start vmmIn your log it appears like stop vmm and start vmm were called twice, kind of a result of some race. Whether does it always happen? Also, for the second start vmm, the _manage_vm (internal) command is not happening which suggests an error.
venoom27
commented
1 year ago Here is the results of # grep -F "acpi[" /var/log/messages
Aug 12 21:21:39 foo acpi[3824]: suspend at 20230812 21:21:39
Aug 12 21:25:34 foo acpi[2369]: suspend at 20230812 21:25:34
Aug 13 20:48:56 foo acpi[2527]: suspend at 20230813 20:48:56
Aug 13 20:49:15 foo acpi[2623]: resumed at 20230813 20:49:15
Aug 19 07:51:36 foo acpi[2765]: suspend at 20230819 07:51:36
I also ran another acpiconf -s 3 so it would show up in the logs. Here is the new result. It resumed correctly but froze on console and had to be hard shutdown.
grep -F "with arguments" /var/log/wifibox.log
2023-08-19T21:16:53-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: start
2023-08-19T21:16:53-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: _manage_vm
2023-08-19T21:26:08-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: start
2023-08-19T21:26:08-0400 DEBUG Program started as /usr/local/sbin/wifibox, with arguments: _manage_vm
# grep -F "acpi[" /var/log/messages
Aug 19 21:23:57 foo acpi[2669]: suspend at 20230819 21:23:57
Based on those logs, even though service wifibox suspend might be called, stop vmm is not issued. What do you have in /usr/local/etc/rc.d/wifibox?
grep -E "(suspend|resume)_cmd" /usr/local/etc/rc.d/wifibox# grep -E "(suspend|resume)_cmd" /usr/local/etc/rc.d/wifibox
suspend_cmd="${command} stop vmm"
resume_cmd="${command} start vmm"Do you have the wifibox service enabled?
service -e | grep -F wifibox# service -e | grep -F wifibox
/usr/local/etc/rc.d/wifibox
I do not see the reason why the suspend event would not be handled properly. Could you please try all this with a clean install, if possible? I will also do the same to see if my system has any non-standard setting that may influence the results.
venoom27
commented
1 year ago Unfortunately, I cannot easily do a clean install on this system and I didn't get a snapshot in before I started with wifibox.
pgj
commented
1 year ago For what it is worth, with enough RAM you can just run FreeBSD without having it installed to a hard disk. That is what I usually do to test Wifibox with different versions. The only requirement that it will need a wired connection to download the files from the Internet when the file system in the memory is being populated.
For example, for 13.2-RELEASE, something along these lines:
memstick image from the releases site.dd if=FreeBSD-13.2-RELEASE-amd64-memstick.img of=/dev/da0 bs=1m conv=sync where /dev/da0 is the device of the USB device.tmpfs partition: mount -t tmpfs tmpfs /mnttar -xvf /usr/freebsd-dist/base.txz -C /mnt && tar -xvf /usr/freebsd-dist/kernel.txz -C /mntdevfs file system for the bootstrapped file system: mount -t devfs devfs /mnt/devchroot /mntdhclient em0 (I have an Intel NIC which uses the em driver, this might be different).pkg: pkg updateportsnap fetch extractgit: pkg install gitdev branch): cd /tmp && git clone https://github.com/pgj/freebsd-wifibox-port -b devvenoom27
commented
1 year ago Thanks so much for the info and I will try it out this evening.
venoom27
commented
1 year ago I started it this evening and was building the wifibox port but my time for my only hardline connect ran out so I will have to try again tomorrow. I will keep you updated.
venoom27
commented
1 year ago OK I tired again and I was able to install Wifibox/Alpine but when I got through the middle of wifibox-core it errored out saying "kernel: pid 74380 (getty), kid 0, uid 0, was killed: failed to reclaim memory" so I am not able to install it through the live cd.
For the purpose of troubleshooting I will do a fresh install in the next day or two and see if it works. I will let you know my results. Also you wouldn't know a link to how to backup zfs and then restore it would you?
venoom27
commented
1 year ago Great news I did a clean install and suspend works with wifibox installed. I figured out how to install it on another dataset so I still have the old install. I did not set up any passthru. Please let me know how you want to proceed.
pgj
commented
1 year ago Sorry that I did not respond to your earlier comment, doing it now.
I got through the middle of wifibox-core it errored out saying "kernel: pid 74380 (getty), kid 0, uid 0, was killed: failed to reclaim memory" so I am not able to install it through the live cd.
That is probably because the RAM is not enough to hold all the files and run programs. I have 8 GB in my machine and I have to clean up the file system in memory regularly to avoid similar situations.
you wouldn't know a link to how to backup zfs and then restore it would you?
Unfortunately I do not use ZFS for performance reasons. I deemed that with 8 GB it is not worth the trouble. I do not get in touch with other FreeBSD installations these days so I do not have a working knowledge of these operations. But it is good that you could figure it out!
pgj
commented
1 year ago Great news I did a clean install and suspend works with wifibox installed. I figured out how to install it on another dataset so I still have the old install. I did not set up any passthru. Please let me know how you want to proceed.
Configure SUSPEND_VMM as the recovery method, enable the wifibox service, set up the passthrough. Test if it the wireless connection works and then try it with suspend/resume.
venoom27
commented
1 year ago I got the passthru up and running and the SUSPEND_VMM flag on and it suspends and wakes the way it should.
pgj
commented
1 year ago That is great news! I can now cut a new release with this fix.
pgj
commented
1 year ago The support for SUSPEND_VMM has been added to main in d5f07d2f. I am now closing the ticket.
Description
When installing the main branch of wifbox-apline with broadcom-wl drive my suspend to ram stops working. When I say stops working it will go to sleep but it will not wake. When I was testing some of the earlier dev branchs and the broadcom-wl driver was not assigned a device wlan0 this was not an issue. It seems to be once the broadcom-wl module got loaded and assigned to wlan0 in the wifibox this became an issue. If wifbox is not installed suspend to ram works.
Host operating system
Wireless NIC
Wifibox version
Disk image type and version
The kind of VM image in use, e.g. Wifibox/Alpine, and its version.
Changes to the default configuration files
Logs
Additional context
Let me know if you need any other information.
Have you tried to turn it on and off?