Mainsailos looses wifi after a while

xp1ratex commented 3 years ago

Mainsailos looses wifi after a while, top shows nothing out of the ordinary, free show nothing either. If i reconnect the ip in unifi it comes back. Rasberry pi 4 8Gb klipper v0.9.1-627-g121052ad moonraker v0.7.1-32-g42f61ce mainsail v1.6.0

TOP free

any clues where to start looking? never had this problem before on octoprint.

KwadFan commented 3 years ago

Please share the output of sudo iwconfig wlan0

xp1ratex commented 3 years ago

Please share the output of sudo iwconfig wlan0

Here it is, got tired of trying to get it to work so i connected it with a ethernet cable instead, now it works flawless. wlan0

dstack commented 3 years ago

I ran into a similar issue. Specifically, Wifi-based networking would start, then seriously degrade, leading to timeouts, and then an eventual disconnect.

Steps to re-produce:

Fresh install of MainsailOS
SSID and password in wpa-supplicant, specifically 2.4Ghz network
Raspberry Pi 4 B 2Gb, located approx. 30ft from AP.

I got about 2-3 minutes of solid < 10ms pings, then it would jump to 2000+ms and over 30 seconds, disconnect. I tried this setup with and without KlipperScreen (which has a built-in wifi manager I thought could be interfering) and also with a DSI touchscreen (the touchscreen made it fail faster, but I was able to get this behavior consistently without the DSI screen).

Troubleshooting:

I tried a variety of power setups, including connecting to a 5V 25W power supply. Underpowering of course produced worse results, but standard and/or over-powering didn't change the behavior.
I tried swapping out the boot disk with a standard RaspberryOS Lite, as well as an older install of RetroPi. Neither of these installations could produce the same result, which is to say, they worked fine.
I was able to repeat the result when swapping out the Pi. I tried with 2 other RPi 4 B 2Gbs, an Rpi 400, and an RPi 4 B 8Gb. All of them were able to reproduce the issue.
I tried a variety of locations for this Pi (not a practical fix since the printer cannot move much from where it is), and only got some success by placing the Pi VERY close to the AP (< 6ft)

So, the major variables I altered in the troubleshooting were the location, the hardware, and the OS image, and the only noticeable result came from swapping out the OS.

The responsible response here would have been to switch back to using a Raspberry OS base, with Kiauh, and then putting the Mainsail stack on top, however ...

My Workaround:

I updated the raspberry pi firmware, since this specific Pi 4 was one of the older ones in my collection. This improved the situation a little, as I was able to get 5-7 minutes of solid connection on average. This is normally not advisable, and you run the risk of bricking your Pi, so I will not provide the instructions on how to do this.
Based on the fact that connecting the DSI screen made the issue worse, reading about an older issue concerning DSI screens and 2.4Ghz wifi specifically, and following a hunch, I changed this Pi to use the 5Ghz network instead. None of my Pis do this, since I specifically relegate all the "IoT" / "Small-server" devices to the 2.4Ghz. This provided a reasonably stable connection, even with the DSI screen attached. I say reasonably stable, because I have noticed a periodic latency spike. Generally, other devices on my network can ping this Pi with < 10ms latency, occasionally jumping to around 150ms, which I would consider normal, however, every 5 minutes or so, there's a huge latency spike resulting in a 3000+ms ping or two, which then goes back to <10ms.

Conclusions: It would be easy to dismiss this as part of the known issue with some DSI screens and 2.4Ghz wifi on Raspberry Pi 4s, except that it does not explain why RetroPi and base RaspberryOS Lite did not suffer from this issue. It can't be location, it can't be hardware, and while it could be power, I'd find it hard to believe that all 5 of the power supplies I used were somehow not pushing 5.1V.

Without digging in, I don't know if MainsailOS is doing anything special with wifi management, but this OS image was the only one that would reproduce the issue, every time, regardless of other variables.

meteyou commented 3 years ago

thx for your report. the interesting thing is, that mainsailOS is a nearly pure raspberry pi os lite. we just disable the power management of the wifi chip, because this setting set the wifi chip sometimes in a sleep mode, when it is idle.

jottr commented 2 years ago

@meteyou I couldn't find the place where you actually disable power management.
Could you kindly point me to the relevant place in the source code?

KwadFan commented 2 years ago

@meteyou I couldn't find the place where you actually disable power management. Could you kindly point me to the relevant place in the source code?

Hello again, So here is your pointer. This is implemented through the "network" Module of CustomPIOS, the framework that we use to modify the Lite Image.

Regards

AndreyShpilevoy commented 2 years ago

Also, have the same issue with RPi4. Yesterday I installed the latest version day before and raised with exact same issue already twice. 1st time during printing test cube, second time today in the morning. I checked my wifi router - raspberry pi was disconnected. Interesting that after 10-15 minutes it's moved back to online without any manipulations from my side.

Markus-1984 commented 2 years ago

Same issue here, but with a RPi3B. After a few minutes, mainsail can´t connect to moonraker. After a while it´s connected again, but only for few minutes.

0Jayy07 commented 2 years ago

Same issue but with a RPi3B+ and Pi Zero 2W

Accidentally when the printer prints or is just in standby, Mainsail loses the connection to the wifi. The pi is still listed in the Fritzbox but you can no longer connect to the Printer

dz0ny commented 2 years ago

Same issue but with a RPi3B+ and Pi Zero 2W

Accidentally when the printer prints or is just in standby, Mainsail loses the connection to the wifi. The pi is still listed in the Fritzbox but you can no longer connect to the Printer

Exactly same issue and setup, can you check if you have huge spike in WIFI channel allocation when you see dropout

under

0Jayy07 commented 2 years ago

I will check it these days and let you know!

On Sun, 9 Jan 2022, 20:34 Janez Troha, @.***> wrote:

Same issue but with a RPi3B+ and Pi Zero 2W

Accidentally when the printer prints or is just in standby, Mainsail loses the connection to the wifi. The pi is still listed in the Fritzbox but you can no longer connect to the Printer

Exactly same issue and setup, can you check if you have huge spike in WIFI channel allocation when you see dropout

[image: image] https://user-images.githubusercontent.com/239513/148697799-a13d6903-3704-4ebd-9b84-89dcb5b87d6d.png under [image: image] https://user-images.githubusercontent.com/239513/148697807-56746410-d745-45c5-bda9-0be9d5e3df3c.png

— Reply to this email directly, view it on GitHub https://github.com/mainsail-crew/MainsailOS/issues/47#issuecomment-1008360282, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXDVBUH7UGF3ZGN2U4QE53LUVHPL7ANCNFSM5A2QWX3A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

tommyl2210 commented 2 years ago

I am also loosing connection. Sometimes shortly after booting, sometimes when sending a gcode command during execution of another gcode command. e.g. change Z-offset during homeing. But also during printing... Print works normal but no access through wifi is possible. This didnt happen with exact same hardware and octoprint.

dz0ny commented 2 years ago

So enabling back the wifi power_save helps with the reconnection, before I had to restart by unplugging the power. Now it reconnects after a minute, I also tried with the new firmware-brcm80211 from the RPI repo, but I didn't get any improvement http://archive.raspberrypi.org/debian/pool/main/f/firmware-nonfree/firmware-brcm80211_20210315-3+rpt4_all.deb

meteyou commented 2 years ago

This didnt happen with exact same hardware and octoprint.

this is really funny, because octoPI (the raspberry pi image of octoprint) and mainsailOS uses the exact same settings...

dz0ny commented 2 years ago

I think this is something else, is something else masquerading as a wifi issue. I've tried RPI lite (official) and I get the same issue.

meteyou commented 2 years ago

mainsailOS is based on a default rpi lite and then customizerPI just disable the power management of the wifi chip.

octoPI have only an additional service to ping the router or something else to have a traffic the full time.

dz0ny commented 2 years ago

octoPI have only an additional service to ping the router or something else to have a traffic the full time.

https://github.com/guysoft/OctoPi/blob/devel/src/modules/octopi/filesystem/root/etc/systemd/system/networkcheck.service https://github.com/guysoft/OctoPi/blob/devel/src/modules/octopi/filesystem/root/etc/systemd/system/networkcheck.timer https://gist.github.com/dz0ny/b90bcca9d60f6e8a2a33c55132d68c52#file-networkcheck

Lets see if it helps

meteyou commented 2 years ago

But I think this is not enabled per default in octopi...

Aussie84d commented 2 years ago

I'm having the same issue. Could it be related to idle_timeout?

0Jayy07 commented 2 years ago

[image: image.png] 1min Scale 2.4Ghz

[image: image.png] 10min scale 2.4Ghz

[image: image.png] 1min Scale 5Ghz

[image: image.png] 10min Scale 5Ghz

Am So., 23. Jan. 2022 um 00:48 Uhr schrieb Dan @.***>:

I'm having the same issue. Could it be related to idle_timeout?

— Reply to this email directly, view it on GitHub https://github.com/mainsail-crew/MainsailOS/issues/47#issuecomment-1019378545, or unsubscribe https://github.com/notifications/unsubscribe-auth/AXDVBUGRUZKLDIQMHVGSNM3UXM65HANCNFSM5A2QWX3A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

da97ed commented 2 years ago

Same issue but with a RPi3B+ and Pi Zero 2W Accidentally when the printer prints or is just in standby, Mainsail loses the connection to the wifi. The pi is still listed in the Fritzbox but you can no longer connect to the Printer

Exactly same issue and setup, can you check if you have huge spike in WIFI channel allocation when you see dropout

under

I can confirm this. Got those spikes as well when loosing connection to mainsail. Unfortunately I`m loosing the connection quite often. I did a full re-install of MainsailOS and disconnected all my webcams. But the issue seems to consist. I am also unable to ping my raspberry during that off time. Terminal returns "host is down". Any help on this is much appreciated!! Im using Pi3B+ btw

meteyou commented 2 years ago

@da97ed pls report this bug to raspberry pi os... we just use a generic raspberry pi os lite with add klipper, moonraker, nginx (for mainsail) and mjpegstreamer...

gurusonwheels commented 2 years ago

i have same issue with pi4 driving me nuts .. have mainsailos also

meteyou commented 2 years ago

@gurusonwheels @da97ed @Aussie84d @dz0ny @tommyl2210 @namelessFPV @Markus-1984 @xp1ratex

to diagnose the problem we would need some data from you. can you please check if you have 2.4ghz and 5ghz active and if both networks have the same SSID name?

furthermore it would be good to know if "band steering" is active on your router.

dz0ny commented 2 years ago

to diagnose the problem we would need some data from you. can you please check if you have 2.4ghz and 5ghz active and if both networks have the same SSID name?

yes

furthermore it would be good to know if "band steering" is active on your router.

yes, cannot be turned off

dz0ny commented 2 years ago

I'am also getting much better stability after I've upgraded to latest kernel:

Linux prusa 5.15.18-v7+ #1520 SMP Fri Feb 4 11:58:51 GMT 2022 armv7l GNU/Linux
pi@prusa:~ $ sudo rpi-update next

Only two disconnects in three days:

[   12.329929] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -23 :toe_ol
[   12.329953] brcmfmac: CONSOLE: wl0: wl_open
[   12.344142] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save enabled
[   12.369933] brcmfmac: CONSOLE: wl0: wlc_enable_probe_req: state down, deferring setting of host flags
[   12.369955] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -23 :ndoe
[   12.369964] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -23 :ndoe
[   12.369972] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -23 :ndoe
[   12.520038] brcmfmac: CONSOLE: wl0: wl_open
[   12.560045] brcmfmac: CONSOLE: wl0: wlc_enable_probe_req: state down, deferring setting of host flags
[   12.720059] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -23 :ndoe
[   13.680030] brcmfmac: CONSOLE: wl0: link up (wl0)
[   13.680059] brcmfmac: CONSOLE: E/GTK: wlc_bcol_process_gtk_rekey: 4 handshake frame, skip
[   13.780048] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x1 tid 0
[   13.800145] brcmfmac: CONSOLE: E/GTK: wlc_bcol_process_gtk_rekey: 4 handshake frame, skip
[   19.950200] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -23 :ndoe
[   25.792963] uart-pl011 3f201000.serial: no DMA platform data
[   33.769761] cam1-reg: disabling
[   33.769799] cam-dummy-reg: disabling
[57434.824651] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -23 :ndoe
[57436.084561] brcmfmac: CONSOLE: wl0: link up (wl0)
[57436.084596] brcmfmac: CONSOLE: E/GTK: wlc_bcol_process_gtk_rekey: 4 handshake frame, skip
[57436.114553] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x1 tid 0
[57436.144595] brcmfmac: CONSOLE: E/GTK: wlc_bcol_process_gtk_rekey: 4 handshake frame, skip
[57442.054766] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -23 :ndoe
[58542.314757] ieee80211 phy0: send_key_to_dongle: wsec_key error (-52)
[58542.337456] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -51 :wsec_key
[58542.477534] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -23 :ndoe
[58543.437310] brcmfmac: CONSOLE: wl0: link up (wl0)
[58543.437339] brcmfmac: CONSOLE: E/GTK: wlc_bcol_process_gtk_rekey: 4 handshake frame, skip
[58543.517397] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x1 tid 0
[58543.517435] brcmfmac: CONSOLE: E/GTK: wlc_bcol_process_gtk_rekey: 4 handshake frame, skip
[58549.097426] brcmfmac: CONSOLE: wl0: wlc_iovar_op: BCME -23 :ndoe
pi@prusa:~ $ uptime
 13:50:46 up 3 days,  3:28,  1 user,  load average: 0.27, 0.11, 0.09

meteyou commented 2 years ago

@dz0ny thx four this important info! can you also try to disable "band steering" and test it for some days and if it dont work better, than try to rename one of the networks (2.4ghz or 5ghz) to have seperate names. maybe the rpi have issues, when the router change the band and lost the connection.

dz0ny commented 2 years ago

Same outcome (disabled all advanced things by forcing router 11g mode), different logs this time

[  291.819320] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x3e6 tid 0
[  380.948945] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x42 tid 0
[  381.029541] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x50 tid 0
[  381.296604] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x52 tid 0
[  381.406537] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x58 tid 0
[  381.476528] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x5c tid 0
[  397.426418] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x24d tid 0
[  619.636843] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x377 tid 0
[  619.916795] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x3ba tid 0
[  631.989933] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0xf27 tid 0
[  632.226819] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0xf32 tid 0
[ 1797.738846] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0xd98 tid 0
[ 1797.778791] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0xd9a tid 0
[ 1848.972071] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x90d tid 0
[ 2092.039516] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x178 tid 0
[ 2092.069574] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x17a tid 0
[ 2250.983654] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x65a tid 0
[ 2251.063732] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x662 tid 0
[ 2251.203634] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x665 tid 0
[ 2251.283651] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x669 tid 0
[ 2251.390614] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x66a tid 0
[ 2251.510542] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0x66b tid 0
[ 2257.630640] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0xbd9 tid 0
[ 2257.680683] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0xbe1 tid 0
[ 2257.833668] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0xbe3 tid 0
[ 2257.910657] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0xbe9 tid 0
[ 2257.950799] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0xbeb tid 0
[ 2258.070622] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0xbf0 tid 0
[ 2258.260881] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0xbf1 tid 0
[ 2259.750630] brcmfmac: CONSOLE: wl0.0: wlc_send_bar: seq 0xd72 tid 0

pi@prusa:~ $ sudo cat /sys/kernel/debug/ieee80211/phy0/revinfo
vendorid: 0x14e4
deviceid: 0x43e2
radiorev: 64.61.160
chip: BCM43430/2
chippkg: 4
corerev: 39
boardid: 0x0726
boardvendor: 0x14e4
boardrev: P101
driverrev: 9.88
ucoderev: 0
bus: 0
phytype: 12
phyrev: 1
anarev: 0
nvramrev: 00000000
clmver: API: 15.0
Data: 9.9.5_SS
Compiler: 1.29.4
ClmImport: 1.47.0
Customization: V12_181206
Creation: 2020-04-23 11:01:34

Frankly, this looks like a bug in a firmware blob.

KwadFan commented 2 years ago

Hello, since there are many peoples, having the same Issue with connection lost, we decided to develop a solution. It isn't the best solution but the most simpliest to avoid conflicts with changing system-depended firmware blobs or similar.

Please try out https://github.com/mainsail-crew/sonar, it is in WIP status, but we could archieve keeping an OrangePI with armbian online all the time during a period of one week, and they are even more picky in that topic.

Any suggestions to improve or change the service are welcome, please use Issue tracker of the project for that.

Regards

da97ed commented 2 years ago

thank you! works for me

ccfman2004 commented 1 year ago

I know this is an old thread but I am having the exact same issue with WiFi on Mainsail 2.5.1 with everything up to date.

The only fix so far is to turn my printer off and the back on. I didn't always have this issue. It only appeared in April after an update.

github-actions[bot] commented 1 year ago

Ahoi!

It looks like this ticket is a request for help (or similar). Many helpful people will not see your message here and you are unlikely to get a useful response.

We use github to handle bugreports, feature requests and planning new releases.

Please use our Discord-Server for help: discord.gg/mainsail

This ticket will be automatically closed.

Fair wind and a following sea! ~ Your friendly MainsailGithubBot

PS: I'm just an automated script, not a real sailor.

mainsail-crew / MainsailOS

Mainsailos looses wifi after a while #47