Frequent WiFi disconnects

bjornarsu commented 6 years ago

I'm experiencing frequent, random wifi disconnects with volumio using a Raspberry Pi 3 w/ internal WiFi. I believe this look very similar to the a bug in the WiFi driver mentioned here. I'm still able to patch in a ethernet cable and connect with SSH after wifi goes down, so volumio has not crashed.

Version info: System Version 2.348 Released Fri Dec 22 22:48:12 CET 2017 Linux volumio 4.9.65-v7+ #1056 SMP Fri Nov 24 13:58:07 GMT 2017 armv7l GNU/Linux

Setup: Raspberry Pi 3 w/ PiFi DAC+ Network: Unifi UAP AC LR

Output from dmesg after WiFi disconnect below:

[ 644.102757] brcmfmac: brcmf_sdio_hostmail: Unknown mailbox data content: 0x40012 [ 713.449497] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [ 716.009505] brcmfmac: _brcmf_set_multicast_list: Setting mcast_list failed, -110 [ 718.569495] brcmfmac: _brcmf_set_multicast_list: Setting allmulti failed, -110 [ 721.129500] brcmfmac: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -110 [ 728.809539] brcmfmac: _brcmf_set_multicast_list: Setting mcast_list failed, -110 [ 731.369478] brcmfmac: _brcmf_set_multicast_list: Setting allmulti failed, -110 [ 733.929472] brcmfmac: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -110 [ 741.609461] brcmfmac: brcmf_cfg80211_disconnect: error (-110) [ 743.001018] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xC1E1 [ 747.369447] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [ 747.369551] brcmfmac: power management disabled [ 749.929485] brcmfmac: brcmf_cfg80211_set_power_mgmt: error (-110) [ 752.489484] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [ 752.489500] brcmfmac: brcmf_cfg80211_get_channel: chanspec failed (-110) [ 755.049510] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [ 755.049529] brcmfmac: brcmf_cfg80211_get_tx_power: error (-110) [ 757.609438] brcmfmac: _brcmf_set_multicast_list: Setting mcast_list failed, -110 [ 760.169431] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [ 760.169441] brcmfmac: brcmf_cfg80211_get_channel: chanspec failed (-110) [ 762.729445] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [ 762.729457] brcmfmac: brcmf_cfg80211_get_tx_power: error (-110) [ 764.011437] CIFS VFS: Server ARKIV has not responded in 120 seconds. Reconnecting... [ 765.289422] brcmfmac: _brcmf_set_multicast_list: Setting allmulti failed, -110 [ 767.849417] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [ 767.849427] brcmfmac: brcmf_cfg80211_get_channel: chanspec failed (-110) [ 770.409495] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [ 770.409508] brcmfmac: brcmf_cfg80211_get_tx_power: error (-110) [ 772.969411] brcmfmac: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -110 [ 783.209425] brcmfmac: brcmf_vif_set_mgmt_ie: vndr ie set error : -110 [ 783.209433] brcmfmac: brcmf_cfg80211_scan: scan error (-110)

bjornarsu commented 6 years ago

Update: I tried what somebody suggested in the thread of the similar issue I linked to (to set the wifi interface in promiscuous mode) and so far I have not had a single WiFi disconnect. What I did was adding the line 'sudo ifconfig wlan0 promisc' into Wifistart.sh so that the WiFi interface is set to promiscuous mode on each boot.

So to me it seems this issue is indeed a bug in the RPI 3 internal WiFi firmware, and it's not ideal to have to set the wlan0 interface to promiscuous mode by default. Are there any patches in the pipeline for Raspbian?

macmpi commented 6 years ago

Seems like brcm firmware fix is finally well on the way... (been tracking original issue for months) You may test it from here. Let's see if it is released soon now, and I'm sure a following Volumio2 release will include it.

bjornarsu commented 6 years ago

Thank you @macmpi. I have installed that firmware update now, and will report back with results soon.

bjornarsu commented 6 years ago

I have run Volumio with the new wifi firmware loaded now (and promiscuous mode deactivated again) and so far it's looking very good - no wifi crashes at all. It seems to run very stable, and I so far can't see any negative impacts. I can recommend upgrading to this firmware for anyone having similar issues.

macmpi commented 6 years ago

Sweet... I'm sure lots of Volumio users on Pi3 and Pi0W will love that!

Let's have it finish its round of pre-release tests, and monitor when brcmfmac43430-sdio.bin release version lands here (more recent than current 2 years old): the following Volumio2 build should then nicely pick-it-up...

kayue commented 6 years ago

My Pi3b has been experiencing very poor Wifi connection too, and I am testing the new brcmfmac43430-sdio.bin framework now. No problem so far. Will report if I found any issues. Thanks @bjornarsu for the tips.

kayue commented 6 years ago

I am using Volumio to stream Spotify, and music still being interrupted every once a while. I couldn't found any log tho.

However I believe the wifi performance has been improved because at last I can connect to the web UI now, which often fail in the past.

volumio commented 6 years ago

Great find @macmpi ! IMHO we can include it as soon as it's confirmed to improve the situation and the foundation releases it for production

macmpi commented 6 years ago

Indeed @volumio : current Pi build script already takes care of that, so it should be "automagically" in, as long as Foundation releases it within firmware-brcm80211 package.

macmpi commented 6 years ago

@volumio Please note new firmware is now live and available in Foundation's firmware-brcm80211 apt package. So any next Volumio2 build should now get the updated firmware file. ;)

volumio commented 6 years ago

Great news! I plan to release a new version this week. Thanks for reporting

macmpi commented 6 years ago

@bjornarsu did you have a chance to test newest Volumio release? Have wifi connections improved then? Thanks for feedback; you may eventually close the issue if resolved.

bjornarsu commented 6 years ago

@macmpi I will check it sometime this month and report back. I'm a bit busy these days, and my previous version Volumio setup with firmware patch is working good, and I would like to wait till I have some spare time in case the update breaks anything. :)

macmpi commented 6 years ago

Note: this wireless firmware saga is not over, since the early Feb 'interim' firmware seemed to cause issues (particularly to Pi0W), so it has been reverted, awaiting from a newer version.

Therefore newly built Volumio v2.376 contains the reverted firmware (same as original):

volumio@volumio:~$ apt-cache policy firmware-brcm80211
firmware-brcm80211:
  Installed: 1:0.43+rpi6
  Candidate: 1:0.43+rpi6

Only v2.368 contained the 'interim' firmware.

bjornarsu commented 6 years ago

I have not seen any issues in 2.368, WiFi has been stable. Does the old Cypress firmware even work in a stable manner for anyone using the Rpi 3?

volumio commented 6 years ago

@macmpi what about hosting ourselves the proper working firmware?

macmpi commented 6 years ago

@bjornarsu can you confirm package version you currently use by:

sudo apt-get update
apt-cache policy firmware-brcm80211

@volumio In fact I think we just have to trust Pi Foundation's determination in deciding which one is causing the least number of issues...(they supposedly do significant testing, and collect lots of users feedback). So I'd say better use their packages at all times I think (as it currently is in Volumio build).

I routinely check status on related forum & github topics referenced in this discussion. Hopefully a final version will emerge soon-enough: this problem has been on for years, we've never been so close...

volumio commented 6 years ago

@macmpi hope so.. Fingers crossed then

macmpi commented 6 years ago

Ok, my bad, v2.378 and earlier never contained the fixed firmware. It will come in some future releases: until then you may install it with:

wget http://archive.raspberrypi.org/debian/pool/main/f/firmware-nonfree/firmware-brcm80211_20161130-3+rpt3_all.deb
sudo dpkg -i firmware-brcm80211_20161130-3+rpt3_all.deb
rm firmware-brcm80211_20161130-3+rpt3_all.deb

Note: this is still for tinkerers:

it may still have issues with Pi0W in particular, until a version with updated kernel comes-up
if you do install, it is advised to wipe all data at next OTA upgrade (you may save/restore settings with Back-up/Restore plugin)

macmpi commented 6 years ago

Actually, and maybe simpler, users impacted by this issue may want to give the new Pi3+ beta image a shot, as it should improve things on this front too. Thanks for feedback.

bjornarsu commented 6 years ago

@macmpi The installed version according to apt is 1:0.43+rpi6, but it looks like the firmware that actually is installed and in use is the new one that I installed manually before the update. It could be that when I last upgraded Volumio, the manually installed newer firmware was never overwritten with the old version, since the version in the apt repository never changed?

The firmware file currently on my system is: -rw-r--r-- 1 root root 388739 Jan 24 20:40 /lib/firmware/brcm/brcmfmac43430-sdio.bin brcmfmac: Firmware version = wl0: Oct 23 2017 03:55:53 version 7.45.98.38 (r674442 CY) FWID 01-e58d219f

macmpi commented 6 years ago

Yes, due to Volumio system layout with overlayfs, things you alter manually may take precedence over later OTA. So best it to start fresh, or wipe all user data. (latest is indeed 7.45.98.38)

gloyboy commented 4 years ago

I, too, have had frequent disconnect problems with Volumio and a HiFiBerry Digi+ on a fresh install on a Raspberry Pi 4b. I checked the power supply, which is the official 3amp foundation appliance, and it checks out fine. I powered my 64GB USB Memory stick from a powered USB hub, so power gaps were not a problem with my setup.

After even installing a new 1GB network switch, I finally changed Volumio networking from Ethernet (eth0) only to wireless only (wlan0) and it seem better, but still would not stay up for more than a maximum of 2 days. Still running on wireless (wlan0), I disabled the HotSpot and it has been running fine every since. Not sure why my network topology did not play nice with the HotSpot, but turning it off solved the constant disconnect problems.

YMMV

bjornarsu commented 3 years ago

@volumio Hi again! After the latest Volumio this issue is back for me. The firmware version currently installed after the latest Volumio update is the following:

brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Mar 2 2020 23:30:41 version 7.45.202 (r724630 CY) FWID 01-72f6ece2

bjornarsu commented 3 years ago

@volumio Update: I have now downloaded and installed the very latest brcmfmac43455-sdio firmware from Cypress from https://community.cypress.com/docs/DOC-21490 and I'm currently testing if it's stable. I'll report back after testing it for a few days.

volumio commented 3 years ago

Thanks. I wonder what might have changed since we did not change the FW

duchenpaul commented 3 years ago

@volumio Update: I have now downloaded and installed the very latest brcmfmac43455-sdio firmware from Cypress from https://community.cypress.com/docs/DOC-21490 and I'm currently testing if it's stable. I'll report back after testing it for a few days.

Hey, @bjornarsu, I would like to know how to patch this firmware, like all of sudden, two of my pis cannot connect to wifi due to this reason, I can only access files using card read to get the logs like below. my pis are pi 3 B and zero W. Weird is another pi 3 B does not have this issue at all.

Oct  9 22:05:32 venus kernel: [   87.603046] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
Oct  9 22:05:32 venus kernel: [   87.603070] brcmfmac: brcmf_cfg80211_get_channel: chanspec failed (-110)
Oct  9 22:06:34 venus kernel: [  149.603043] brcmfmac: brcmf_link_down: WLC_DISASSOC failed (-110)
Oct  9 22:06:37 venus kernel: [  152.243020] brcmfmac: brcmf_do_escan: error (-110)
Oct  9 22:06:37 venus kernel: [  152.243040] brcmfmac: brcmf_cfg80211_scan: scan error (-110)
Oct  9 22:06:40 venus kernel: [  154.803044] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
Oct  9 22:06:42 venus kernel: [  157.363074] brcmfmac: _brcmf_set_multicast_list: Setting mcast_list failed, -110
Oct  9 22:06:45 venus kernel: [  159.923117] brcmfmac: _brcmf_set_multicast_list: Setting allmulti failed, -110
Oct  9 22:06:47 venus kernel: [  162.483091] brcmfmac: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -110
Oct  9 22:06:55 venus kernel: [  170.163388] brcmfmac: _brcmf_set_multicast_list: Setting mcast_list failed, -110
Oct  9 22:06:57 venus kernel: [  172.723044] brcmfmac: _brcmf_set_multicast_list: Setting allmulti failed, -110
Oct  9 22:07:00 venus kernel: [  175.283076] brcmfmac: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -110
Oct  9 22:07:10 venus kernel: [  185.523038] brcmfmac: brcmf_do_escan: error (-110)
Oct  9 22:07:10 venus kernel: [  185.523058] brcmfmac: brcmf_cfg80211_scan: scan error (-110)
Oct  9 22:07:13 venus kernel: [  188.083037] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
Oct  9 22:07:13 venus kernel: [  188.083060] brcmfmac: brcmf_cfg80211_get_tx_power: error (-110)
Oct  9 22:07:15 venus kernel: [  190.643051] brcmfmac: brcmf_do_escan: error (-110)
Oct  9 22:07:15 venus kernel: [  190.643071] brcmfmac: brcmf_cfg80211_scan: scan error (-110)
Oct  9 22:07:18 venus kernel: [  193.283033] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
Oct  9 22:07:18 venus kernel: [  193.283054] brcmfmac: brcmf_cfg80211_get_tx_power: error (-110)
Oct  9 22:08:21 venus kernel: [  256.563079] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
Oct  9 22:08:21 venus kernel: [  256.563102] brcmfmac: brcmf_cfg80211_get_channel: chanspec failed (-110)

bjornarsu commented 3 years ago

@duchenpaul I still have these issues even with the latest brcmfmac firmware I mentioned. I will try more version until I find one that works. Maybe we have to go back to an older firmware than the one currently in Volumio even...

@volumio Then I would guess it changed upstream in Raspbian?

macmpi commented 3 years ago

Note that Pi Foundation releases may include ad-hoc bug-fixes and customized nvram-file parameters (like for wifi/BT coexistence modes), that you may not get while installing files from Cypress source...

So it's probably wise to track Foundation releases here and try those packages. (Up to date packages are automatically installed when updating to Foundation's latest OS releases -Buster and 20190114-1+rpt8 package as of now- but as Volumio may be based on earlier Raspbian, they need to be specifically installed in raspberryconfig.sh script)

sebastianeferrer commented 3 years ago

Hi everyone! A bugfix for CYW43455 has arrived!

You should check this out -> https://github.com/RPi-Distro/firmware-nonfree/issues/8

volumio / Volumio2

Frequent WiFi disconnects #1462