wlan freezes in raspberry pi 3/PiZeroW (Not 3B+)

[dh-connect]

I put the same sd card (running debian 8 jessie, kernel 4.1.19) from the raspberry pi 2 with usb wifi (EDIMAX EW-7811UN Wireless USB Adapter, 150 Mbit/s, IEEE802.11b/g/n) into the new raspberry pi 3 using integrated wlan. Since then the wlan freezes after while (several hours) of usage couldn't find out if it's due to havy wifi usage or not, because I haven't change the software I guess it has to do with the new hardware. When the wlan freezes the pi can't be reached any longer, neither ifdown + ifup nor restart networking service helps in this case, I have to reboot the system to get it back to work, syslog doesn't say much only this: dhcpcd[522]: wlan0: fe80::8af7:c7ff:fece:5912: expired option 25,

I've tried to change these settings so far, but without improvement:

sudo nano /etc/network/interfaces wireless-power off

sudo nano /etc/sysctl.conf at the end of the file add the following line vm.min_free_kbytes = 16384

sudo nano /boot/cmdline.txt At the end of the line, add: smsc95xx.turbo_mode=N dwc_otg.dma_enable=1 dwc_otg.dma_burst_size=256

[clivem]

EDIMAX EW-7811UN.... That's using rtl8188cus chipset, IIRC.

If you haven't already got one, create /etc/modprobe.d/8192cu.conf, with content....

Disable power management

options 8192cu rtw_power_mgnt=0 rtw_enusbss=0

[Hecatron]

The rpi3 actually uses the brcmfmac driver for the inbuilt wifi there is an issue that requires the power saving / mangement to be turned off

I think the newer raspian kernels have patched this already to disable power saving by default but I don't think it's in this 4.5 branch yet

What I'm doing at the moment (gentoo install) is the following at bootup to disable the power saving on the wifi card

iw wlan0 set power_save off

[clivem]

The rpi3 actually uses the brcmfmac driver for the inbuilt wifi

Yes, I know. Oh I see. He's not using the EDIMAX EW-7811UN dongle anymore. He used to use it with RPi2.

[dh-connect]

yes I don't use the usb wifi any longer, how do I set up the cmd line to turn off the power management? crontab @reboot iw wlan0 set power_save off

[Hecatron]

Not sure for raspian, since I'm using gentoo it'll be different

[pelwell]

Try this: https://gist.github.com/mkb/40bf48bc401ffa0cc4d3

[dh-connect]

Seems to work since I have turned the powermanagement off I haven't had another wlan crash.

[dh-connect]

Just to mentioned it, to restart the wlan automatically after a crash, this here helps: sudo cp /etc/wpa_supplicant/ifupdown.sh /etc/ifplugd/action.d/ifupdown

[popcornmix]

BTW, latest apt-get upgrade kernel has power management disabled by default. @dh-connect does this work for you if you remove your current workaround?

[dh-connect]

it's still crashing after the latest upgrade, now i get this error in syslog: brcmfmac: brcmf_sdio_bus_txdata: out of bus->txq !!!

[pelwell]

When you say it's crashing, are there symptoms other than the error message?

[dh-connect]

no, just the one I have posted here but it is in the log many times

the wlan stops working, i can still work with it but to get the wlan back working I have to reboot it

[pelwell]

Thanks - I think "wlan stops working" counts as a symptom.

[dh-connect]

I've tried a few things, but wlan still breaks down

to answer the question above when I take back the configuration wireless-power off in /etc/network/interfaces and reboot and check the settings with iwconfig the power management ist turned back on so by default I wouldn't say that this is diasbled so I will leave the configuration

I tried that with kernel 4.1.19 and now also with kernel 4.1.20 ... no change

when the wlan crashed and i try to turn it back on with ifdown and ifup wlan0 I get this: Error for wireless request "Set Power Management" (8B2C) : SET failed on device wlan0 ; Invalid exchange.

I also got a few more error in syslog:

dhcpcd[532]: wlan0: xxx: expired option 25

Mar 21 17:29:35 raspberrypi kernel: [ 6627.337503] brcmfmac: _brcmf_set_multicast_list: Setting mcast_list failed, -52 Mar 21 17:29:36 raspberrypi wpa_supplicant[6318]: Successfully initialized wpa_supplicant Mar 21 17:29:36 raspberrypi dhcpcd[532]: wlan0: carrier lost

Mar 21 17:29:43 raspberrypi kernel: [ 6635.337616] brcmfmac: _brcmf_set_multicast_list: Setting mcast_list failed, -52

Mar 21 17:29:45 raspberrypi kernel: [ 6637.337588] brcmfmac: brcmf_do_escan: error (-52) Mar 21 17:29:45 raspberrypi kernel: [ 6637.337602] brcmfmac: brcmf_cfg80211_scan: scan error (-52)

Mar 21 17:29:47 raspberrypi kernel: [ 6639.337596] brcmfmac: _brcmf_set_multicast_list: Setting allmulti failed, -52 Mar 21 17:29:49 raspberrypi kernel: [ 6641.337632] brcmfmac: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -52

is there anything else I could try?

[dh-connect]

also these:

Mar 21 21:26:55 raspberrypi dhcpcd[526]: wlan0: xxx: expired option 25 Mar 21 21:28:54 raspberrypi kernel: [ 1958.899715] brcmfmac: brcmf_sdio_hostmail: Unknown mailbox data content: 0x40012 Mar 21 21:30:16 raspberrypi dhcpcd[526]: wlan0: xxx is unreachable, expiring it

[pelwell]

I'm not surprised that iwconfig thinks the device has power-saving enabled - I blocked it within the driver itself, and either the state is saved in the higher layers or there is another change required in order to report it correctly. Either way, the evidence is strong that we have avoided the power-saving bugs, but some other problems still remain.

Do you have any rough figures for the time-to-failure and roughly how much data might have been transferred (from ifconfig)?

[dh-connect]

yes I do, when I have just the webserver running with not much traffic (less than 100 MB) it lasts a day or two, when i transfer large data files like 1 GB wlan crashes within 1 hour

[dh-connect]

anything I can provide to help to find the bug?

here are some error from syslog:

Mar 29 14:20:56 raspberrypi dhcpcd[535]: wlan0: xxx: expired option 25 Mar 29 14:30:15 raspberrypi dhcpcd[535]: wlan0: xxx is unreachable, expiring it Mar 29 17:18:42 raspberrypi kernel: [186148.102420] brcmfmac: brcmf_sdio_bus_txdata: out of bus->txq !!! Mar 29 17:18:43 raspberrypi kernel: [186149.101045] brcmfmac: brcmf_sdio_bus_txdata: out of bus->txq !!! Mar 29 17:18:43 raspberrypi kernel: [186149.101145] brcmfmac: brcmf_sdio_bus_txdata: out of bus->txq !!! Mar 29 17:18:44 raspberrypi kernel: [186150.101209] brcmfmac: brcmf_sdio_bus_txdata: out of bus->txq !!! Mar 29 17:18:50 raspberrypi wpa_supplicant[478]: wlan0: CTRL-EVENT-DISCONNECTED bssid=xxx reason=3 locally_generated=1 Mar 29 17:18:50 raspberrypi kernel: [186156.181033] brcmfmac: brcmf_cfg80211_disconnect: error (-52) Mar 29 17:18:52 raspberrypi kernel: [186158.181028] brcmfmac: send_key_to_dongle: wsec_key error (-52) Mar 29 17:18:54 raspberrypi kernel: [186160.181046] brcmfmac: send_key_to_dongle: wsec_key error (-52) Mar 29 17:18:56 raspberrypi kernel: [186162.181048] brcmfmac: send_key_to_dongle: wsec_key error (-52) Mar 29 17:18:58 raspberrypi kernel: [186164.181049] brcmfmac: send_key_to_dongle: wsec_key error (-52) Mar 29 17:18:58 raspberrypi kernel: [186164.185477] cfg80211: Calling CRDA to update world regulatory domain Mar 29 17:18:58 raspberrypi dhcpcd[535]: wlan0: carrier lost Mar 29 17:18:58 raspberrypi wpa_supplicant[7354]: Successfully initialized wpa_supplicant Mar 29 17:18:58 raspberrypi kernel: [186164.314511] brcmfmac: brcmf_cfg80211_reg_notifier: not a ISO3166 code Mar 29 17:18:58 raspberrypi kernel: [186164.314541] cfg80211: World regulatory domain updated: Mar 29 17:18:58 raspberrypi kernel: [186164.314548] cfg80211: DFS Master region: unset Mar 29 17:18:58 raspberrypi kernel: [186164.314555] cfg80211: (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp), (dfs_cac_time) Mar 29 17:18:58 raspberrypi kernel: [186164.314565] cfg80211: (2402000 KHz - 2472000 KHz @ 40000 KHz), (N/A, 2000 mBm), (N/A) Mar 29 17:18:58 raspberrypi kernel: [186164.314573] cfg80211: (2457000 KHz - 2482000 KHz @ 40000 KHz), (N/A, 2000 mBm), (N/A) Mar 29 17:18:58 raspberrypi kernel: [186164.314581] cfg80211: (2474000 KHz - 2494000 KHz @ 20000 KHz), (N/A, 2000 mBm), (N/A) Mar 29 17:18:58 raspberrypi kernel: [186164.314592] cfg80211: (5170000 KHz - 5250000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (N/A) Mar 29 17:18:58 raspberrypi kernel: [186164.314602] cfg80211: (5250000 KHz - 5330000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (0 s) Mar 29 17:18:58 raspberrypi kernel: [186164.314611] cfg80211: (5490000 KHz - 5730000 KHz @ 160000 KHz), (N/A, 2000 mBm), (0 s) Mar 29 17:18:58 raspberrypi kernel: [186164.314645] cfg80211: (5735000 KHz - 5835000 KHz @ 80000 KHz), (N/A, 2000 mBm), (N/A) Mar 29 17:18:58 raspberrypi kernel: [186164.314654] cfg80211: (57240000 KHz - 63720000 KHz @ 2160000 KHz), (N/A, 0 mBm), (N/A)

[pelwell]

Thanks for the offer, but this is in the hands of Broadcom now.

[juched78]

Any update from Broadcom if this is a bug which will be fixed? I now have a cron job setup to bring down and up wlan0 when it fails to ping.

[dh-connect]

quick update from my side, i could get the problem fixed seems to be driver related, i installed Ubuntu MATE 16.04 with kernel 4.4.8 and haven't had any problems with wifi since

i mean they advertise is: "Ubuntu MATE 16.04 also has fully working Bluetooth and Wifi on the Raspberry Pi 3" which seems true

maybe it also works with a new Debian release, which i can not tell

[pelwell]

@juched78 Are you running a 4.4 kernel? If not, please run sudo rpi-update to get the latest 4.4.8 build and see if that suffers the same problem.

The Broadcom drivers have changed significantly since 4.1, and our 4.4 tree includes back-ports of some fixes that went into 4.5. I'm not aware of any outstanding bugs apart from the failure to wake from sleep (power management is still disabled) - channels 12 & 13 are usable where permitted, and Ad Hoc mode doesn't crash - but there may still be lurking issues.

[pelwell]

Oh, there is one reported bug still in 4.4.8 - apparently heavy use of hostapd can lead to a kernel warning (see https://github.com/raspberrypi/linux/issues/1375).

[juched78]

I am running: Linux XXX 4.4.8-v7+ #880 SMP Fri Apr 22 21:55:04 BST 2016 armv7l GNU/Linux

Apr 27 2016 11:06:18 Copyright (c) 2012 Broadcom version 9b52ab7b475f4a056658fd2d95d2440b32167390 (clean) (release)

With my Netgear R7000 running Shibby Tomato, around 2 days in the wifi drops, and in the sys logs I see:

CTRL-EVENT-DISCONNECTED
brcmfmac: brcmf_link_down: WLC_DISASSOC failed (-52)
brcmfmac: send_key_to_dongle: wsec_key error (-52)
...
brcmfmac: brcmf_do_escan: error (-52)
...
wpa_supplicant[506]: wlan0: CTRL-EVENT-REGDOM-CHANGE init=CORE type=WORLD
...
brcmfmac: brcmf_cfg80211_reg_notifier: not a ISO3166 code

(then I see it scan and re-pick my country code CA)

brcmfmac: _brcmf_set_multicast_list: Setting allmulti failed, -52
brcmfmac: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -52
brcmfmac: _brcmf_set_multicast_list: Setting mcast_list failed, -52

Then it seems to never reconnect...

Using sudo ifdown wlan0 followed by sudo ifup wlan0 brings back my connection.

[juched78]

Just upgraded to: Linux JuchePi 4.4.8-v7+ #881 SMP Sat Apr 30 12:16:50 BST 2016 armv7l GNU/Linux

Not sure what is all different from the 22nd to the 30th. I will monitor the connection.

[maciex]

My RPi 3 also hit that problem. I got few different kernel messages. Mainly one of those below. After that I can' get the WiFi working, bringing wlan0 down then up does not help.

May 09 21:24:25 osmc kernel: brcmfmac: brcmf_sdio_hostmail: Unknown mailbox data content: 0x40012
May 09 22:00:15 osmc kernel: brcmfmac: brcmf_sdio_bus_txdata: out of bus->txq !!!
May 09 22:00:18 osmc kernel: brcmfmac: brcmf_sdio_bus_txdata: out of bus->txq !!!

May 10 00:51:10 osmc kernel: brcmfmac: brcmf_cfg80211_get_tx_power: error (-52)
May 10 00:51:12 osmc kernel: brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110

May 10 00:53:16 osmc kernel: brcmfmac: brcmf_do_escan: error (-52)
May 10 00:53:16 osmc kernel: brcmfmac: brcmf_cfg80211_scan: scan error (-52)

Raspberry is powered from original power adapter for version 3. I'm running latest OSMC: $ uname -a Linux osmc 4.4.8-3-osmc #1 SMP PREEMPT Sun May 1 18:57:43 UTC 2016 armv7l GNU/Linux

[juched78]

Still monitoring. I had openhab go offline after running 3 days but for some reason I could still ssh into the Pi which I usually couldn't. The top of the hour and the wifi script ran to bring down and bring up the connection and then it reconnected to my openhab org. Odd. Will keep watching.

[duncanmcdowell]

I am also experiencing the same issue - dmesg trace as follows:

send_key_to_dongle: wsec_key error (-52)
brcmf_cfg80211_del_station: SCB_DEAUTHENTICATE_FOR_REASON failed -52
brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
brcmf_cfg80211_get_tx_power: error (-52)

Usage:

rp3 is being used as a router/access point

Connectivity length seems random - I've had as high as two weeks, and as bad as a few minutes. Lately it's been going out every 20 minutes or so. Bringing wlan0 down and back up does not resolve the issue - a full reboot is required.

Problem seems to be exacerbated while streaming Netflix from my AppleTV. Though this was not the case when I had the two weeks of uptime.

I'm on 4.4.10-v7+

[maciex]

I switched the channel from 13 to 6 to check if that could be the problem (there were some defects about the high channels) and since then I haven't had a WiFi freez. But that could be a coincidence...

[maciex]

Changing access point channels didn't help. WiFi still breaks. Last few times I had to restart few times in a row to get it working.

[loganwendholt]

I experience this issue specifically when I try to do an SFTP transfer between the rpi3 and my Galaxy S5 phone. When I try to perform the same transfer from my laptop, everything runs smoothly.

Running latest kernel from rpi-update:

Linux raspberrypi 4.4.11-v7+ #888 SMP Mon May 23 20:10:33 BST 2016 armv7l GNU/Linux

Error message from syslog:

May 29 18:10:46 raspberrypi kernel: [  178.605907] brcmfmac: brcmf_sdio_hostmail: Unknown mailbox data content: 0x40012

It seems that the only solution after this error is a reboot.

[juched78]

I have had mine drop off the network twice in the past week. First time I was in a rush so just unplugged and rebooted. Few days later it happened again, rebooted again and then ran full system updates (including firmware) and will monitor. Have it mounted with no monitor near by, so getting details on the error needs more effort :)

[xtecox]

Same problem here. It always freeze when transfer big files by sftp. Just rebooting to solve

[BenoitSvB]

This issue maybe related to https://github.com/raspberrypi/linux/issues/1313

[pelwell]

Broadcom say that #1313 is a non-issue, and the latest kernel no longer shows those messages.

I've been unable to reproduce this problem. Has anybody been able to capture a packet trace around the time of failure?

[pelwell]

If anyone has time to do some more testing with a debug-enabled driver module it would be much appreciated:

1) Run sudo rpi-update and reboot. This is to get your kernel up to the same level as mine so that the module is compatible.

2) Download and install the updated driver module:

BRCM80211=/lib/modules/`uname -r`/kernel/drivers/net/wireless/brcm80211
BRCMFMAC=$BRCM80211/brcmfmac
wget -O brcmfmac.ko "https://docs.google.com/uc?authuser=0&id=0B8VsfKAD4-NOR1ZxWS00ZmFrR1k&export=download"
wget -O brcmutil.ko "https://docs.google.com/uc?authuser=0&id=0B8VsfKAD4-NOM0ZDd3FvYUNwZXc&export=download"
sudo mv $BRCMFMAC/brcmfmac.ko{,.orig}
sudo cp brcmfmac.ko $BRCMFMAC
sudo sh -c "echo options brcmfmac debug=0x100000 > /etc/modprobe.d/brcmfmac.conf"
BRCMUTIL=$BRCM80211/brcmutil
sudo mv $BRCMUTIL/brcmutil.ko{,.orig}
sudo cp brcmutil.ko $BRCMUTIL/brcmutil.ko

Reboot to activate the new modules.

3) Use your Pi as normal, then if your WiFi freezes run:

dmesg > wifi_freeze.txt

and upload it to your favourite pasting site (or create a Gist). One or two logs should be plenty.

To restore the original version of the module:

BRCM80211=/lib/modules/`uname -r`/kernel/drivers/net/wireless/brcm80211
sudo mv $BRCM80211/brcmfmac/brcmfmac.ko{.orig,}
sudo mv $BRCM80211/brcmutil/brcmutil.ko{.orig,}

Thanks in advance.

[pelwell]

Hang on for a moment while we verify that the debug output really is enabled.

[pelwell]

You will also need to enable a debug feature on the driver:

sudo sh -c "echo options brcmfmac debug=0x100000 > /etc/modprobe.d/brcmfmac.conf"

I've amended the instructions above.

After a reboot your dmesg output should include something like this:

[   10.848903] brcmfmac: CONSOLE: hndarm_armr addr: 0x18003000, cr4_idx: 0
[   10.860475] brcmfmac: CONSOLE: 000000.001
[   10.869471] brcmfmac: CONSOLE: RTE (SDIO-CDC) 7.45.41.26 (r640327) on BCM43430 r1 @ 37.4/81.6/81.6MHz
[   10.883644] brcmfmac: CONSOLE: 000000.001 sdpcmdcdc0: Broadcom SDPCMD CDC driver
[   10.896090] brcmfmac: CONSOLE: 000000.005 reclaim section 0: Returned 47716 bytes to the heap
[   10.909734] brcmfmac: CONSOLE: 000000.007 wlc_bmac_info_init: host_enab 1
[   10.921417] brcmfmac: CONSOLE: 000000.026 wl0: Broadcom BCM43430 802.11 Wireless Controller 7.45.41.26 (r640327)
[   10.936777] brcmfmac: CONSOLE: 000000.027 TCAM: 256 used: 179 exceed:0
[   10.936794] brcmfmac: CONSOLE: 000000.028 reclaim section 1: Returned 81268 bytes to the heap
[   10.936803] brcmfmac: CONSOLE: 000000.029 sdpcmd_dpc: Enable
[   10.938242] brcmfmac: brcmf_c_preinit_dcmds: Firmware version = wl0: May 27 2016 00:13:38 version 7.45.41.26 (r640327) FWID 01-df77e4a7
[   10.949404] brcmfmac: CONSOLE: 000000.125 wl0: wlc_enable_probe_req: state down, deferring setting of host flags
[   10.963663] brcmfmac: brcmf_cfg80211_reg_notifier: not a ISO3166 code
[   10.969865] brcmfmac: CONSOLE: 000000.150 wl0: wlc_enable_probe_req: state down, deferring setting of host flags
[   10.969876] brcmfmac: CONSOLE: 000000.151 wl0: wlc_enable_probe_req: state down, deferring setting of host flags
[   11.189639] brcmfmac: CONSOLE: 000000.368 wl0: wl_open

[BenoitSvB]

@pelwell after executing your instructions I no longer have wifi...

root@pi3b:/home/pi# dmesg | grep brcmf [ 15.582665] brcmfmac: Unknown symbol brcmu_dbg_hex_dump (err 0) [ 15.613709] brcmfmac: Unknown symbol brcmu_dbg_hex_dump (err 0)

[pelwell]

Try this:

BRCMUTIL=/lib/modules/`uname -r`/kernel/drivers/net/wireless/brcm80211/brcmutil
wget -O brcmutil.ko "https://docs.google.com/uc?authuser=0&id=0B8VsfKAD4-NOM0ZDd3FvYUNwZXc&export=download"
sudo mv $BRCMUTIL/brcmutil.ko{,.orig}
sudo cp brcmutil.ko $BRCMUTIL

And reboot.

[BenoitSvB]

wlan0 does not associate. wireless.txt (in one of many reboots I saw an association for a few minutes though, not catched it (yet) in dmesg)

[loganwendholt]

Seems like the issue may have been resolved for me by upgrading from 4.4.11-v7+ to 4.4.15-v7+

I tried to recreate the problems I was having with SFTP transfers from an Android phone, but I'm not seeing any problems as of right now.

[BenoitSvB]

@pelwell after a long wait wlan0 succeeded to associate; appended dmesg to previous log: wireless.txt waiting now for freeze or association-loss hope this is helpfull

[BenoitSvB]

@pelwell quickly lost connection again; appended dmesg to: wireless.txt

[pelwell]

Thank you. It was slow for me the first time. I've been busy getting a clean Raspbian and applying the patches to try to reproduce the problem - I'll continue anyway.

[BenoitSvB]

@pelwell wireless.txt and reassociated again: appended dmesg again do you want me to continue?

[BenoitSvB]

@pelwell :lost association again wireless_associationloss.txt

[BenoitSvB]

@pelwell it is switching on/off irregularly wireless_associationloss.txt

[pelwell]

I think you'd better switch back now before my inbox overflows.

[BenoitSvB]

ok; I will revert to my €3 MT7601U dongle. ;)