search

raspberrypi / linux

Kernel source tree for Raspberry Pi-provided kernel builds. Issues unrelated to the linux kernel should be posted on the community forum at https://forums.raspberrypi.com/
Other
11.01k stars 4.95k forks source link

wlan freezes in raspberry pi 3B+ #2453

Open ghost opened 6 years ago

ghost commented 6 years ago

(see also https://github.com/raspberrypi/linux/issues/1342 )

I've also got that problem with wifi dying.

Mar 17 18:25:28 hassass kernel: [10279.186321] brcmfmac: brcmf_sdio_hostmail: Unknown mailbox data content: 0x40012
Mar 17 18:25:30 hassass kernel: [10281.665090] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout
Mar 17 18:25:30 hassass kernel: [10281.665622] brcmfmac: brcmf_sdio_checkdied: firmware trap in dongle
Mar 17 18:25:30 hassass kernel: [10281.665638] brcmfmac: brcmf_run_escan: error (-110)
Mar 17 18:25:30 hassass kernel: [10281.665647] brcmfmac: brcmf_cfg80211_scan: scan error (-110)
Mar 17 18:26:30 hassass kernel: [10341.665866] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout

This is with 4.14.27-v7+ and with /sbin/iw dev wlan0 set power_save off /sbin/ifconfig wlan0 promisc in /etc/rc.local.

[    4.112717] brcmfmac: F1 signature read @0x18000000=0x15264345
[    4.119827] brcmfmac: brcmf_fw_map_chip_to_name: using
brcm/brcmfmac43455-sdio.bin for chip 0x004345(17221) rev 0x000006
[    4.120314] usbcore: registered new interface driver brcmfmac
[    4.440371] brcmfmac: brcmf_c_preinit_dcmds: Firmware version = wl0: Feb
27 2018 03:15:32 version 7.45.154 (r684107 CY) FWID 01-4fbe0b04
[    4.440958] brcmfmac: brcmf_c_preinit_dcmds: CLM version = API: 12.2
Data: 9.10.105 Compiler: 1.29.4 ClmImport: 1.36.3 Creation: 2018-03-09
18:56:28
[   10.911757] brcmfmac: power management disabled
[   12.016088] brcmfmac: power management disabled
[ 2074.090674] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg
failed w/status -5
[ 2074.090687] brcmfmac: brcmf_cfg80211_get_tx_power: error (-5)
[ 2074.090745] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing
to do.
[ 2074.090753] brcmfmac: brcmf_link_down: WLC_DISASSOC failed (-5)
[ 2074.610583] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing
to do.
[ 2074.611992] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing
to do.
[ 2074.613945] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing
to do.
[ 2074.613971] brcmfmac: brcmf_cfg80211_get_channel: chanspec failed (-5)
[ 2074.729716] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing
to do.
[ 2074.729733] brcmfmac: brcmf_cfg80211_reg_notifier: Country code iovar
returned err = -5
[ 2074.871693] usbcore: deregistering interface driver brcmfmac
[ 2074.929084] brcmfmac: F1 signature read @0x18000000=0x15264345
[ 2074.936897] brcmfmac: brcmf_fw_map_chip_to_name: using
brcm/brcmfmac43455-sdio.bin for chip 0x004345(17221) rev 0x000006
[ 2074.937139] usbcore: registered new interface driver brcmfmac
[ 2075.118180] brcmfmac: brcmf_c_preinit_dcmds: Firmware version = wl0: Feb
27 2018 03:15:32 version 7.45.154 (r684107 CY) FWID 01-4fbe0b04
[ 2075.118706] brcmfmac: brcmf_c_preinit_dcmds: CLM version = API: 12.2
Data: 9.10.105 Compiler: 1.29.4 ClmImport: 1.36.3 Creation: 2018-03-09
18:56:28
[ 2075.215365] brcmfmac: power management disabled
[ 2075.263751] brcmfmac: power management disabled
[ 2085.475001] brcmfmac: power management disabled
[ 2124.380808] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing
to do.
[ 2124.381146] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing
to do.
[ 2124.381156] brcmfmac: brcmf_cfg80211_get_channel: chanspec failed (-5)
[ 2124.622345] usbcore: deregistering interface driver brcmfmac
[ 2124.705432] brcmfmac: F1 signature read @0x18000000=0x15264345
[ 2124.714194] brcmfmac: brcmf_fw_map_chip_to_name: using
brcm/brcmfmac43455-sdio.bin for chip 0x004345(17221) rev 0x000006
[ 2124.716213] usbcore: registered new interface driver brcmfmac
[ 2124.929556] brcmfmac: brcmf_c_preinit_dcmds: Firmware version = wl0: Feb
27 2018 03:15:32 version 7.45.154 (r684107 CY) FWID 01-4fbe0b04
[ 2124.929993] brcmfmac: brcmf_c_preinit_dcmds: CLM version = API: 12.2
Data: 9.10.105 Compiler: 1.29.4 ClmImport: 1.36.3 Creation: 2018-03-09
18:56:28
[ 2125.105218] brcmfmac: power management disabled
[ 2125.150290] brcmfmac: power management disabled
[ 8237.434034] brcmfmac: brcmf_sdio_hostmail: Unknown mailbox data content:
0x40012
[ 8239.890302] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout
[ 8239.890822] brcmfmac: brcmf_sdio_checkdied: firmware trap in dongle
[ 8239.890835] brcmfmac: brcmf_run_escan: error (-110)
[ 8239.890845] brcmfmac: brcmf_cfg80211_scan: scan error (-110)
[ 8254.280425] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg
failed w/status -5
[ 8254.280438] brcmfmac: brcmf_cfg80211_get_tx_power: error (-5)
[ 8254.280491] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing
to do.
[ 8254.280498] brcmfmac: brcmf_link_down: WLC_DISASSOC failed (-5)
[ 8254.800394] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing
to do.
[ 8254.803873] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing
to do.
[ 8254.808353] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing
to do.
[ 8254.808370] brcmfmac: brcmf_cfg80211_get_channel: chanspec failed (-5)
[ 8254.881402] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing
to do.
[ 8254.881420] brcmfmac: brcmf_cfg80211_reg_notifier: Country code iovar
returned err = -5
[ 8255.001550] usbcore: deregistering interface driver brcmfmac
[ 8255.071184] brcmfmac: F1 signature read @0x18000000=0x15264345
[ 8255.077098] brcmfmac: brcmf_fw_map_chip_to_name: using
brcm/brcmfmac43455-sdio.bin for chip 0x004345(17221) rev 0x000006
[ 8255.077348] usbcore: registered new interface driver brcmfmac
[ 8257.730418] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout
[ 8257.751038] brcmfmac: brcmf_c_get_clm_name: retrieving revision info
failed (-110)
[ 8257.751049] brcmfmac: brcmf_c_process_clm_blob: get CLM blob file name
failed (-110)
[ 8257.751068] brcmfmac: brcmf_c_preinit_dcmds: download CLM blob file
failed, -110
[ 8257.751076] brcmfmac: brcmf_bus_started: failed: -110
[ 8257.751114] brcmfmac: brcmf_sdio_firmware_callback: dongle is not
responding
[ 8304.417684] usbcore: deregistering interface driver brcmfmac
[ 8304.486099] brcmfmac: F1 signature read @0x18000000=0x15264345
[ 8304.493613] brcmfmac: brcmf_fw_map_chip_to_name: using
brcm/brcmfmac43455-sdio.bin for chip 0x004345(17221) rev 0x000006
[ 8304.494078] usbcore: registered new interface driver brcmfmac
[ 8304.686761] brcmfmac: brcmf_c_preinit_dcmds: Firmware version = wl0: Feb
27 2018 03:15:32 version 7.45.154 (r684107 CY) FWID 01-4fbe0b04
[ 8304.687203] brcmfmac: brcmf_c_preinit_dcmds: CLM version = API: 12.2
Data: 9.10.105 Compiler: 1.29.4 ClmImport: 1.36.3 Creation: 2018-03-09
18:56:28
[ 8304.829994] brcmfmac: power management disabled
[ 8304.907662] brcmfmac: power management disabled
[ 8357.441791] brcmfmac: brcmf_sdio_hostmail: Unknown mailbox data content:
0x40012
[ 8359.891146] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout
[ 8359.891655] brcmfmac: brcmf_sdio_checkdied: firmware trap in dongle
[ 8359.891668] brcmfmac: brcmf_run_escan: error (-110)
[ 8359.891677] brcmfmac: brcmf_cfg80211_scan: scan error (-110)
[ 8371.731226] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout
[ 8371.731731] brcmfmac: brcmf_sdio_checkdied: firmware trap in dongle
[ 8371.731746] brcmfmac: brcmf_cfg80211_get_channel: chanspec failed (-110)
[ 8373.941267] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg
failed w/status -5
[ 8373.941280] brcmfmac: brcmf_cfg80211_get_tx_power: error (-5)
[ 8373.941330] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing
to do.
[ 8373.941338] brcmfmac: brcmf_link_down: WLC_DISASSOC failed (-5)
[ 8374.461245] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing
to do.
[ 8374.461942] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing
to do.
[ 8374.463553] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing
to do.
[ 8374.463573] brcmfmac: brcmf_cfg80211_get_channel: chanspec failed (-5)
[ 8374.564729] brcmfmac: brcmf_fil_cmd_data: bus is down. we have nothing
to do.
[ 8374.564750] brcmfmac: brcmf_cfg80211_reg_notifier: Country code iovar
returned err = -5
[ 8374.702401] usbcore: deregistering interface driver brcmfmac
[ 8374.759839] brcmfmac: F1 signature read @0x18000000=0x15264345
[ 8374.767561] brcmfmac: brcmf_fw_map_chip_to_name: using
brcm/brcmfmac43455-sdio.bin for chip 0x004345(17221) rev 0x000006
[ 8374.771137] usbcore: registered new interface driver brcmfmac
[ 8377.411255] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout
[ 8377.431924] brcmfmac: brcmf_c_get_clm_name: retrieving revision info
failed (-110)
[ 8377.431934] brcmfmac: brcmf_c_process_clm_blob: get CLM blob file name
failed (-110)
[ 8377.431941] brcmfmac: brcmf_c_preinit_dcmds: download CLM blob file
failed, -110
[ 8377.431949] brcmfmac: brcmf_bus_started: failed: -110
[ 8377.432003] brcmfmac: brcmf_sdio_firmware_callback: dongle is not
responding
[ 8424.133114] usbcore: deregistering interface driver brcmfmac
[ 8424.229631] brcmfmac: F1 signature read @0x18000000=0x15264345
[ 8424.237210] brcmfmac: brcmf_fw_map_chip_to_name: using
brcm/brcmfmac43455-sdio.bin for chip 0x004345(17221) rev 0x000006
[ 8424.239352] usbcore: registered new interface driver brcmfmac
[ 8424.460736] brcmfmac: brcmf_c_preinit_dcmds: Firmware version = wl0: Feb
27 2018 03:15:32 version 7.45.154 (r684107 CY) FWID 01-4fbe0b04
[ 8424.461174] brcmfmac: brcmf_c_preinit_dcmds: CLM version = API: 12.2
Data: 9.10.105 Compiler: 1.29.4 ClmImport: 1.36.3 Creation: 2018-03-09
18:56:28
[ 8424.646993] brcmfmac: power management disabled
[ 8424.708633] brcmfmac: power management disabled
FrancisHGR commented 4 years ago

I checked power supply, everything alright there, SD card could be source of problem though its strange that it always throws the same problem. HW chip could also a be a problem, the long issue list of people having the exact same problem in this thread leaves me curious though...

FrancisHGR commented 4 years ago

Quick update, maybe helps for finding the problem:

I wrote a one-liner cronjob script that pings my router every 3 min and if hes not reachable makes a restart. Since this script is running, strangely no wifi breakdown has happened...

/3 * ping 192.168.1.1 | grep -q Unreachable; sudo shutdown -r now

Update: Wifi breakdown happens, but much less frequent (may 2 times per day). But... additionally my bluetooth breaks down after a certain uptime of the system... is the wifi and bluetooth on the same chip or runs the same firmware?

kernelpanic19 commented 4 years ago

I have a 3b+ and two RPI 4s running Buster that all exhibit the issue. I installed the updated Cypress firmware on the 3b+. Still happens. However I have a suspicion about what causes the issues. If I place the RPIs in close proximity to my AP, they don't have the issue. When i have them on the other side of the house, it happens. I am wondering if the firmware bugs out with a weak connection. I am curious if others having issues have a similar setup and can test by moving their RPI closer to their AP.

lolz3r commented 4 years ago

Is there any update on this? I have the same issue when using my 3B+ (buster and latest firmware) as a hotspot with hostapd, tried to update everything (including the wireless firmware) and nothing changed. The only "fix" is to periodically reboot the system, but that's not a solution!

kernelpanic19 commented 4 years ago

After several days of testing, I can confirm that the issue only occurs with low signal strength. Did anyone else with the brcmfmac firmware halt issue test moving their AP and RPi closer? You can compare before/after signal like this: pi@pidp-11:/opt/pidp11 $ iwconfig wlan0 | egrep 'Rate|Link' Bit Rate=433.3 Mb/s Tx-Power=31 dBm
Link Quality=68/70 Signal level=-42 dBm

FrancisHGR commented 4 years ago

Quick update, maybe helps for finding the problem:

I wrote a one-liner cronjob script that pings my router every 3 min and if hes not reachable makes a restart. Since this script is running, strangely no wifi breakdown has happened...

/3 * ping 192.168.1.1 | grep -q Unreachable; sudo shutdown -r now

Update: Wifi breakdown happens, but much less frequent (may 2 times per day). But... additionally my bluetooth breaks down after a certain uptime of the system... is the wifi and bluetooth on the same chip or runs the same firmware?

Was able to further narrow the problem down due to my own carelessness. I was aiming to send a ping every 3 mins to my router, however by mistake did it so continously. So my raspberry stayed up on average for maybe 1 full day.

After realizing my error, I changed the cronjob line to just a single ping every 3 mins. And now, my Raspberry's wifi fails within maybe 30 secs after reboot.

So I can deduct, a constant connection check with the router will keep the wifi alive...

Batoo76 commented 4 years ago

Hi! I also face the bug. If the ping workaround works, maybe other ways also do to "shake-up" the driver and avoïd a deadlock? I can try this, I'm on a RPi4, kali, nexmon driver.

best!

FYI

Source mon0 has encountered an error. Kismet will attempt to re-open the source in 5 seconds. (1 failures) INFO: Attempting to re-open source mon0 INFO: mon0 interface 'mon0' looks like it is a Broadcom binary driver found in the Raspberry Pi and some Android devices; this will ONLY work with the nexmon patches INFO: mon0 interface 'mon0' is already in monitor mode ALERT: SOURCEOPEN Source mon0 successfully re-opened

No more messages viewed despite the "successfully re-opened" message.

pelwell commented 4 years ago

We don't support the nexmon patches.

Batoo76 commented 4 years ago

We don't support the nexmon patches.

Thanks, but seems the bug exist on both branches, so if there is a way for the standard driver, maybe it applies to the nexmon, and work myself on it from this info.

pelwell commented 4 years ago

Thanks, but seems the bug exist on both branches, so if there is a way for the standard driver, maybe it applies to the nexmon, and work myself on it from this info.

Yes, it's probably common.

Fatflo commented 4 years ago

I am having the same issue with raspbian_2018_10_11, it appears randomly once per week on one of my 10 connected pi zero W

figeac888 commented 4 years ago

Hello there,

Same issue with rpi 4 raspbian buster latest firmware / kernel Happens almost everyday

Linux USA 4.19.95-v7l+ #1291 SMP Tue Jan 14 14:54:26 GMT 2020 armv7l GNU/Linux

/opt/vc/bin/vcgencmd version Jan 14 2020 15:01:27 Copyright (c) 2012 Broadcom version 8da878341f0c4a12f73d227c0c5b6ca99219292e (clean) (release) (start)

[76229.046423] brcmfmac: brcmf_sdio_hostmail: mailbox indicates firmware halted [76261.584586] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76264.144636] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76264.144650] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [76275.984804] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76278.544850] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76278.544863] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [76292.865062] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76295.425149] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76295.425162] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [76310.385333] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76312.945373] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76312.945386] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [76336.625740] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76339.185805] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76339.185818] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [76348.945941] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76351.505972] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76351.505985] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [76354.066015] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76356.626053] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76356.626066] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [76365.106219] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76367.666231] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76367.666246] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [76373.666348] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76376.226422] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76376.226437] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [76378.786455] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76381.346478] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76381.346493] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [76383.906532] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76386.466535] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [76386.466550] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110

brubbel commented 4 years ago

FYI https://community.cypress.com/docs/DOC-19000 There is something moving for hostapd.

pelwell commented 4 years ago

Thanks - we are aware.

sub0ne commented 4 years ago

seems to be the same issue like my https://github.com/raspberrypi/linux/issues/3258

I don't know how this think can be used that much if wifi is that instable...I am switching to a wired connection now...

tunip3 commented 4 years ago

this is also an issue on on the pi 4 with ubuntu server

itspa1 commented 4 years ago

Hey, I went through the whole conversation, I'm also getting the same errors on my raspberry pi 3b+ while running monitor mode for a long time, it freezes and I get the same mailbox indicates firmware halted and firmware trap in dongle.

pelwell commented 4 years ago

Our firmwares do not support monitor mode. We do not support the use of hacked firmware.

itspa1 commented 4 years ago

@pelwell so does that mean, I'll need to use an external WiFi adapter to achieve this?

pelwell commented 4 years ago

Yes, if it is monitor mode that triggers the problem. You're not violating any warranty by running modified firmware, but we won't investigate any issues that you encounter while doing so.

itspa1 commented 4 years ago

@pelwell thanks for the reply

vintozver commented 4 years ago

same with my pi3b+ & pi4b, happening on all APs

klodner commented 4 years ago

I have the same issue, currently on 3B+, all updated, in AP mode via hostapd, it can happen several times a day, keeps running like 1 day maximum.

SSID diappears, and hostapd cannot be restarted, dmesg below, modprobe brcmfmac gives the same errors

[ 2739.587819] brcmfmac: brcmf_sdio_hostmail: mailbox indicates firmware halted [ 2809.676128] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [ 2812.236156] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [ 2812.236167] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [ 2829.196608] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [ 2831.756403] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [ 2831.756415] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [ 3042.238695] brcmfmac: brcmf_cfg80211_del_station: SCB_DEAUTHENTICATE_FOR_REASON failed -110 [ 3043.198684] brcmfmac: brcmf_netdev_wait_pend8021x: Timed out waiting for no pending 802.1x packets [ 3045.758742] brcmfmac: send_key_to_dongle: wsec_key error (-110) [ 3048.318812] brcmfmac: brcmf_cfg80211_change_station: Setting SCB (de-)authorize failed, -110 [ 3049.278717] brcmfmac: brcmf_netdev_wait_pend8021x: Timed out waiting for no pending 802.1x packets [ 3051.838759] brcmfmac: send_key_to_dongle: wsec_key error (-110) [ 3054.398785] brcmfmac: brcmf_cfg80211_del_station: SCB_DEAUTHENTICATE_FOR_REASON failed -110 [ 3056.958817] brcmfmac: brcmf_cfg80211_change_station: Setting SCB (de-)authorize failed, -110 [ 3059.518831] brcmfmac: brcmf_cfg80211_del_station: SCB_DEAUTHENTICATE_FOR_REASON failed -110 [ 3062.078866] brcmfmac: brcmf_cfg80211_del_station: SCB_DEAUTHENTICATE_FOR_REASON failed -110

Daniel085 commented 4 years ago

I have the same issue as well. I tried rpi-udate hoping things were better on a newer set of firmware, but alas I can't even get things to stay up 24h.

Daniel085 commented 4 years ago

It might be worth noting that while the behavior is the same, the new firmware (after rpm-update) issues different messages in dmesg: [ 1041.374172] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 4108.221557] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 4108.697417] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 4122.417189] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 4122.785941] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 4293.588764] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 4295.594731] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 4297.598964] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 4299.593303] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 4301.592953] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 5488.683244] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 5489.151684] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 5509.119914] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 5509.635644] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 8197.467768] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 8200.898028] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 8207.141497] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 8210.655801] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 8218.250063] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 8414.262896] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 8419.132416] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 8426.204748] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 8426.689355] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 9719.754335] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 9723.364529] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 9727.034231] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 9729.254992] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 9789.372693] brcmfmac: send_key_to_dongle: wsec_key error (-52) [ 9790.035084] brcmfmac: send_key_to_dongle: wsec_key error (-52) [28482.036896] brcmfmac: send_key_to_dongle: wsec_key error (-52) [28482.626269] brcmfmac: send_key_to_dongle: wsec_key error (-52) [28509.236509] brcmfmac: send_key_to_dongle: wsec_key error (-52) [28509.833906] brcmfmac: send_key_to_dongle: wsec_key error (-52) [28516.072262] brcmfmac: send_key_to_dongle: wsec_key error (-52) [28516.916884] brcmfmac: send_key_to_dongle: wsec_key error (-52) [34233.519020] brcmfmac: send_key_to_dongle: wsec_key error (-52) [34240.705187] brcmfmac: send_key_to_dongle: wsec_key error (-52) [42613.561046] brcmfmac: send_key_to_dongle: wsec_key error (-52) [57066.745807] brcmfmac: send_key_to_dongle: wsec_key error (-52) [57067.109665] brcmfmac: send_key_to_dongle: wsec_key error (-52) [57083.157216] brcmfmac: send_key_to_dongle: wsec_key error (-52) [57083.504258] brcmfmac: send_key_to_dongle: wsec_key error (-52) [59679.081017] brcmfmac: brcmf_vif_set_mgmt_ie: vndr ie set error : -52

magnusahlden commented 4 years ago

I bought a raspberry pi 4 today and installed the latest debian from raspberrys homepage. Got this immediately. Network card completly dead...

pelwell commented 4 years ago

When you say "Got this immediately", what is your use case? Are you running as a client or an AP?

magnusahlden commented 4 years ago

as a client

pelwell commented 4 years ago

What is the AP?

What do you get from running dmesg | grep brcmfmac?

Benzomat commented 4 years ago

FYI https://community.cypress.com/docs/DOC-19000 There is something moving for hostapd.

Hi everyone,

about 2 weeks ago I installed upgrades on my rPI Zero Ws and got an old, buggy driver from the repositories. I extracted the driver files from the archive cypress-firmware-v4.14.77-2020_0115.tar.gz in the archive cypress-fmac-v4.14.77-2020_0115.zip of the above mentioned link and replaced the needed files:

/lib/firmware/brcm/brcmfmac43430-sdio.bin
/lib/firmware/brcm/brcmfmac43430-sdio.clm_blob

After that the wifi on my Zero Ws started working again. I haven't had any problems so far. Make sure to replace the ones needed by your Pi model which might need different ones from the archive.

Btw. there is a new version available (which I haven't tested yet): https://community.cypress.com/docs/DOC-19375

VictorvandenHoven commented 4 years ago

FYI https://community.cypress.com/docs/DOC-19000 There is something moving for hostapd.

Hi everyone,

about 2 weeks ago I installed upgrades on my rPI Zero Ws and got an old, buggy driver from the repositories. I extracted the driver files from the archive cypress-firmware-v4.14.77-2020_0115.tar.gz in the archive cypress-fmac-v4.14.77-2020_0115.zip of the above mentioned link and replaced the needed files:

/lib/firmware/brcm/brcmfmac43430-sdio.bin
/lib/firmware/brcm/brcmfmac43430-sdio.clm_blob

After that the wifi on my Zero Ws started working again. I haven't had any problems so far. Make sure to replace the ones needed by your Pi model which might need different ones from the archive.

Btw. there is a new version available (which I haven't tested yet): https://community.cypress.com/docs/DOC-19375

Hi, Would like to try the same on my rpi 3+

I suppose that I need to replace the brcm/brcmfmac43455-sdio files from the archive? Linux iotpi 4.19.97-v7+ #1294 SMP Thu Jan 30 13:15:58 GMT 2020 armv7l GNU/Linux

root@iotpi:/var/log# dmesg | grep brcmfmac [ 5.497091] brcmfmac: F1 signature read @0x18000000=0x15264345 [ 5.506173] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6 [ 5.507097] usbcore: registered new interface driver brcmfmac [ 5.740621] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6 [ 5.791383] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Mar 2 2020 23:30:41 version 7.45.202 (r724630 CY) FWID 01-72f6ece2 [ 7.929123] brcmfmac: power management disabled

Kind regards

VictorvandenHoven commented 4 years ago

So I changed the two files:

5.791383] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Mar 2 2020 23:30:41 version 7.45.202 (r724630 CY) FWID 01-72f6ece2

vs

6.051870] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Nov 13 2019 04:39:05 version 7.45.198 (r723338 CY) FWID 01-369a6798

And after the first 24h the wifi still works!!!!

VictorvandenHoven commented 4 years ago

Maybe it is too early to say that it worked! But, so far, so good!

kcem commented 4 years ago

Here you have workaround:


#!/usr/bin/env bash

function handle {
    echo "Unload brcmfmac"
    rmmod brcmfmac
    echo "Load brcmfmac"
    modprobe brcmfmac
}

function handle_exit {
    echo "Stop monitoring..."
}

trap handle_exit EXIT

echo "Start monitoring..."
TS="0"
while read line; do
    CUR="$(date +%s)"
    if [[ "$((CUR-TS))" -le 5 ]]; then
        continue
    fi
    TS="$CUR"
    handle
done < <(journalctl -f | grep --line-buffered 'ieee80211 .*\-110')
VictorvandenHoven commented 4 years ago

Nice Will try if problems come back.

brubbel commented 4 years ago

Here you have workaround:

Problem is that this only reloads the kernel module, while most problems are due to firmware crashes on the BCM chipset. Sadly, this requires a full restart of the chipset (unbind/bind cycle).

kcem commented 4 years ago

This works for me. I have Rock Pi 4 and Manjaro. After module reload WiFI start again. I'll remove this patch when firmware will be fixed.

I think after this upgrade: linux-firmware (20200204.b791e15-1 -> 20200224.efcfa03-1) I have -110 error less than once a week. Ealier this error happened more than once per day.

dadav commented 4 years ago

This works for me. I have Rock Pi 4 and Manjaro. After module reload WiFI start again. I'll remove this patch when firmware will be fixed.

I think after this upgrade: linux-firmware (20200204.b791e15-1 -> 20200224.efcfa03-1) I have -110 error less than once a week. Ealier this error happened more than once per day.

Your script works for me too. raspi zero w (4.19.93-Re4son+)

CanalWood commented 4 years ago

Hello, I've the same issue on my Rasp Pi4. I would like to do a Hotspot with RaspAP but the wifi crach if there are more than 1 Computer on this AP. I've lost many day on this pb!

I've replace 2 files and actually all is good : Cypress DOC-19000 brcmfmac43455-sdio.clm_blob brcmfmac43455-sdio.bin

But I don't test the new release : Cypress DOC-19375

Before release : Firmware: BCM4345/6 wl0: Mar 2 2020 23:30:41 version 7.45.202 (r724630 CY) FWID 01-72f6ece2 After release : Firmware: BCM4345/6 wl0: Nov 13 2019 04:39:05 version 7.45.198 (r723338 CY) FWID 01-369a6798

Thanks for all!

Bedore I've this error :

[ 197.008483] brcmfmac: brcmf_sdio_hostmail: mailbox indicates firmware halted [ 404.368443] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [ 406.926798] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [ 406.926813] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [ 459.940513] brcmfmac: brcmf_cfg80211_del_station: SCB_DEAUTHENTICATE_FOR_REASON failed -110 [ 462.978918] brcmfmac: brcmf_cfg80211_stop_ap: SET SSID error (-110) [ 465.537713] brcmfmac: brcmf_cfg80211_stop_ap: BRCMF_C_DOWN error -110 [ 468.096663] brcmfmac: brcmf_cfg80211_stop_ap: setting AP mode failed -110 [ 473.213666] brcmfmac: brcmf_cfg80211_stop_ap: BRCMF_C_UP error -110 [ 475.772642] brcmfmac: brcmf_set_mpc: fail to set mpc [ 483.528729] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [ 486.087228] brcmfmac: _brcmf_set_multicast_list: Setting mcast_list failed, -110 [ 488.646189] brcmfmac: _brcmf_set_multicast_list: Setting allmulti failed, -110 [ 491.214645] brcmfmac: _brcmf_set_multicast_list: Setting mcast_list failed, -110 [ 493.763522] brcmfmac: _brcmf_set_multicast_list: Setting allmulti failed, -110 [ 498.881215] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [ 501.439695] brcmfmac: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -110 [ 506.556996] brcmfmac: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -110 [ 511.674420] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [ 511.674440] brcmfmac: brcmf_cfg80211_get_tx_power: error (-110) [ 519.350710] brcmfmac: _brcmf_set_multicast_list: Setting mcast_list failed, -110 [ 524.468219] brcmfmac: _brcmf_set_multicast_list: Setting allmulti failed, -110 [ 527.027214] brcmfmac: _brcmf_set_multicast_list: Setting BRCMF_C_SET_PROMISC failed, -110 [ 543.898408] brcmfmac: brcmf_cfg80211_disconnect: error (-110) [ 545.947519] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [ 545.947534] brcmfmac: brcmf_cfg80211_get_channel: chanspec failed (-110) [ 545.947569] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [ 545.947580] brcmfmac: brcmf_cfg80211_get_tx_power: error (-110) [ 551.574530] brcmfmac: brcmf_cfg80211_del_ap_iface: interface_remove failed -110 [ 551.583138] brcmfmac: brcmf_cfg80211_add_iface: iface validation failed: err=-16

pelwell commented 4 years ago

We don't support random builds of the Cypress firmware - use sudo apt update; sudo apt upgrade to get the latest Raspberry Pi version (the Mar 02 2020 firmware with a new "nvram" .txt file) and see if the issue still remains.

fuxjezz commented 4 years ago

I think I'm affected as well.

Have been running this PI in "AP-mode" with hostapd. Functions fine, but for some reason it locks up after a couple of hours/days.

The raspbian install has not been "tainted" by either the Cypress firmware or by using rpi-update. The only thing I configured in an attempt to address this issue is the execution of:

iw dev wlan0 set power_save off

after a fresh boot.

What's strange is that unloading / loading the kernel module results in errors on the first try(/tries?). If i retry it a couple of times (as the script shared by kcem perhaps does as well), the module eventually gets reloaded "cleanly" (without errors in dmesg).

This is, however, not an effective workaround for my situation: hostapd could not be restarted because one of the capabilities is not supported anymore (after the rmmod/modprobe). Apparently, after a reboot of the pi, this (-40 ht something) capability magically is usable by hostapd again :S

Before I pursue a strategy in resolving this issue that is not supported (Cypress firmware is rumoured to be effective) , I'm looking into something that is.

Is there anything I can do to troubleshoot / fix this issue which is better supported than using older Cypress firmware?

CanalWood commented 4 years ago

Hello, @pelwell, I've systematically do the update of my Rasp before the installation of RaspAP. I've do this operation many times and there are systematically this error.

I would like to add a comment. After fews days of the driver update, my RaspAP is down randomly. I've switch to this solution to resolve my issue :

fuxjezz commented 4 years ago

Just had this occuring on both my AP's . One of them is an RPI3 and one of them is an RPI4. The dmesg output below (from uptime(11 days ago)/starting hostapd till the incident) is from the RPI4. As with the RPI3 I could not unload/load the brcmfmac module in 1 try, needed 3 retries :S

Any news yet on an "officially approved" remedy? Installing Cypress firmware like ALTopOfPopROck (and others) suggested is tempting...

[ 66.421453] device wlan0 entered promiscuous mode [ 70.625656] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready [ 71.031548] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready [1023248.445128] brcmfmac: brcmf_sdio_hostmail: mailbox indicates firmware halted [1023274.565002] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023277.125004] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023277.125019] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [1023331.845016] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023334.405018] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023334.405032] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [1023353.685006] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023356.245016] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023356.245030] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [1023431.605012] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023434.165013] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023434.165028] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [1023441.125032] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023443.685026] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023443.685041] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [1023543.525005] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023546.085014] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023546.085029] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [1023597.685006] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023600.245007] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023600.245021] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [1023642.965029] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023645.525009] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023645.525025] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [1023660.805004] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023663.365017] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023663.365032] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [1023749.685011] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023752.245013] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023752.245028] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 [1023754.805006] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023757.365003] brcmfmac: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110 [1023757.365018] brcmfmac: brcmf_cfg80211_get_station: GET STA INFO failed, -110 root@wap2:/home/fux# lsmod Module Size Used by bnep 20480 2 hci_uart 40960 1 btbcm 16384 1 hci_uart serdev 20480 1 hci_uart bluetooth 389120 24 hci_uart,bnep,btbcm ecdh_generic 28672 1 bluetooth xt_state 16384 0 ip6table_filter 16384 1 ip6_tables 24576 1 ip6table_filter nfnetlink_log 20480 7 nfnetlink 16384 4 nfnetlink_log xt_NFLOG 16384 4 xt_limit 16384 4 xt_comment 16384 3 xt_tcpudp 16384 2 xt_conntrack 16384 5 iptable_filter 16384 1 iptable_nat 16384 0 nf_nat_ipv4 16384 1 iptable_nat nf_nat 36864 1 nf_nat_ipv4 nf_conntrack 135168 4 xt_state,xt_conntrack,nf_nat_ipv4,nf_nat nf_defrag_ipv6 20480 1 nf_conntrack nf_defrag_ipv4 16384 1 nf_conntrack vc4 176128 0 brcmfmac 311296 0 drm_kms_helper 184320 1 vc4 brcmutil 16384 1 brcmfmac v3d 73728 0 gpu_sched 28672 1 v3d snd_soc_core 192512 1 vc4 snd_compress 20480 1 snd_soc_core snd_pcm_dmaengine 16384 1 snd_soc_core sha256_generic 20480 0 syscopyarea 16384 1 drm_kms_helper drm 442368 5 v3d,vc4,gpu_sched,drm_kms_helper sysfillrect 16384 1 drm_kms_helper sysimgblt 16384 1 drm_kms_helper fb_sys_fops 16384 1 drm_kms_helper drm_panel_orientation_quirks 16384 1 drm cfg80211 647168 1 brcmfmac rfkill 28672 5 bluetooth,cfg80211 snd_bcm2835 24576 1 bcm2835_codec 36864 0 bcm2835_v4l2 45056 0 v4l2_mem2mem 24576 1 bcm2835_codec snd_pcm 102400 4 vc4,snd_pcm_dmaengine,snd_bcm2835,snd_soc_core bcm2835_mmal_vchiq 32768 2 bcm2835_codec,bcm2835_v4l2 v4l2_common 16384 1 bcm2835_v4l2 snd_timer 32768 1 snd_pcm videobuf2_dma_contig 20480 1 bcm2835_codec videobuf2_vmalloc 16384 1 bcm2835_v4l2 snd 73728 7 snd_compress,snd_timer,snd_bcm2835,snd_soc_core,snd_pcm videobuf2_memops 16384 2 videobuf2_dma_contig,videobuf2_vmalloc videobuf2_v4l2 24576 3 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem raspberrypi_hwmon 16384 0 hwmon 16384 1 raspberrypi_hwmon videobuf2_common 45056 4 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2 videodev 200704 6 bcm2835_codec,v4l2_common,videobuf2_common,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2 media 36864 3 bcm2835_codec,videodev,v4l2_mem2mem vc_sm_cma 36864 1 bcm2835_mmal_vchiq rpivid_mem 16384 0 uio_pdrv_genirq 16384 0 uio 20480 1 uio_pdrv_genirq 8021q 32768 0 garp 16384 1 8021q stp 16384 1 garp llc 16384 2 garp,stp ip_tables 24576 8 iptable_filter,iptable_nat x_tables 32768 10 xt_state,xt_comment,ip_tables,iptable_filter,xt_tcpudp,xt_limit,ip6table_filter,xt_conntrack,ip6_tables,xt_NFLOG ipv6 454656 26 root@wap2:/home/fux# rmmod brcmfmac root@wap2:/home/fux# modprobe brcmfmac root@wap2:/home/fux# dmesg | tail [1023832.136041] usbcore: deregistering interface driver brcmfmac [1023834.974741] brcmfmac: F1 signature read @0x18000000=0x15264345 [1023834.982221] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6 [1023834.982582] usbcore: registered new interface driver brcmfmac [1023835.241676] brcmfmac: brcmf_sdio_hostmail: mailbox indicates firmware halted [1023837.604999] brcmfmac: brcmf_sdio_bus_rxctl: resumed on timeout [1023837.605346] brcmfmac: brcmf_sdio_checkdied: firmware trap in dongle [1023837.605364] brcmfmac: brcmf_bus_started: failed: -110 [1023837.605387] brcmfmac: brcmf_attach: dongle is not responding: err=-110 [1023837.645411] brcmfmac: brcmf_sdio_firmware_callback: brcmf_attach failed root@wap2:/home/fux# rmmod brcmfmac root@wap2:/home/fux# modprobe brcmfmac root@wap2:/home/fux# dmesg | tail [1023846.587363] brcmfmac: F1 signature read @0x18000000=0x15264345 [1023846.594040] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6 [1023846.594368] usbcore: registered new interface driver brcmfmac [1023846.699790] brcmfmac: brcmf_sdio_txfail: sdio error, abort command and terminate frame [1023846.699937] brcmfmac: brcmf_sdio_txfail: sdio error, abort command and terminate frame [1023846.700080] brcmfmac: brcmf_sdio_txfail: sdio error, abort command and terminate frame [1023846.700182] brcmfmac: brcmf_sdio_dpc: failed backplane access over SDIO, halting operation [1023846.700194] brcmfmac: brcmf_bus_started: failed: -84 [1023846.700207] brcmfmac: brcmf_attach: dongle is not responding: err=-84 [1023846.745178] brcmfmac: brcmf_sdio_firmware_callback: brcmf_attach failed root@wap2:/home/fux# rmmod brcmfmac root@wap2:/home/fux# modprobe brcmfmac root@wap2:/home/fux# dmesg | tail [1023846.700194] brcmfmac: brcmf_bus_started: failed: -84 [1023846.700207] brcmfmac: brcmf_attach: dongle is not responding: err=-84 [1023846.745178] brcmfmac: brcmf_sdio_firmware_callback: brcmf_attach failed [1023855.282555] usbcore: deregistering interface driver brcmfmac [1023857.168342] brcmfmac: F1 signature read @0x18000000=0x15264345 [1023857.174479] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6 [1023857.174761] usbcore: registered new interface driver brcmfmac [1023857.355910] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6 [1023857.365973] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Mar 2 2020 23:30:41 version 7.45.202 (r724630 CY) FWID 01-72f6ece2 [1023857.455857] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready root@wap2:/home/fux# (at this point I could restart hostapd again)

fuxjezz commented 4 years ago

I have a feeling that after using Slack or MS Teams videoconferencing, this issue surfaces more often. Since - in my case - the restarting of hostapd after unloading/reloading the kernel module does not always work, i've implemented a "reboot pi" script (triggered from cron) that just reboots the pi:

cat /etc/cron.d/reboot_if_hit_by_bug 

* * * * * root /etc/local/reboot.sh
* * * * * root sleep 30 && /etc/local/reboot.sh
cat /etc/local/reboot.sh
#!/bin/sh

firmwareProblem=`dmesg | grep 'GET STA INFO failed,'`;

if [ ! -z "$firmwareProblem" ]
then
    dateStamp=`date '+%s'`;
    dmesg > '/root/'"${dateStamp}"'.error'; 
    /sbin/shutdown -r now
    sleep 20
    /sbin/reboot -f
fi

this works, but is ofc annoying.

I've recently ditched the USB wifi nics that were previously attached to my raspberries and I would very much like to start experiencin stable WIFI using the onboard wifi adapters...

@pelwell : do you perhaps have any pointers to share on how to proceed with debugging this issue?

Any feedback appreciated!

fuxjezz commented 4 years ago

Did some videoconferencing again today, another occurance.

Would it help if i started collecting pcaps @JamesH65 ?

JamesH65 commented 4 years ago

I'm a bit out of touch on Wifi issues at the moment. We have to rely on Cypress fixing any firmware bugs, and that can be time consuming. There have been some recent firmware changes that appears to have been released to others, but they seem very slow supplying it to us. @pelwell Any news on new FW from Cypress?

pelwell commented 4 years ago

As you can see in the log above, we're shipping new firmware:

[1023857.365973] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Mar 2 2020 23:30:41 version 7.45.202 (r724630 CY) FWID 01-72f6ece2

Both 43438 and 43455 firmware was updated with a patch for the "kr00k" vulnerability.

The problem with the log above is the timestamps - we're not going to get any traction with a bug report that:

  1. occurs in a non-standard mode (host AP),
  2. requires external hardware (AP clients),
  3. takes up to 11 days, and
  4. gives different results from user to user (or environment to environment).
fuxjezz commented 4 years ago

@pelwell : Your "1" seems to be sort of a "given", there isn't really anything we can do about that, right?

As for your other points: there do seem to be circumstances that speed up the occurrence of these freezes. Video conferencing - for instance - seems to significantly speed up the rate at which these firmware freezes occur.

Would it be helpful if I were to experiment with a scripted stresstest of some sorts?

JamesH65 commented 4 years ago

Anything that shows the issue in a sane timescale (hours rather than weeks) would clearly be of use to both us and the Cypress engineers. It's always a short replication time that makes an engineer more likely to find an issue.

mweinelt commented 4 years ago

occurs in a non-standard mode (host AP),

That is very sad to hear. I want to contribute my information nonetheless.

I'm running hostapd on an RPi4 with https://github.com/raspberrypi/linux/commit/11da20790279cd3a180d7ee89c0dffab636d972d.

Linux dione 5.6.10 #1-NixOS SMP Thu Jan 1 00:00:01 UTC 1970 aarch64 GNU/Linux

The firmware loaded is visible below, it's https://github.com/rpi-distro/firmware-nonfree/commits/616fc2dd4df421e3974179d9e46d45e7006aeb28.

[   13.628726] brcmfmac: F1 signature read @0x18000000=0x15264345
[   13.636636] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[   13.646269] usbcore: registered new interface driver brcmfmac
[   13.860613] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[   13.883539] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Mar  2 2020 23:30:41 version 7.45.202 (r724630 CY) FWID 01-72f6ece2

AP operation works fine for ~3.5 days. I have ~8 ESP8266 connected, so light but constant MQTT traffic. Then the firmware crashes:

[314842.479050] ieee80211 phy0: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[314842.488254] ieee80211 phy0: brcmf_fill_bss_param: Failed to get bss info (-110)
[315278.446714] ieee80211 phy0: send_key_to_dongle: wsec_key error (-110)
[315279.924970] ieee80211 phy0: brcmf_fw_crashed: Firmware has halted or crashed

Now all requests to the firmware will fail with -110.

[315281.006707] ieee80211 phy0: brcmf_cfg80211_change_station: Setting SCB (de-)authorize failed, -110
[315283.566711] ieee80211 phy0: brcmf_cfg80211_del_station: SCB_DEAUTHENTICATE_FOR_REASON failed -110
[315387.502641] ieee80211 phy0: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[315390.062625] ieee80211 phy0: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[315390.071825] ieee80211 phy0: brcmf_cfg80211_get_station: GET STA INFO failed, -110
[315392.622627] ieee80211 phy0: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[315395.182624] ieee80211 phy0: brcmf_proto_bcdc_query_dcmd: brcmf_proto_bcdc_msg failed w/status -110
[315395.191823] ieee80211 phy0: brcmf_cfg80211_get_station: GET STA INFO failed, -110

I've tried reloading brcmfmac, but that won't reset the firmware for me. Even rebooting doesn't do that. I have to poweroff, and reconnect the power supply, after which the AP returns to a working condition.

I can contribute debug logging from brcfmac if that would be helpful, I don't have a clear reproducer either, since all connected STA are doing the same thing all day long: IPv4, HTTP, DHCP, DNS, NTP, MQTT.