search

openwrt / mt76

mac80211 driver for MediaTek MT76x0e, MT76x2e, MT7603, MT7615, MT7628 and MT7688
751 stars 343 forks source link

mt7915e: Message timeout while waiting for mcu response #690

Open mriemensbergersw opened 2 years ago

mriemensbergersw commented 2 years ago

I'm seeing the following timeouts on a mt7621 board with mt7915e wifi:

[63464.928109] mt7915e 0000:02:00.0: Message 00005aed (seq 15) timeout
[63485.407585] mt7915e 0000:02:00.0: Message 0000aded (seq 1) timeout
[63505.887090] mt7915e 0000:02:00.0: Message 00005aed (seq 2) timeout
[63526.366557] mt7915e 0000:02:00.0: Message 000094ed (seq 3) timeout
[63546.846045] mt7915e 0000:02:00.0: Message 000094ed (seq 4) timeout
[63567.325544] mt7915e 0000:02:00.0: Message 00005aed (seq 5) timeout
[63587.805033] mt7915e 0000:02:00.0: Message 00005aed (seq 6) timeout
[63608.284525] mt7915e 0000:02:00.0: Message 00005aed (seq 7) timeout
[63628.764011] mt7915e 0000:02:00.0: Message 000025ed (seq 8) timeout

Once those messages show up, wifi is unusable. A reboot is necessary to fix it.

The board is running openwrt 21.02 with the mt76 from c67df0d3130a51d79b558f0329c2ca289c73b16e and with the latest mt76 master (minus multi-bss and WED, plus "mt76: lock mt76_queue before access").

The issue only reproduces with lots (40+) of connected active clients sometimes only after 40 hour and later after boot. Unfortunately, I can also not directly reproduce it locally myself. I only have reports from multiple deployed accesspoints.

It appears as if the mcu is locked up or at least communication with the mcu is no longer possible. SO it appears to be either a problem with the firmware or a problem with the communication between driver and firmware. Any ideas or suggestions further debug or fix the problem would be appreciated.

mriemensbergersw commented 2 years ago

Another error log is:

[1313.808417] mt7915e 0000:02:00.0: Message 000007ed (seq 3) timeout
[1334.288216] mt7915e 0000:02:00.0: Message 000007ed (seq 4) timeout

This one happend right after a network restart as far as I know. The message seems to be from mt76_connac_mcu_set_pm() likely in mt7915_start() (the only alternative place would be mt7915_stop().

bittorf commented 2 years ago

I can confirm that. Archer C6U v1, OpenWRT trunk r20332 and r20519

[   56.852678] mt7615e 0000:02:00.0: Message 000026ed (seq 7) timeout
[   77.332674] mt7615e 0000:02:00.0: Message 000025ed (seq 8) timeout
[   97.812685] mt7615e 0000:02:00.0: Message 000049ed (seq 9) timeout
[   97.820139] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1: link becomes ready
[  118.292680] mt7615e 0000:02:00.0: Message 000025ed (seq 10) timeout
[  118.298961] wlan1: failed to move IBSS STA 64:09:80:7f:48:86 to state 3 (-145) - keeping it anyway
[  138.772675] mt7615e 0000:02:00.0: Message 000049ed (seq 11) timeout
[  161.812680] mt7615e 0000:02:00.0: Message 000049ed (seq 12) timeout
[  182.292666] mt7615e 0000:02:00.0: Message 000025ed (seq 13) timeout
archer-C6U-v1-routerwand
up: 191.42 load: 6.11 rest: 3.17 1.25 1/109 7795
[  205.332663] mt7615e 0000:02:00.0: Message 000049ed (seq 14) timeout

the 5 GHz network does not come up sometimes: 

24: wlan1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 3c:84:6a:8c:75:78 brd ff:ff:ff:ff:ff:ff
25: wlan1-1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 3e:84:6a:8c:75:78 brd ff:ff:ff:ff:ff:ff permaddr 3c:84:6a:8c:75:78

# 5 GHz
root@box:~ iw dev wlan1 info
Interface wlan1
    ifindex 24
    wdev 0x100000006
    addr 3c:84:6a:8c:75:78
    type IBSS
    wiphy 1
    txpower 23.00 dBm
    multicast TXQ:
        qsz-byt qsz-pkt flows   drops   marks   overlmt hashcol tx-bytes    tx-packets
        0   0   0   0   0   0   0   0       0

root@box:~ iw dev wlan1-1 info
Interface wlan1-1
    ifindex 25
    wdev 0x100000007
    addr 3e:84:6a:8c:75:78
    type AP
    wiphy 1
    txpower 23.00 dBm
    multicast TXQ:
        qsz-byt qsz-pkt flows   drops   marks   overlmt hashcol tx-bytes    tx-packets
        0   0   0   0   0   0   0   0       0

root@box:~ uci show wireless
[...] (omitting 2 GHz)
wireless.radio1=wifi-device
wireless.radio1.type='mac80211'
wireless.radio1.country='US'
wireless.radio1.path='1e140000.pcie/pci0000:00/0000:00:01.0/0000:02:00.0'
wireless.radio1.band='5g'
wireless.radio1.chanbw='20'
wireless.radio1.channel='40'
wireless.radio1.htmode='VHT40'

wireless.@wifi-iface[2]=wifi-iface
wireless.@wifi-iface[2].device='radio1'
wireless.@wifi-iface[2].network='wlanadhocRADIO1'
wireless.@wifi-iface[2].mode='adhoc'
wireless.@wifi-iface[2].bssid='02:ca:ff:ee:00:40'
wireless.@wifi-iface[2].ssid='ffintern.5GHz'

wireless.@wifi-iface[3]=wifi-iface
wireless.@wifi-iface[3].device='radio1'
wireless.@wifi-iface[3].network='wlanRADIO1'
wireless.@wifi-iface[3].encryption='none'
wireless.@wifi-iface[3].mode='ap'
wireless.@wifi-iface[3].ssid='weimar.freifunk.net'

deleting the "adhoc" interface with
uci del wireless.@wifi-iface[2] && wifi
does not seem to change something...

mriemensbergersw commented 2 years ago

It seems that wifi scanning under certain conditions seems to trigger the issue. Is this related to #511 albeit for a different board? @nbd168 You apparently fixed #511 through a firmware update. Any idea whether mt7915e could exhibit a similar fw issue?

mriemensbergersw commented 2 years ago

@ryderlee1110 @LorenzoBianconi @ptpt52 Since you were involved int #511, any ideas how to debug or fix this issue. Any help or ideas how to make progress in debugging or solving this issue would be appreciated.

everything411 commented 2 years ago

same problem on a Xiaomi Redmi AX6S running 22.03.0-rc6. 5Ghz wifi ping lantancy reaches about 1000ms with very slow connection speed which is about 200KB/s. rebooting the router does not help

[   85.248355] br-lan2: port 4(wlan1) entered blocking state
[   85.253750] br-lan2: port 4(wlan1) entered forwarding state
[ 2399.204837] mt7915e 0000:01:00.0: Message 00005aed (seq 8) timeout
[ 2419.684810] mt7915e 0000:01:00.0: Message 00005aed (seq 9) timeout
[ 2440.164800] mt7915e 0000:01:00.0: Message 00005aed (seq 10) timeout
[ 2460.644787] mt7915e 0000:01:00.0: Message 000026ed (seq 11) timeout
[ 2481.124774] mt7915e 0000:01:00.0: Message 00005aed (seq 12) timeout
[ 2501.604767] mt7915e 0000:01:00.0: Message 000026ed (seq 13) timeout
[ 2522.094915] mt7915e 0000:01:00.0: Message 00005aed (seq 14) timeout
[ 2542.564740] mt7915e 0000:01:00.0: Message 00005aed (seq 15) timeout
[ 2563.044735] mt7915e 0000:01:00.0: Message 00005aed (seq 1) timeout
[ 2583.524726] mt7915e 0000:01:00.0: Message 000026ed (seq 2) timeout
[ 2604.004707] mt7915e 0000:01:00.0: Message 00005aed (seq 3) timeout
[ 2624.484749] mt7915e 0000:01:00.0: Message 000026ed (seq 4) timeout
[ 2644.964696] mt7915e 0000:01:00.0: Message 00005aed (seq 5) timeout
RinCat commented 2 years ago

Same issue on Linksys E8450 with OpenWRT 22.03 stable. Once happened, the wifi is not usable(but can show SSID). It can be temporarily restored after a reboot.

kern.err kernel: [ 5905.757308] mt7915e 0000:01:00.0: Message 00005aed (seq 10) timeout
kern.err kernel: [ 5926.237086] mt7915e 0000:01:00.0: Message 00005aed (seq 11) timeout
kern.err kernel: [ 5946.716874] mt7915e 0000:01:00.0: Message 00005aed (seq 12) timeout
kern.err kernel: [ 5967.196667] mt7915e 0000:01:00.0: Message 000026ed (seq 13) timeout
kern.err kernel: [ 5987.676453] mt7915e 0000:01:00.0: Message 00005aed (seq 14) timeout
kern.err kernel: [ 6008.156298] mt7915e 0000:01:00.0: Message 00005aed (seq 15) timeout
kern.err kernel: [ 6028.636060] mt7915e 0000:01:00.0: Message 00005aed (seq 1) timeout
kern.err kernel: [ 6049.115844] mt7915e 0000:01:00.0: Message 000026ed (seq 2) timeout
daemon.notice hostapd: nl80211: nl80211_recv_beacons->nl_recvmsgs failed: -5
daemon.notice hostapd: nl80211: nl80211_recv_beacons->nl_recvmsgs failed: -5
daemon.notice hostapd: wlan1: AP-STA-DISCONNECTED xx:xx:xx:xx:xx:xx
kern.err kernel: [ 6069.595634] mt7915e 0000:01:00.0: Message 00005aed (seq 3) timeout
kern.err kernel: [ 6090.075434] mt7915e 0000:01:00.0: Message 000025ed (seq 4) timeout
kern.err kernel: [ 6090.081639] wlan1: failed to remove key (0, xx:xx:xx:xx:xx:xx) from hardware (-110)
kern.err kernel: [ 6110.555218] mt7915e 0000:01:00.0: Message 00005aed (seq 5) timeout
kern.err kernel: [ 6131.035021] mt7915e 0000:01:00.0: Message 000025ed (seq 6) timeout
kern.info kernel: [ 6131.041226] wlan1: HW problem - can not stop rx aggregation for xx:xx:xx:xx:xx:xx tid 0
kern.err kernel: [ 6151.514715] mt7915e 0000:01:00.0: Message 00005aed (seq 7) timeout
kern.err kernel: [ 6171.994301] mt7915e 0000:01:00.0: Message 000025ed (seq 8) timeout
kern.debug kernel: [ 6172.000545] ------------[ cut here ]------------
kern.warn kernel: [ 6172.005174] WARNING: CPU: 1 PID: 1710 at 0xffffffc0088caa68 [mac80211@00000000d8798a7e+0x79000]
kern.debug kernel: [ 6172.013862] Modules linked in: pppoe ppp_async nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet pppox ppp_generic nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack mt7915e mt7615e mt7615_common mt76_connac_lib mt76 mac80211 cfg80211 slhc nfnetlink nf_reject_ipv6 nf_reject_ipv4 nf_log_ipv6 nf_log_ipv4 nf_log_common nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c hwmon crc_ccitt compat seqiv leds_gpio xhci_plat_hcd gpio_button_hotplug
kern.debug kernel: [ 6172.072741] CPU: 1 PID: 1710 Comm: hostapd Tainted: G S                5.10.138 #0
kern.debug kernel: [ 6172.080300] Hardware name: Linksys E8450 (UBI) (DT)
kern.debug kernel: [ 6172.085171] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
kern.debug kernel: [ 6172.091179] pc : 0xffffffc0088caa68 [mac80211@00000000d8798a7e+0x79000]
kern.debug kernel: [ 6172.097789] lr : 0xffffffc0088ca914 [mac80211@00000000d8798a7e+0x79000]
kern.debug kernel: [ 6172.104392] sp : ffffffc01158b800
kern.debug kernel: [ 6172.107697] x29: ffffffc01158b800 x28: ffffff80009d21c0
kern.debug kernel: [ 6172.113003] x27: ffffffc01158bdf0 x26: 0000000000000000
kern.debug kernel: [ 6172.118308] x25: ffffffc010acf000 x24: ffffff80021c90a0
kern.debug kernel: [ 6172.123612] x23: ffffff80021c60e8 x22: ffffffc0088c9144
kern.debug kernel: [ 6172.128916] x21: 0000000000000003 x20: ffffff8000d91a00
kern.debug kernel: [ 6172.134220] x19: ffffff800667f000 x18: 0000000000000161
kern.debug kernel: [ 6172.139525] x17: 00000000ffffffff x16: 0000000000000002
kern.debug kernel: [ 6172.144829] x15: ffffffc010a09998 x14: 0000000000000423
kern.debug kernel: [ 6172.150132] x13: 0000000000000161 x12: ffffffc01158b2f8
kern.debug kernel: [ 6172.155436] x11: ffffffc010a61998 x10: 00000000fffff000
kern.debug kernel: [ 6172.160740] x9 : ffffffc010a61998 x8 : 0000000000000000
kern.debug kernel: [ 6172.166044] x7 : ffffffc010a09998 x6 : 0000000000000001
kern.debug kernel: [ 6172.171348] x5 : 0000000000000000 x4 : 0000000000000000
kern.debug kernel: [ 6172.176652] x3 : 0000000000000000 x2 : 0000000000000001
kern.debug kernel: [ 6172.181956] x1 : 0000000000000002 x0 : 00000000ffffff92
kern.debug kernel: [ 6172.187261] Call trace:
kern.debug kernel: [ 6172.189705]  0xffffffc0088caa68 [mac80211@00000000d8798a7e+0x79000]
kern.debug kernel: [ 6172.195968]  0xffffffc0088c9144 [mac80211@00000000d8798a7e+0x79000]
kern.debug kernel: [ 6172.202227]  0xffffffc0088bf82c [mac80211@00000000d8798a7e+0x79000]
kern.debug kernel: [ 6172.208488]  0xffffffc0088c238c [mac80211@00000000d8798a7e+0x79000]
kern.debug kernel: [ 6172.214747]  0xffffffc0088d8a08 [mac80211@00000000d8798a7e+0x79000]
kern.debug kernel: [ 6172.221009]  0xffffffc0088865c4 [cfg80211@00000000e034fde9+0x44000]
kern.debug kernel: [ 6172.227265]  0xffffffc0106bc8d4
kern.debug kernel: [ 6172.230397]  0xffffffc0106bcc60
kern.debug kernel: [ 6172.233527]  0xffffffc0106bbab8
kern.debug kernel: [ 6172.236658]  0xffffffc0106bc234
kern.debug kernel: [ 6172.239790]  0xffffffc0106bb2e4
kern.debug kernel: [ 6172.242921]  0xffffffc0106bb570
kern.debug kernel: [ 6172.246052]  0xffffffc01062c500
kern.debug kernel: [ 6172.249184]  0xffffffc01062f858
kern.debug kernel: [ 6172.252315]  0xffffffc01062f924
kern.debug kernel: [ 6172.255446]  0xffffffc01062f990
kern.debug kernel: [ 6172.258578]  0xffffffc010010f30
kern.debug kernel: [ 6172.261709]  0xffffffc010010ffc
kern.debug kernel: [ 6172.264841]  0xffffffc0107fb124
kern.debug kernel: [ 6172.267973]  0xffffffc0107fb540
kern.debug kernel: [ 6172.271104]  0xffffffc0100025c8
kern.warn kernel: [ 6172.274236] ---[ end trace e69d212579d70cca ]---
kern.err kernel: [ 6192.473892] mt7915e 0000:01:00.0: Message 00005aed (seq 9) timeout
kern.err kernel: [ 6212.953483] mt7915e 0000:01:00.0: Message 000025ed (seq 10) timeout
daemon.notice hostapd: nl80211: wpa_driver_nl80211_event_receive->nl_recvmsgs failed: -5
daemon.notice hostapd: nl80211: wpa_driver_nl80211_event_receive->nl_recvmsgs failed: -5
kern.err kernel: [ 6233.433080] mt7915e 0000:01:00.0: Message 0000aded (seq 11) timeout
kern.err kernel: [ 6253.912682] mt7915e 0000:01:00.0: Message 00005aed (seq 12) timeout
kern.err kernel: [ 6274.392295] mt7915e 0000:01:00.0: Message 0000aded (seq 13) timeout
kern.err kernel: [ 6294.871907] mt7915e 0000:01:00.0: Message 00005aed (seq 14) timeout
kern.err kernel: [ 6315.351511] mt7915e 0000:01:00.0: Message 0000aded (seq 15) timeout
kern.err kernel: [ 6335.831119] mt7915e 0000:01:00.0: Message 00005aed (seq 1) timeout
kern.err kernel: [ 6356.310752] mt7915e 0000:01:00.0: Message 00005aed (seq 2) timeout
kern.err kernel: [ 6376.790372] mt7915e 0000:01:00.0: Message 00005aed (seq 3) timeout
kern.err kernel: [ 6397.270008] mt7915e 0000:01:00.0: Message 000026ed (seq 4) timeout
daemon.notice hostapd: nl80211: wpa_driver_nl80211_event_receive->nl_recvmsgs failed: -5
daemon.notice hostapd: nl80211: wpa_driver_nl80211_event_receive->nl_recvmsgs failed: -5
kern.err kernel: [ 6417.749653] mt7915e 0000:01:00.0: Message 00005aed (seq 5) timeout
kern.err kernel: [ 6438.229291] mt7915e 0000:01:00.0: Message 0000aded (seq 6) timeout
kern.err kernel: [ 6458.708933] mt7915e 0000:01:00.0: Message 00005aed (seq 7) timeout
kern.err kernel: [ 6479.188575] mt7915e 0000:01:00.0: Message 00005aed (seq 8) timeout
kern.err kernel: [ 6499.668244] mt7915e 0000:01:00.0: Message 00005aed (seq 9) timeout
kern.err kernel: [ 6520.147894] mt7915e 0000:01:00.0: Message 000026ed (seq 10) timeout
kern.err kernel: [ 6540.627544] mt7915e 0000:01:00.0: Message 00005aed (seq 11) timeout
ktgeek commented 2 years ago

This is pure anecdotal evidence… but i was getting this daily with schedutil as the scheduler on my BRT3200. I switched back to ondemand a few days ago and it hasn’t happened since.Schedutil had the CPU running at a higher frequency on average, which could be more heat… but that’s only speculation in what could be making the difference.

nbd168 commented 2 years ago

You can write 1 to /sys/kernel/debug/ieee80211/phy*/mt76/fw_debug_wm and the firmware will print debug messages. Hopefully it will indicate where it is crashing

RinCat commented 2 years ago

New log with debug info.

daemon.notice hostapd: wlan1: AP-STA-DISCONNECTED xx:xx:xx:xx:xx:xx
daemon.info hostapd: wlan1: STA xx:xx:xx:xx:xx:xx IEEE 802.11: disassociated
kern.info kernel: [43330.952585] ieee80211 phy1: WA: Invalid baDirection
kern.info kernel: [43331.092026] ieee80211 phy1: WM: ( 362.767836:56:MQM-W)[WR] BSS usage overflows during removing entry
kern.info kernel: [43331.101243] ieee80211 phy1: WA: ucWlanIdx = 2, fgEnable = 0, ucAc = 15, u4MaxQLen = 2193
kern.info kernel: [43331.109342] ieee80211 phy1: WA: ==>rxmDeleteStaRec, prStaRecord->fgIsInUse = 1
daemon.info hostapd: wlan1: STA xx:xx:xx:xx:xx:xx IEEE 802.11: deauthenticated due to inactivity (timer DEAUTH/REMOVE)
kern.info kernel: [43333.329291] ieee80211 phy1: WM: ( 365.004561:57:BSS-E)_whCapSetGeneric_Falcon DW0 = 0x0, DW1 = 0x0
kern.info kernel: [43333.338648] ieee80211 phy1: WM: ( 365.013930:58:TXC-E)heACtrlInitStaRec(): prOperMode=401308, u2OperMode=51!
daemon.info hostapd: wlan1: STA xx:xx:xx:xx:xx:xx IEEE 802.11: associated (aid 1)
daemon.notice hostapd: wlan1: AP-STA-CONNECTED xx:xx:xx:xx:xx:xx
daemon.info hostapd: wlan1: STA xx:xx:xx:xx:xx:xx RADIUS: starting accounting session A65A6B5BBA020564
daemon.info hostapd: wlan1: STA xx:xx:xx:xx:xx:xx WPA: pairwise key handshake completed (RSN)
daemon.notice hostapd: wlan1: EAPOL-4WAY-HS-COMPLETED xx:xx:xx:xx:xx:xx
kern.info kernel: [43333.498698] ieee80211 phy1: WA: Invalid baDirection
daemon.notice netifd: Network device 'lan3' link is down
kern.info kernel: [44112.187084] mt7530 mdio-bus:00 lan3: Link is Down
kern.info kernel: [44112.192047] br-lan: port 3(lan3) entered disabled state
kern.info kernel: [44114.411110] mt7530 mdio-bus:00 lan3: Link is Up - 10Mbps/Full - flow control off
kern.info kernel: [44114.418538] br-lan: port 3(lan3) entered blocking state
kern.info kernel: [44114.423783] br-lan: port 3(lan3) entered forwarding state
daemon.notice netifd: Network device 'lan3' link is up
daemon.notice hostapd: wlan1: AP-STA-DISCONNECTED xx:xx:xx:xx:xx:xx
kern.info kernel: [45118.277898] ieee80211 phy1: WA: Invalid baDirection
kern.info kernel: [45118.377341] ieee80211 phy1: WM: (2149.624953:59:MQM-W)[WR] BSS usage overflows during removing entry
kern.info kernel: [45118.386498] ieee80211 phy1: WA: ucWlanIdx = 2, fgEnable = 0, ucAc = 15, u4MaxQLen = 2193
kern.info kernel: [45118.394617] ieee80211 phy1: WA: ==>rxmDeleteStaRec, prStaRecord->fgIsInUse = 1
kern.info kernel: [45120.562681] ieee80211 phy1: WM: (2151.809768:60:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45120.621582] ieee80211 phy1: WM: (2151.868636:61:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45120.631182] ieee80211 phy1: WM: (2151.870955:62:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45120.820749] ieee80211 phy1: WM: (2152.067794:63:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45120.924803] ieee80211 phy1: WM: (2152.171828:64:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45121.052827] ieee80211 phy1: WM: (2152.299819:65:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45121.148159] ieee80211 phy1: WM: (2152.395125:66:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45121.224042] ieee80211 phy1: WM: (2152.470961:67:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45121.967933] ieee80211 phy1: WM: (2153.214705:68:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45122.004239] ieee80211 phy1: WM: (2153.250991:69:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45122.058598] ieee80211 phy1: WM: (2153.305343:70:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45122.158862] ieee80211 phy1: WM: (2153.405593:71:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45122.209652] ieee80211 phy1: WM: (2153.456343:72:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45122.249646] ieee80211 phy1: WM: (2153.496352:73:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45122.300190] ieee80211 phy1: WM: (2153.546859:74:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45122.433545] ieee80211 phy1: WM: (2153.680190:75:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45122.626761] ieee80211 phy1: WM: (2153.873366:76:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45122.664796] ieee80211 phy1: WM: (2153.911391:77:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45122.715881] ieee80211 phy1: WM: (2153.962478:78:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45123.606024] ieee80211 phy1: WM: (2154.852401:79:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45123.750816] ieee80211 phy1: WM: (2154.997145:80:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45123.803435] ieee80211 phy1: WM: (2155.049758:81:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45123.841175] ieee80211 phy1: WM: (2155.087478:82:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45123.915872] ieee80211 phy1: WM: (2155.162185:83:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45123.968354] ieee80211 phy1: WM: (2155.214644:84:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45124.200948] ieee80211 phy1: WM: (2155.447188:85:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45124.253620] ieee80211 phy1: WM: (2155.499831:86:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45125.039152] ieee80211 phy1: WM: (2156.285170:87:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45125.118165] ieee80211 phy1: WM: (2156.364180:88:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45125.216554] ieee80211 phy1: WM: (2156.462539:89:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45125.271558] ieee80211 phy1: WM: (2156.517531:90:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45125.338751] ieee80211 phy1: WM: (2156.584700:91:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45125.378336] ieee80211 phy1: WM: (2156.624282:92:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45129.198870] ieee80211 phy1: WM: (2160.443892:93:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45129.299369] ieee80211 phy1: WM: (2160.544387:94:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45129.350606] ieee80211 phy1: WM: (2160.595595:95:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45129.385885] ieee80211 phy1: WM: (2160.630874:96:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45130.374976] ieee80211 phy1: WM: (2161.619704:97:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45130.575223] ieee80211 phy1: WM: (2161.819930:98:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45130.627603] ieee80211 phy1: WM: (2161.872298:99:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45130.652921] ieee80211 phy1: WM: (2161.897597:00:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45131.138047] ieee80211 phy1: WM: (2162.382613:01:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45131.338398] ieee80211 phy1: WM: (2162.582930:02:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45131.398539] ieee80211 phy1: WM: (2162.643050:03:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45131.423713] ieee80211 phy1: WM: (2162.668227:04:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45131.915340] ieee80211 phy1: WM: (2163.159713:05:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45132.115899] ieee80211 phy1: WM: (2163.360244:06:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45132.169260] ieee80211 phy1: WM: (2163.413588:07:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45132.194533] ieee80211 phy1: WM: (2163.438857:08:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45132.303296] ieee80211 phy1: WM: (2163.547591:09:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45132.685933] ieee80211 phy1: WM: (2163.930098:10:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45132.886154] ieee80211 phy1: WM: (2164.130294:11:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45132.919905] ieee80211 phy1: WM: (2164.164046:12:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45132.970370] ieee80211 phy1: WM: (2164.214492:13:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45133.023084] ieee80211 phy1: WM: (2164.267196:14:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45133.048529] ieee80211 phy1: WM: (2164.292647:15:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45133.466300] ieee80211 phy1: WM: (2164.710280:16:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45133.666518] ieee80211 phy1: WM: (2164.910476:17:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45133.715064] ieee80211 phy1: WM: (2164.959029:18:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45133.740265] ieee80211 phy1: WM: (2164.984206:19:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45134.240052] ieee80211 phy1: WM: (2165.483870:20:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45134.440257] ieee80211 phy1: WM: (2165.684035:21:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45134.492839] ieee80211 phy1: WM: (2165.736617:22:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45134.518140] ieee80211 phy1: WM: (2165.761885:23:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45135.011981] ieee80211 phy1: WM: (2166.255599:24:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45135.212285] ieee80211 phy1: WM: (2166.455886:25:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45135.264817] ieee80211 phy1: WM: (2166.508406:26:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45135.290034] ieee80211 phy1: WM: (2166.533614:27:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45135.786304] ieee80211 phy1: WM: (2167.029738:28:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45135.986425] ieee80211 phy1: WM: (2167.229842:29:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45136.034853] ieee80211 phy1: WM: (2167.278243:30:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45136.060142] ieee80211 phy1: WM: (2167.303542:31:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45136.559570] ieee80211 phy1: WM: (2167.802840:32:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45136.759791] ieee80211 phy1: WM: (2168.003005:33:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45136.812247] ieee80211 phy1: WM: (2168.055465:34:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45136.837555] ieee80211 phy1: WM: (2168.080733:35:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45137.325420] ieee80211 phy1: WM: (2168.568496:36:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45137.525690] ieee80211 phy1: WM: (2168.768721:37:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45137.574206] ieee80211 phy1: WM: (2168.817244:38:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45137.599440] ieee80211 phy1: WM: (2168.842452:39:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45138.178630] ieee80211 phy1: WM: (2169.421523:40:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.err kernel: [45139.786647] mt7915e 0000:01:00.0: Message 00005aed (seq 12) timeout
kern.info kernel: [45142.307029] ieee80211 phy1: WM: (2173.548842:41:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45145.013339] ieee80211 phy1: WM: (2176.254592:42:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45145.065006] ieee80211 phy1: WM: (2176.306228:43:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45145.836993] ieee80211 phy1: WM: (2177.078048:44:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45145.888722] ieee80211 phy1: WM: (2177.129744:45:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45148.182678] ieee80211 phy1: WM: (2179.423171:46:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45148.234389] ieee80211 phy1: WM: (2179.474868:47:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45150.809577] ieee80211 phy1: WM: (2182.049422:48:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45150.861332] ieee80211 phy1: WM: (2182.101180:49:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45152.310281] ieee80211 phy1: WM: (2183.549758:50:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.err kernel: [45160.266375] mt7915e 0000:01:00.0: Message 00005aed (seq 13) timeout
kern.info kernel: [45162.313183] ieee80211 phy1: WM: (2193.550277:51:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45172.316519] ieee80211 phy1: WM: (2203.551192:52:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45174.540641] ieee80211 phy1: WM: (2205.774794:53:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45174.550245] ieee80211 phy1: WM: (2205.784438:54:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.err kernel: [45180.746092] mt7915e 0000:01:00.0: Message 00005aed (seq 14) timeout
kern.info kernel: [45182.319240] ieee80211 phy1: WM: (2213.551528:55:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45185.540491] ieee80211 phy1: WM: (2216.772017:56:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45185.556862] ieee80211 phy1: WM: (2216.788253:57:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45187.302266] ieee80211 phy1: WM: (2218.533370:58:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45187.758172] ieee80211 phy1: WM: (2218.989180:59:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45187.806759] ieee80211 phy1: WM: (2219.037734:60:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45188.760500] ieee80211 phy1: WM: (2219.991256:61:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45189.160918] ieee80211 phy1: WM: (2220.391585:62:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45189.366811] ieee80211 phy1: WM: (2220.597243:63:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45190.311268] ieee80211 phy1: WM: (2221.541640:64:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45190.719559] ieee80211 phy1: WM: (2221.949843:65:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45191.416074] ieee80211 phy1: WM: (2222.646163:66:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45191.742789] ieee80211 phy1: WM: (2222.972823:67:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45192.137262] ieee80211 phy1: WM: (2223.367202:68:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45192.322726] ieee80211 phy1: WM: (2223.552596:69:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45192.342945] ieee80211 phy1: WM: (2223.572829:70:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45192.511457] ieee80211 phy1: WM: (2223.741317:71:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45192.562978] ieee80211 phy1: WM: (2223.792830:72:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45192.647238] ieee80211 phy1: WM: (2223.877059:73:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45192.698829] ieee80211 phy1: WM: (2223.928634:74:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45192.764228] ieee80211 phy1: WM: (2223.994033:75:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45192.815864] ieee80211 phy1: WM: (2224.045638:76:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45193.144496] ieee80211 phy1: WM: (2224.374190:77:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45193.363808] ieee80211 phy1: WM: (2224.593459:78:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45193.437601] ieee80211 phy1: WM: (2224.667220:79:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45193.630190] ieee80211 phy1: WM: (2224.859786:80:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45193.872414] ieee80211 phy1: WM: (2225.101943:81:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45194.160300] ieee80211 phy1: WM: (2225.389754:82:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45194.365851] ieee80211 phy1: WM: (2225.595259:83:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45194.902935] ieee80211 phy1: WM: (2226.132216:84:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45195.108563] ieee80211 phy1: WM: (2226.337783:85:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45195.675736] ieee80211 phy1: WM: (2226.904830:86:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45195.881392] ieee80211 phy1: WM: (2227.110427:87:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45196.917589] ieee80211 phy1: WM: (2228.146376:88:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45197.123218] ieee80211 phy1: WM: (2228.351973:89:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45197.678516] ieee80211 phy1: WM: (2228.907119:90:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45197.884079] ieee80211 phy1: WM: (2229.112655:91:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45198.689421] ieee80211 phy1: WM: (2229.917800:92:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45198.895042] ieee80211 phy1: WM: (2230.123366:93:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45198.935507] ieee80211 phy1: WM: (2230.163802:94:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45199.140645] ieee80211 phy1: WM: (2230.368911:95:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45199.681530] ieee80211 phy1: WM: (2230.909652:96:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45199.900329] ieee80211 phy1: WM: (2231.128402:97:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45200.435122] ieee80211 phy1: WM: (2231.663039:98:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45200.640734] ieee80211 phy1: WM: (2231.868636:99:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45200.945863] ieee80211 phy1: WM: (2232.173690:00:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.info kernel: [45201.151548] ieee80211 phy1: WM: (2232.379348:01:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
kern.err kernel: [45201.225819] mt7915e 0000:01:00.0: Message 000026ed (seq 15) timeout
bittorf commented 2 years ago 
# tested with snapshot r20539...r20813 (and earlier):

$ echo 1 >/sys/kernel/debug/ieee80211/phy1/mt76/fw_debug
$ wifi up
# next command hangs for ~16 sec:
$ echo 1 >/sys/kernel/debug/ieee80211/phy1/mt76/fw_debug

$ dmesg
[47859.318557] ieee80211 phy1: N9: (47895.829:CMD-E)EXT_CMD_OPT_BIT_0_ACK
[47859.345363] ieee80211 phy1: N9: (47895.856:CMD-E)EXT_CMD_OPT_BIT_0_ACK
[47860.589939] ieee80211 phy1: N9: Wifi ASSERT @ ../wifi/open_corebase/core/wificore/mlm/bss_cfg.c:225
[47880.848577] mt7615e 0000:02:00.0: Message 000026ed (seq 8) timeout
[47901.328563] mt7615e 0000:02:00.0: Message 000025ed (seq 9) timeout
[47921.808570] mt7615e 0000:02:00.0: Message 000049ed (seq 10) timeout
[47942.288564] mt7615e 0000:02:00.0: Message 000013ed (seq 11) timeout
[47962.768557] mt7615e 0000:02:00.0: Message 000025ed (seq 12) timeout
[47962.774844] wlan1: failed to move IBSS STA 64:09:80:7f:48:86 to state 3 (-145) - keeping it anyway
[47983.248555] mt7615e 0000:02:00.0: Message 000049ed (seq 13) timeout
[48006.288556] mt7615e 0000:02:00.0: Message 000049ed (seq 14) timeout
[48026.768630] mt7615e 0000:02:00.0: Message 000025ed (seq 15) timeout

# a huge load is introduced when this happens:
root@archer-C6U-v1-routerwand:~ :) uptime
 11:11:30 up 43 min,  load average: 7.27, 7.64, 5.61

root@archer-C6U-v1-routerwand:~ :) nproc
4
everything411 commented 2 years ago 
root@OpenWrt:~# dmesg
[75410.438944] ieee80211 phy1: WM: (2370.905700:20:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75420.441856] ieee80211 phy1: WM: (2380.906127:21:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75430.444861] ieee80211 phy1: WM: (2390.906646:22:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75440.448327] ieee80211 phy1: WM: (2400.907623:23:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75450.451538] ieee80211 phy1: WM: (2410.908355:24:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75460.454558] ieee80211 phy1: WM: (2420.908874:25:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75470.457900] ieee80211 phy1: WM: (2430.909728:26:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75480.461567] ieee80211 phy1: WM: (2440.910918:27:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75490.464830] ieee80211 phy1: WM: (2450.911681:28:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75500.468029] ieee80211 phy1: WM: (2460.912383:29:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75510.471025] ieee80211 phy1: WM: (2470.912902:30:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75520.474113] ieee80211 phy1: WM: (2480.913512:31:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75530.476946] ieee80211 phy1: WM: (2490.913848:32:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75540.479884] ieee80211 phy1: WM: (2500.914306:33:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75550.483306] ieee80211 phy1: WM: (2510.915221:34:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75560.487012] ieee80211 phy1: WM: (2520.916442:35:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75570.490028] ieee80211 phy1: WM: (2530.916961:36:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75580.492631] ieee80211 phy1: WM: (2540.917083:37:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75590.495247] ieee80211 phy1: WM: (2550.917205:38:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75600.498281] ieee80211 phy1: WM: (2560.917754:39:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75610.501859] ieee80211 phy1: WM: (2570.918853:40:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75620.505485] ieee80211 phy1: WM: (2580.919982:41:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75630.508532] ieee80211 phy1: WM: (2590.920531:42:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75640.511174] ieee80211 phy1: WM: (2600.920684:43:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75650.514125] ieee80211 phy1: WM: (2610.921142:44:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75660.517251] ieee80211 phy1: WM: (2620.921783:45:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75670.520292] ieee80211 phy1: WM: (2630.922332:46:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75680.523630] ieee80211 phy1: WM: (2640.923187:47:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75690.527158] ieee80211 phy1: WM: (2650.924224:48:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75700.530782] ieee80211 phy1: WM: (2660.925353:49:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75710.533431] ieee80211 phy1: WM: (2670.925536:50:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75720.536644] ieee80211 phy1: WM: (2680.926238:51:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75730.539363] ieee80211 phy1: WM: (2690.926482:52:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75740.542231] ieee80211 phy1: WM: (2700.926849:53:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75750.545770] ieee80211 phy1: WM: (2710.927917:54:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75760.548550] ieee80211 phy1: WM: (2720.928191:55:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75770.551911] ieee80211 phy1: WM: (2730.929076:56:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75780.555421] ieee80211 phy1: WM: (2740.930083:57:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75790.558802] ieee80211 phy1: WM: (2750.930968:58:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75800.561983] ieee80211 phy1: WM: (2760.931670:59:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75810.565518] ieee80211 phy1: WM: (2770.932708:60:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75820.568983] ieee80211 phy1: WM: (2780.933685:61:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75830.572018] ieee80211 phy1: WM: (2790.934234:62:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75840.574990] ieee80211 phy1: WM: (2800.934722:63:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75850.577898] ieee80211 phy1: WM: (2810.935149:64:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75860.581128] ieee80211 phy1: WM: (2820.935882:65:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75870.583713] ieee80211 phy1: WM: (2830.935973:66:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75880.586950] ieee80211 phy1: WM: (2840.936736:67:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75890.590308] ieee80211 phy1: WM: (2850.937591:68:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75900.593510] ieee80211 phy1: WM: (2860.938293:69:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75910.596835] ieee80211 phy1: WM: (2870.939147:70:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75920.600429] ieee80211 phy1: WM: (2880.940246:71:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75930.603345] ieee80211 phy1: WM: (2890.940673:72:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75940.606358] ieee80211 phy1: WM: (2900.941192:73:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75950.609159] ieee80211 phy1: WM: (2910.941497:74:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75960.612279] ieee80211 phy1: WM: (2920.942138:75:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75970.615028] ieee80211 phy1: WM: (2930.942413:76:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75980.618338] ieee80211 phy1: WM: (2940.943206:77:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[75990.621867] ieee80211 phy1: WM: (2950.944244:78:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76000.624834] ieee80211 phy1: WM: (2960.944732:79:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76010.627740] ieee80211 phy1: WM: (2970.945159:80:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76020.630938] ieee80211 phy1: WM: (2980.945861:81:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76030.634228] ieee80211 phy1: WM: (2990.946655:82:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76040.637906] ieee80211 phy1: WM: (3000.947845:83:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76050.641449] ieee80211 phy1: WM: (3010.948913:84:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76060.644596] ieee80211 phy1: WM: (3020.949554:85:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76070.647321] ieee80211 phy1: WM: (3030.949798:86:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76080.650120] ieee80211 phy1: WM: (3040.950103:87:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76090.653687] ieee80211 phy1: WM: (3050.951171:88:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76100.656384] ieee80211 phy1: WM: (3060.951385:89:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76110.659973] ieee80211 phy1: WM: (3070.952483:90:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76120.663041] ieee80211 phy1: WM: (3080.953063:91:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76130.666414] ieee80211 phy1: WM: (3090.953948:92:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76140.669920] ieee80211 phy1: WM: (3100.954955:93:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76150.673405] ieee80211 phy1: WM: (3110.955962:94:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76160.675922] ieee80211 phy1: WM: (3120.955993:95:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76170.678517] ieee80211 phy1: WM: (3130.956084:96:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76180.681364] ieee80211 phy1: WM: (3140.956451:97:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76190.684770] ieee80211 phy1: WM: (3150.957366:98:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76200.688462] ieee80211 phy1: WM: (3160.958587:99:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76210.691014] ieee80211 phy1: WM: (3170.958648:00:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76220.694156] ieee80211 phy1: WM: (3180.959289:01:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76230.697662] ieee80211 phy1: WM: (3190.960326:02:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76240.701272] ieee80211 phy1: WM: (3200.961425:03:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76250.704555] ieee80211 phy1: WM: (3210.962218:04:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76260.708019] ieee80211 phy1: WM: (3220.963195:05:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76270.711569] ieee80211 phy1: WM: (3230.964263:06:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76280.714824] ieee80211 phy1: WM: (3240.965026:07:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76290.718085] ieee80211 phy1: WM: (3250.965789:08:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76300.721146] ieee80211 phy1: WM: (3260.966369:09:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76310.724389] ieee80211 phy1: WM: (3270.967132:10:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76320.727437] ieee80211 phy1: WM: (3280.967681:11:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76330.730185] ieee80211 phy1: WM: (3290.967956:12:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76340.733666] ieee80211 phy1: WM: (3300.968932:13:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76350.736469] ieee80211 phy1: WM: (3310.969238:14:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76360.739964] ieee80211 phy1: WM: (3320.970245:15:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76370.743369] ieee80211 phy1: WM: (3330.971160:16:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76380.746642] ieee80211 phy1: WM: (3340.971954:17:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76390.749586] ieee80211 phy1: WM: (3350.972411:18:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76400.753219] ieee80211 phy1: WM: (3360.973541:19:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76410.756005] ieee80211 phy1: WM: (3370.973846:20:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76420.759469] ieee80211 phy1: WM: (3380.974822:21:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76430.762888] ieee80211 phy1: WM: (3390.975738:22:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76440.765804] ieee80211 phy1: WM: (3400.976165:23:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76450.768623] ieee80211 phy1: WM: (3410.976501:24:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76460.771342] ieee80211 phy1: WM: (3420.976745:25:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76470.774201] ieee80211 phy1: WM: (3430.977111:26:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76480.777224] ieee80211 phy1: WM: (3440.977630:27:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76490.780131] ieee80211 phy1: WM: (3450.978057:28:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76500.783546] ieee80211 phy1: WM: (3460.978973:29:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76510.786550] ieee80211 phy1: WM: (3470.979491:30:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76520.789917] ieee80211 phy1: WM: (3480.980376:31:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76530.792684] ieee80211 phy1: WM: (3490.980651:32:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76540.796005] ieee80211 phy1: WM: (3500.981475:33:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76550.798838] ieee80211 phy1: WM: (3510.981841:34:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76560.802548] ieee80211 phy1: WM: (3520.983062:35:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76570.805559] ieee80211 phy1: WM: (3530.983581:36:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76580.809140] ieee80211 phy1: WM: (3540.984649:37:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76590.811699] ieee80211 phy1: WM: (3550.984740:38:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76600.814570] ieee80211 phy1: WM: (3560.985107:39:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76610.817609] ieee80211 phy1: WM: (3570.985656:40:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76620.820616] ieee80211 phy1: WM: (3580.986175:41:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76630.823994] ieee80211 phy1: WM: (3590.987060:42:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76640.827155] ieee80211 phy1: WM: (3600.987731:43:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76650.830792] ieee80211 phy1: WM: (3610.988891:44:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76660.833536] ieee80211 phy1: WM: (3620.989135:45:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76670.836759] ieee80211 phy1: WM: (3630.989867:46:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76680.839970] ieee80211 phy1: WM: (3640.990600:47:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76690.843389] ieee80211 phy1: WM: (3650.991515:48:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76700.846817] ieee80211 phy1: WM: (3660.992461:49:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76710.849975] ieee80211 phy1: WM: (3670.993133:50:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76720.853262] ieee80211 phy1: WM: (3680.993926:51:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76730.856404] ieee80211 phy1: WM: (3690.994598:52:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76740.859876] ieee80211 phy1: WM: (3700.995574:53:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76750.863373] ieee80211 phy1: WM: (3710.996581:54:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76760.866080] ieee80211 phy1: WM: (3720.996795:55:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76770.869780] ieee80211 phy1: WM: (3730.998016:56:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76780.872610] ieee80211 phy1: WM: (3740.998351:57:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76790.876228] ieee80211 phy1: WM: (3750.999480:58:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76800.879209] ieee80211 phy1: WM: (3760.999969:59:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76810.882448] ieee80211 phy1: WM: (3771.000732:60:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76820.885298] ieee80211 phy1: WM: (3781.001067:61:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76830.888878] ieee80211 phy1: WM: (3791.002166:62:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76840.892226] ieee80211 phy1: WM: (3801.003021:63:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76850.895223] ieee80211 phy1: WM: (3811.003539:64:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76860.898373] ieee80211 phy1: WM: (3821.004180:65:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76870.901883] ieee80211 phy1: WM: (3831.005218:66:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76880.904780] ieee80211 phy1: WM: (3841.005614:67:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76890.907759] ieee80211 phy1: WM: (3851.006133:68:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76900.911038] ieee80211 phy1: WM: (3861.006896:69:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76910.914159] ieee80211 phy1: WM: (3871.007537:70:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76920.917256] ieee80211 phy1: WM: (3881.008147:71:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76930.920679] ieee80211 phy1: WM: (3891.009093:72:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76940.923578] ieee80211 phy1: WM: (3901.009490:73:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76950.927150] ieee80211 phy1: WM: (3911.010558:74:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76960.930207] ieee80211 phy1: WM: (3921.011138:75:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76970.933073] ieee80211 phy1: WM: (3931.011504:76:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76980.936751] ieee80211 phy1: WM: (3941.012695:77:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[76990.939677] ieee80211 phy1: WM: (3951.013152:78:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77000.942794] ieee80211 phy1: WM: (3961.013763:79:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77010.946066] ieee80211 phy1: WM: (3971.014556:80:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77020.949166] ieee80211 phy1: WM: (3981.015166:81:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77030.952570] ieee80211 phy1: WM: (3991.016082:82:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77040.955827] ieee80211 phy1: WM: (4001.016845:83:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77050.959281] ieee80211 phy1: WM: (4011.017791:84:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77060.962200] ieee80211 phy1: WM: (4021.018249:85:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77070.965115] ieee80211 phy1: WM: (4031.018676:86:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77080.968821] ieee80211 phy1: WM: (4041.019866:87:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77090.971582] ieee80211 phy1: WM: (4051.020141:88:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77100.975141] ieee80211 phy1: WM: (4061.021209:89:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77110.978743] ieee80211 phy1: WM: (4071.022338:90:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77120.982017] ieee80211 phy1: WM: (4081.023132:91:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77130.984870] ieee80211 phy1: WM: (4091.023467:92:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77140.987742] ieee80211 phy1: WM: (4101.023864:93:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77150.991002] ieee80211 phy1: WM: (4111.024627:94:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77160.994389] ieee80211 phy1: WM: (4121.025542:95:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77170.997897] ieee80211 phy1: WM: (4131.026550:96:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77181.000791] ieee80211 phy1: WM: (4141.026946:97:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77191.003404] ieee80211 phy1: WM: (4151.027068:98:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77201.007089] ieee80211 phy1: WM: (4161.028289:99:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77211.010251] ieee80211 phy1: WM: (4171.028960:00:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77221.012788] ieee80211 phy1: WM: (4181.028991:01:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77231.015613] ieee80211 phy1: WM: (4191.029327:02:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77241.018349] ieee80211 phy1: WM: (4201.029571:03:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77251.021040] ieee80211 phy1: WM: (4211.029784:04:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77261.024116] ieee80211 phy1: WM: (4221.030364:05:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77271.027708] ieee80211 phy1: WM: (4231.031463:06:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77281.030362] ieee80211 phy1: WM: (4241.031646:07:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77291.033768] ieee80211 phy1: WM: (4251.032562:08:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77301.036464] ieee80211 phy1: WM: (4261.032745:09:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77311.038979] ieee80211 phy1: WM: (4271.032775:10:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77321.042355] ieee80211 phy1: WM: (4281.033660:11:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77331.045865] ieee80211 phy1: WM: (4291.034698:12:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77341.049457] ieee80211 phy1: WM: (   6.068500:13:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77351.052700] ieee80211 phy1: WM: (  16.069263:14:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77361.055491] ieee80211 phy1: WM: (  26.069568:15:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77371.058272] ieee80211 phy1: WM: (  36.069843:16:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77381.061300] ieee80211 phy1: WM: (  46.070392:17:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77391.064826] ieee80211 phy1: WM: (  56.071430:18:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77401.068486] ieee80211 phy1: WM: (  66.072590:19:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77411.071948] ieee80211 phy1: WM: (  76.073566:20:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77421.075486] ieee80211 phy1: WM: (  86.074604:21:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77431.078677] ieee80211 phy1: WM: (  96.075306:22:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77441.082240] ieee80211 phy1: WM: ( 106.076374:23:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77451.085149] ieee80211 phy1: WM: ( 116.076801:24:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77461.087987] ieee80211 phy1: WM: ( 126.077167:25:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77471.090612] ieee80211 phy1: WM: ( 136.077289:26:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77481.093989] ieee80211 phy1: WM: ( 146.078174:27:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77491.096873] ieee80211 phy1: WM: ( 156.078571:28:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77501.099765] ieee80211 phy1: WM: ( 166.078968:29:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77511.102738] ieee80211 phy1: WM: ( 176.079456:30:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77521.105986] ieee80211 phy1: WM: ( 186.080219:31:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77531.108691] ieee80211 phy1: WM: ( 196.080433:32:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77541.111632] ieee80211 phy1: WM: ( 206.080891:33:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77551.114886] ieee80211 phy1: WM: ( 216.081653:34:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77561.118460] ieee80211 phy1: WM: ( 226.082722:35:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77571.121608] ieee80211 phy1: WM: ( 236.083393:36:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77581.125188] ieee80211 phy1: WM: ( 246.084492:37:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77591.128398] ieee80211 phy1: WM: ( 256.085224:38:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77601.131455] ieee80211 phy1: WM: ( 266.085773:39:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77611.134686] ieee80211 phy1: WM: ( 276.086506:40:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77621.138024] ieee80211 phy1: WM: ( 286.087360:41:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77631.141555] ieee80211 phy1: WM: ( 296.088398:42:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77641.144131] ieee80211 phy1: WM: ( 306.088489:43:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77651.146854] ieee80211 phy1: WM: ( 316.088734:44:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77661.149841] ieee80211 phy1: WM: ( 326.089222:45:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77671.152670] ieee80211 phy1: WM: ( 336.089558:46:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77681.155754] ieee80211 phy1: WM: ( 346.090168:47:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77691.158810] ieee80211 phy1: WM: ( 356.090717:48:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77701.162479] ieee80211 phy1: WM: ( 366.091907:49:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77711.165318] ieee80211 phy1: WM: ( 376.092243:50:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77721.168782] ieee80211 phy1: WM: ( 386.093220:51:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77731.172085] ieee80211 phy1: WM: ( 396.094044:52:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77741.175712] ieee80211 phy1: WM: ( 406.095173:53:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77751.179024] ieee80211 phy1: WM: ( 416.095997:54:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77761.182332] ieee80211 phy1: WM: ( 426.096821:55:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77771.185453] ieee80211 phy1: WM: ( 436.097462:56:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77781.188527] ieee80211 phy1: WM: ( 446.098041:57:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77791.192169] ieee80211 phy1: WM: ( 456.099201:58:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77801.195530] ieee80211 phy1: WM: ( 466.100056:59:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77811.198590] ieee80211 phy1: WM: ( 476.100635:60:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
[77821.201484] ieee80211 phy1: WM: ( 486.101032:61:MEM-W)whFbmWfdmaCmdEventXmit: xmit fail (out of resource)
ryderlee1110 commented 2 years ago

Is this a long standing issue? Or can only be reproduced after certain commits?

RinCat commented 2 years ago

Is this a long standing issue? Or can only be reproduced after certain commits?

I can only say a few months ago openwrt snapshots don't have this issue.

ryderlee1110 commented 2 years ago

Is this a long standing issue? Or can only be reproduced after certain commits?

I can only say a few months ago openwrt snapshots don't have this issue.

can you help to bisect those snapshots?

RinCat commented 2 years ago

Is this a long standing issue? Or can only be reproduced after certain commits?

I can only say a few months ago openwrt snapshots don't have this issue.

can you help to bisect those snapshots?

I can try, but the issue happened randomly at max 1-2 times a day. So it will be slow.

ryderlee1110 commented 2 years ago

Is this a long standing issue? Or can only be reproduced after certain commits?

I can only say a few months ago openwrt snapshots don't have this issue.

can you help to bisect those snapshots?

I can try, but the issue happened randomly at max 1-2 times a day. So it will be slow.

It would be helpful if someone can bisect snapshots/commits as log only tells us something gets stuck.

RinCat commented 2 years ago

bisect started, cannot estimate required time.

RinCat commented 2 years ago

I encountered 2 issues that maybe related.

First is happened near https://github.com/openwrt/mt76/commit/506bb0605e3e2fa4c5285e6c8866167691a6d71b, where after:

[63036.892782] ieee80211 phy1: WA: Invalid baDirection
[63036.897915] ieee80211 phy1: WA: Invalid baDirection
[63036.992048] ieee80211 phy1: WM: (2883.981691:21:MQM-W)[WR] BSS usage overflows during removing entry
[63037.001232] ieee80211 phy1: WA: ucWlanIdx = 255, fgEnable = 0, ucAc = 15, u4MaxQLen = 2193
[63037.009531] ieee80211 phy1: WA: ==>rxmDeleteStaRec, prStaRecord->fgIsInUse = 1
[63037.823005] ieee80211 phy1: WA: [in] rxmRecvMacDma1R3
[63037.828068] ieee80211 phy1: WA: --> Error RxD Type:0 on MAC_DMA1 RxQ3, send to host!
[63038.119422] ieee80211 phy1: WA: [in] rxmRecvMacDma1R3
wlan1: STA xx:xx:xx:xx:xx:xx IEEE 802.11: did not acknowledge authentication response

The wifi is unable to connect, which is this issue.

Early than that, near https://github.com/openwrt/mt76/commit/eed44048ca575863ebdf1dc73f441cb9604cd388. After sometime I got very slow speed but WiFi connectable. I think it maybe related.

https://github.com/openwrt/mt76/compare/eed44048ca575863ebdf1dc73f441cb9604cd388...506bb0605e3e2fa4c5285e6c8866167691a6d71b The exact commit is still being tested. I only test around 24h if it not happened, assuming the commit is OK.

everything411 commented 2 years ago

My Xiaomi Redmi AX6S was running a snapshot of openwrt which is built at the end of March (forgot the exact date) before running 22.03.0-rc6. The build from March suffers from the same problem.

ptpt52 commented 2 years ago

I can confirm that before 202207300730 no such an issue, but now the issue comes up to my devices

ptpt52 commented 2 years ago

https://github.com/openwrt/mt76/commit/93e3fce916c62d06892d41bf00d0f4c2926c9a0b

checkout to this commit, no such an issue

@nbd168 @ryderlee1110

ptpt52 commented 2 years ago

UPDATE: checkout this commit, still no issue 5ec78e1ec43d1e39edfea1efb9fd4541fa004af0

ptpt52 commented 2 years ago

UPDATE: checkout to this commit, still no issue 9d601f4eee8f7ca3a64fab9f5e969c62b7d39637

ptpt52 commented 2 years ago

so the issue comes with these commits:

pick e49b6063 wifi: mt76: move mt76_rate_power from core to mt76x02 driver code
pick 3f27f6ad wifi: mt76: mt76x02: simplify struct mt76x02_rate_power
pick c07f3d2d wifi: mt76: mt7921: fix antenna signal are way off in monitor mode
pick 9059a5de wifi: mt76: Remove unused inline function mt76_wcid_mask_test()
pick d75f15dd wifi: mt76: mt7915: fix bounds checking for tx-free-done command
pick 06df7e68 wifi: mt76: mt7915: reserve 8 bits for the index of rf registers
pick ad3d0f8d wifi: mt76: mt7915: rework eeprom tx paths and streams init
pick 66065073 wifi: mt76: mt7915: deal with special variant of mt7916
pick b0114a0a wifi: mt76: mt7915: rework testmode tx antenna setting
pick 6dee964e wifi: mt76: connac: introduce mt76_connac_spe_idx()
pick 48c116d9 wifi: mt76: mt7915: add spatial extension index support
pick db6db4de wifi: mt76: mt7915: set correct antenna for radar detection on MT7915D
pick 2b8f56a7 wifi: mt76: mt7915: fix mt7915_mac_set_timing()
pick d554a025 wifi: mt76: mt7915: move wed init routines in mmio.c
pick 61ce40e6 wifi: mt76: mt7915: enable wed for mt7986 chipset
pick 584a96ec wifi: mt76: mt7915: enable wed for mt7986-wmac chipset
pick 172d68b6 mt76: mt76x02: fix vht rate power array overrun
pick 72b87836 Revert "mt76: use IEEE80211_OFFLOAD_ENCAP_ENABLED instead of MT_DRV_AMSDU_OFFLOAD"
pick 8cc34bdc wifi: mt76: mt7921: set MT_DRV_AMSDU_OFFLOAD for USB/SDIO
pick 2182de50 wifi: mt76: fix receiving LLC packets on mt7615/mt7915
pick 269df4b0 wifi: mt76: fix rx checksum offload on mt7615/mt7915/mt7921
ptpt52 commented 2 years ago

@nbd168 @ryderlee1110

revert commit ad3d0f8db00b81a1abeab9793a9d4064ffa00a04 the issue fixed for me.

ryderlee1110 commented 2 years ago

@nbd168 @ryderlee1110

revert commit ad3d0f8 the issue fixed for me.

Are you sure about that? This patch was just sent couple days ago...

ptpt52 commented 2 years ago

@nbd168 @ryderlee1110 revert commit ad3d0f8 the issue fixed for me.

Are you sure about that? This patch was just sent couple days ago...

it is sure for me. I tested it.

everything411 commented 2 years ago

@nbd168 @ryderlee1110 revert commit ad3d0f8 the issue fixed for me.

Are you sure about that? This patch was just sent couple days ago...

it is sure for me. I tested it.

maybe there are several different bugs causing the same (or similar) error. my device was running a build from March 2022 and sufferred from low speed. so i built 22.03.0-rc6 and upgraded the firmware but it's speed got worse and the device was unconnectable in about 1 day after a reboot. now i'm using https://github.com/padavanonly/immortalwrtmt7622, which contains MTKSDK drivers, and it works well with low lantacy and high speed.

ptpt52 commented 2 years ago

@ryderlee1110 @nbd168 I could fix this issue by this patch:

diff --git a/mt7915/eeprom.c b/mt7915/eeprom.c
index e2482c65..92336849 100644
--- a/mt7915/eeprom.c
+++ b/mt7915/eeprom.c
@@ -206,6 +206,9 @@ void mt7915_eeprom_parse_hw_cap(struct mt7915_dev *dev,

                if (!is_mt7986(&dev->mt76))
                        nss_max = 2;
+
+               if (!nss)
+                       nss = nss_max;
        }

        nss = min_t(u8, min_t(u8, nss_max, nss), path);
ryderlee1110 commented 2 years ago

@ryderlee1110 @nbd168 I could fix this issue by this patch:

diff --git a/mt7915/eeprom.c b/mt7915/eeprom.c
index e2482c65..92336849 100644
--- a/mt7915/eeprom.c
+++ b/mt7915/eeprom.c
@@ -206,6 +206,9 @@ void mt7915_eeprom_parse_hw_cap(struct mt7915_dev *dev,

                if (!is_mt7986(&dev->mt76))
                        nss_max = 2;
+
+               if (!nss)
+                       nss = nss_max;
        }

        nss = min_t(u8, min_t(u8, nss_max, nss), path);

This works - I guess there's an issue in your eeprom, so we need this check just in case. Are you able to submit patch to linux-wireless?

ptpt52 commented 2 years ago

@ryderlee1110 @nbd168 I could fix this issue by this patch:

diff --git a/mt7915/eeprom.c b/mt7915/eeprom.c
index e2482c65..92336849 100644
--- a/mt7915/eeprom.c
+++ b/mt7915/eeprom.c
@@ -206,6 +206,9 @@ void mt7915_eeprom_parse_hw_cap(struct mt7915_dev *dev,

                if (!is_mt7986(&dev->mt76))
                        nss_max = 2;
+
+               if (!nss)
+                       nss = nss_max;
        }

        nss = min_t(u8, min_t(u8, nss_max, nss), path);

This works - I guess there's an issue in your eeprom, so we need this check just in case. Are you able to submit patch to linux-wireless?

It seems more convenient for you to submit a patch, I suggest you do that.

ernie-t commented 2 years ago

@ptpt52 @ryderlee1110 @nbd168 - are there any updates? Does anybody submit the patch?

ryderlee1110 commented 2 years ago

https://github.com/openwrt/mt76/commit/0263711ba362e36d6dcdcd2461c1a7ad7cf5098c you can close this ticket.

atanas-vladimirov commented 1 year ago

Hi,

Looks like we hit this issue on TP-Link EAP615-Wall v1

[1546759.716252] mt7915e 0000:02:00.0: Message 00005aed (seq 10) timeout                                                                                                                      
[1546780.196128] mt7915e 0000:02:00.0: Message 00005aed (seq 11) timeout                                                                                                                      
[1546800.676003] mt7915e 0000:02:00.0: Message 00005aed (seq 12) timeout                                                                                                                      
[1546821.155873] mt7915e 0000:02:00.0: Message 00005aed (seq 13) timeout                                                                                                                      
[1546841.635748] mt7915e 0000:02:00.0: Message 00005aed (seq 14) timeout

Running the 22.03.2

root@ap10:/etc/config# cat /etc/openwrt_release 
DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='22.03.2'
DISTRIB_REVISION='r19803-9a599fee93'
DISTRIB_TARGET='ramips/mt7621'
DISTRIB_ARCH='mipsel_24kc'
DISTRIB_DESCRIPTION='OpenWrt 22.03.2 r19803-9a599fee93'

As a can see this has been fixed in master, and the question is - do you plan to backport it to 22.03? Or I have to update to a master snapshot (or built it myself)?

Thanks, Atanas

simontretter commented 1 year ago

Same issue on OpenWRT snapshot r21471-b7a7a97037 with the AW7915-NP1 card.

Fri Dec 16 15:04:09 2022 kern.err kernel: [ 7217.805605] mt7915e 0000:05:00.0: Message 00005aed (seq 14) timeout Fri Dec 16 15:04:29 2022 kern.err kernel: [ 7238.285427] mt7915e 0000:05:00.0: Message 00005aed (seq 15) timeout Fri Dec 16 15:04:50 2022 kern.err kernel: [ 7258.765338] mt7915e 0000:05:00.0: Message 000026ed (seq 1) timeout Fri Dec 16 15:05:10 2022 kern.err kernel: [ 7279.245276] mt7915e 0000:05:00.0: Message 00005aed (seq 2) timeout Fri Dec 16 15:05:31 2022 kern.err kernel: [ 7299.725261] mt7915e 0000:05:00.0: Message 00005aed (seq 3) timeout Fri Dec 16 15:05:51 2022 kern.err kernel: [ 7320.205022] mt7915e 0000:05:00.0: Message 00005aed (seq 4) timeout Fri Dec 16 15:06:12 2022 kern.err kernel: [ 7340.684938] mt7915e 0000:05:00.0: Message 000026ed (seq 5) timeout Fri Dec 16 15:06:32 2022 kern.err kernel: [ 7361.164873] mt7915e 0000:05:00.0: Message 00005aed (seq 6) timeout Fri Dec 16 15:06:53 2022 kern.err kernel: [ 7381.644852] mt7915e 0000:05:00.0: Message 00005aed (seq 7) timeout Fri Dec 16 15:07:13 2022 kern.err kernel: [ 7402.124726] mt7915e 0000:05:00.0: Message 00005aed (seq 8) timeout Fri Dec 16 15:07:33 2022 kern.err kernel: [ 7422.604647] mt7915e 0000:05:00.0: Message 000026ed (seq 9) timeout Fri Dec 16 15:07:54 2022 kern.err kernel: [ 7443.084555] mt7915e 0000:05:00.0: Message 00005aed (seq 10) timeout Fri Dec 16 15:08:14 2022 kern.err kernel: [ 7463.564713] mt7915e 0000:05:00.0: Message 00005aed (seq 11) timeout Fri Dec 16 15:08:35 2022 kern.err kernel: [ 7484.044454] mt7915e 0000:05:00.0: Message 00005aed (seq 12) timeout Fri Dec 16 15:08:55 2022 kern.err kernel: [ 7504.524364] mt7915e 0000:05:00.0: Message 000026ed (seq 13) timeout Fri Dec 16 15:09:16 2022 kern.err kernel: [ 7525.004309] mt7915e 0000:05:00.0: Message 00005aed (seq 14) timeout

mateuszdraco commented 1 year ago

Same issue with Netgear WAX202:

Wed Jan 18 21:32:59 2023 kern.err kernel: [ 6322.464657] mt7915e 0000:02:00.0: Message 0000aded (seq 6) timeout
Wed Jan 18 21:33:19 2023 kern.err kernel: [ 6342.944737] mt7915e 0000:02:00.0: Message 00005aed (seq 7) timeout
Wed Jan 18 21:33:40 2023 kern.err kernel: [ 6363.424804] mt7915e 0000:02:00.0: Message 0000aded (seq 8) timeout
Wed Jan 18 21:34:00 2023 kern.err kernel: [ 6383.904880] mt7915e 0000:02:00.0: Message 00005aed (seq 9) timeout
Wed Jan 18 21:34:21 2023 kern.err kernel: [ 6404.384952] mt7915e 0000:02:00.0: Message 0000aded (seq 10) timeout
Wed Jan 18 21:34:41 2023 kern.err kernel: [ 6424.865020] mt7915e 0000:02:00.0: Message 00005aed (seq 11) timeout
Wed Jan 18 21:35:02 2023 kern.err kernel: [ 6445.345088] mt7915e 0000:02:00.0: Message 0000aded (seq 12) timeout
Wed Jan 18 21:35:22 2023 kern.err kernel: [ 6465.825154] mt7915e 0000:02:00.0: Message 00005aed (seq 13) timeout
Wed Jan 18 21:35:43 2023 kern.err kernel: [ 6486.305231] mt7915e 0000:02:00.0: Message 0000aded (seq 14) timeout
Wed Jan 18 21:36:03 2023 kern.err kernel: [ 6506.785293] mt7915e 0000:02:00.0: Message 00005aed (seq 15) timeout
Wed Jan 18 21:36:24 2023 kern.err kernel: [ 6527.265366] mt7915e 0000:02:00.0: Message 0000aded (seq 1) timeout
Wed Jan 18 21:36:44 2023 kern.err kernel: [ 6547.745440] mt7915e 0000:02:00.0: Message 00005aed (seq 2) timeout
Wed Jan 18 21:37:05 2023 kern.err kernel: [ 6568.225478] mt7915e 0000:02:00.0: Message 0000aded (seq 3) timeout
Wed Jan 18 21:37:25 2023 kern.err kernel: [ 6588.705554] mt7915e 0000:02:00.0: Message 00005aed (seq 4) timeout
Wed Jan 18 21:37:45 2023 kern.err kernel: [ 6609.185617] mt7915e 0000:02:00.0: Message 0000aded (seq 5) timeout
Wed Jan 18 21:38:06 2023 kern.err kernel: [ 6629.665661] mt7915e 0000:02:00.0: Message 00005aed (seq 6) timeout
Wed Jan 18 21:38:26 2023 kern.err kernel: [ 6650.145721] mt7915e 0000:02:00.0: Message 0000aded (seq 7) timeout
Wed Jan 18 21:38:47 2023 kern.err kernel: [ 6670.625778] mt7915e 0000:02:00.0: Message 00005aed (seq 8) timeout
Wed Jan 18 21:39:07 2023 kern.err kernel: [ 6691.105828] mt7915e 0000:02:00.0: Message 0000aded (seq 9) timeout
Wed Jan 18 21:39:28 2023 kern.err kernel: [ 6711.585871] mt7915e 0000:02:00.0: Message 00005aed (seq 10) timeout
Wed Jan 18 21:39:48 2023 kern.err kernel: [ 6732.065909] mt7915e 0000:02:00.0: Message 0000aded (seq 11) timeout
Wed Jan 18 21:40:09 2023 kern.err kernel: [ 6752.545947] mt7915e 0000:02:00.0: Message 00005aed (seq 12) timeout
Wed Jan 18 21:40:25 2023 user.info : luci: accepted login on / for root from 192.168.0.159
Wed Jan 18 21:40:29 2023 kern.err kernel: [ 6773.026006] mt7915e 0000:02:00.0: Message 00005aed (seq 13) timeout
Wed Jan 18 21:40:39 2023 authpriv.info dropbear[3097]: Child connection from 192.168.0.159:59537
Wed Jan 18 21:40:43 2023 authpriv.notice dropbear[3097]: Password auth succeeded for 'root' from 192.168.0.159:59537
Wed Jan 18 21:40:50 2023 kern.err kernel: [ 6793.506042] mt7915e 0000:02:00.0: Message 00005aed (seq 14) timeout
Wed Jan 18 21:41:10 2023 kern.err kernel: [ 6813.986074] mt7915e 0000:02:00.0: Message 00005aed (seq 15) timeout
Wed Jan 18 21:41:31 2023 kern.err kernel: [ 6834.466108] mt7915e 0000:02:00.0: Message 00005aed (seq 1) timeout
Wed Jan 18 21:41:51 2023 kern.err kernel: [ 6854.946158] mt7915e 0000:02:00.0: Message 00005aed (seq 2) timeout
Wed Jan 18 21:42:12 2023 kern.err kernel: [ 6875.436216] mt7915e 0000:02:00.0: Message 00005aed (seq 3) timeout
Wed Jan 18 21:42:32 2023 kern.err kernel: [ 6895.906246] mt7915e 0000:02:00.0: Message 00005aed (seq 4) timeout
Wed Jan 18 21:42:53 2023 kern.err kernel: [ 6916.386280] mt7915e 0000:02:00.0: Message 0000aded (seq 5) timeout
Wed Jan 18 21:43:13 2023 kern.err kernel: [ 6936.866314] mt7915e 0000:02:00.0: Message 00005aed (seq 6) timeout
Wed Jan 18 21:43:34 2023 kern.err kernel: [ 6957.346346] mt7915e 0000:02:00.0: Message 0000aded (seq 7) timeout
Wed Jan 18 21:43:54 2023 kern.err kernel: [ 6977.826391] mt7915e 0000:02:00.0: Message 00005aed (seq 8) timeout
Wed Jan 18 21:44:15 2023 kern.err kernel: [ 6998.306416] mt7915e 0000:02:00.0: Message 0000aded (seq 9) timeout
Wed Jan 18 21:44:35 2023 kern.err kernel: [ 7018.786455] mt7915e 0000:02:00.0: Message 00005aed (seq 10) timeout
Wed Jan 18 21:44:56 2023 kern.err kernel: [ 7039.266521] mt7915e 0000:02:00.0: Message 0000aded (seq 11) timeout
Wed Jan 18 21:45:16 2023 kern.err kernel: [ 7059.746527] mt7915e 0000:02:00.0: Message 00005aed (seq 12) timeout
Wed Jan 18 21:45:37 2023 kern.err kernel: [ 7080.226573] mt7915e 0000:02:00.0: Message 0000aded (seq 13) timeout
Wed Jan 18 21:45:57 2023 kern.err kernel: [ 7100.706610] mt7915e 0000:02:00.0: Message 00005aed (seq 14) timeout
Wed Jan 18 21:46:03 2023 authpriv.info dropbear[3097]: Exit (root) from <192.168.0.159:59537>: Disconnect received
Wed Jan 18 21:46:17 2023 kern.err kernel: [ 7121.186671] mt7915e 0000:02:00.0: Message 0000aded (seq 15) timeout
Wed Jan 18 21:46:26 2023 authpriv.info dropbear[3124]: Child connection from 192.168.0.159:60900
Wed Jan 18 21:46:30 2023 authpriv.notice dropbear[3124]: Password auth succeeded for 'root' from 192.168.0.159:60900
Wed Jan 18 21:46:38 2023 kern.err kernel: [ 7141.666673] mt7915e 0000:02:00.0: Message 00005aed (seq 1) timeout
Wed Jan 18 21:46:58 2023 kern.err kernel: [ 7162.146713] mt7915e 0000:02:00.0: Message 00005aed (seq 2) timeout
Wed Jan 18 21:47:19 2023 kern.err kernel: [ 7182.626755] mt7915e 0000:02:00.0: Message 000026ed (seq 3) timeout
Wed Jan 18 21:47:39 2023 kern.err kernel: [ 7203.106794] mt7915e 0000:02:00.0: Message 0000aded (seq 4) timeout
Wed Jan 18 21:48:00 2023 kern.err kernel: [ 7223.586814] mt7915e 0000:02:00.0: Message 00005aed (seq 5) timeout
Wed Jan 18 21:48:20 2023 kern.err kernel: [ 7244.066855] mt7915e 0000:02:00.0: Message 0000aded (seq 6) timeout
Wed Jan 18 21:48:41 2023 kern.err kernel: [ 7264.546879] mt7915e 0000:02:00.0: Message 00005aed (seq 7) timeout
Wed Jan 18 21:49:01 2023 kern.err kernel: [ 7285.026919] mt7915e 0000:02:00.0: Message 0000aded (seq 8) timeout
Wed Jan 18 21:49:22 2023 kern.err kernel: [ 7305.506950] mt7915e 0000:02:00.0: Message 00005aed (seq 9) timeout
Wed Jan 18 21:49:42 2023 kern.err kernel: [ 7325.986987] mt7915e 0000:02:00.0: Message 0000aded (seq 10) timeout
Wed Jan 18 21:50:03 2023 kern.err kernel: [ 7346.467031] mt7915e 0000:02:00.0: Message 00005aed (seq 11) timeout
Wed Jan 18 21:50:23 2023 kern.err kernel: [ 7366.947059] mt7915e 0000:02:00.0: Message 0000aded (seq 12) timeout
stweil commented 1 year ago

I'm afraid I have the same issue with Freifunk running on a Zyxel NWA50AX. Is it possible to dump the mt7915 eeprom from the Linux command line? I tried hexdump -n 4096 -C /dev/mtdblock2. Is that the right command? How can I see from that dump whether the antenna selection needs the fix mentioned above because of a bad eeprom?

2023-03-16

# hexdump -s 403 -n 1 -b /dev/mtdblock2
0000193 252
Brain2000 commented 1 year ago

It looks like the original commit was a clean up/refactor, but it ended up omitting one piece of logic:

pre-commit:

        if (!nss_band || nss_band > nss_band_max)
        nss_band = nss_band_max;        <-- note that it is setting nss_band

    if (nss_band > nss) {
        dev_warn(dev->mt76.dev,
             "nss mismatch, nss(%d) nss_band(%d) band(%d) ext_phy(%d)\n",
             nss, nss_band, phy->band_idx, ext_phy);
        nss = nss_band;                  <--- note that it is setting nss
    }

post-commit:

nss = min_t(u8, min_t(u8, nss_max, nss), path);     <-- note that only nss is set here

But in the reduction, as all the variables were renamed, nss_band is now nss and nss is now path.

So this logic was essentially removed with the initial commit https://github.com/openwrt/mt76/commit/ad3d0f8db00b81a1abeab9793a9d4064ffa00a04:

    if (nss > path) {
        dev_warn(dev->mt76.dev,
             "nss mismatch, nss(%d) nss_band(%d) band(%d) ext_phy(%d)\n",
             path, nss, phy->band_idx, ext_phy);
        path = nss;
    }

unless this was intentional

darmach commented 1 year ago

@csyuanc @nbd168 Can you confirm if above scenario mentioned by @Brain2000 could be the root cause?

Brain2000 commented 1 year ago

@darmach Another possible flaw I didn't notice at first:

nss = min_t(u8, min_t(u8, nss_max, nss), path);

This will set nss to the minimum value between nss_max, nss, and path. The pre-commit code never set nss to path. But because the rename occurred, this may have gotten mixed up.

mkyral commented 1 year ago

Today this happen to me on my Turris Omnia. I had to restart router via ethernet.

root@turris:/srv/home/root# grep mt7915e messages |tail
Mar 13 19:39:31 turris kernel: [554176.052299] mt7915e 0000:02:00.0: Message 0000aded (seq 8) timeout
Mar 13 19:39:52 turris kernel: [554196.520900] mt7915e 0000:02:00.0: Message 00002ced (seq 9) timeout
Mar 13 19:40:12 turris kernel: [554217.009554] mt7915e 0000:02:00.0: Message 00002ced (seq 10) timeout
Mar 13 19:40:33 turris kernel: [554237.478208] mt7915e 0000:02:00.0: Message 00005aed (seq 11) timeout
Mar 13 19:40:53 turris kernel: [554257.956872] mt7915e 0000:02:00.0: Message 00002ced (seq 12) timeout
Mar 13 19:41:14 turris kernel: [554278.445516] mt7915e 0000:02:00.0: Message 00002ced (seq 13) timeout
Mar 13 19:41:34 turris kernel: [554298.924205] mt7915e 0000:02:00.0: Message 00002ced (seq 14) timeout
Mar 13 19:41:55 turris kernel: [554319.392842] mt7915e 0000:02:00.0: Message 0000aded (seq 15) timeout
Mar 13 19:42:15 turris kernel: [554339.871491] mt7915e 0000:02:00.0: Message 0000aded (seq 1) timeout
Mar 13 19:42:36 turris kernel: [554360.360153] mt7915e 0000:02:00.0: Message 00002ced (seq 2) timeout

root@turris:/srv/home/root# uname -a
Linux turris 5.15.94 #0 SMP Fri Feb 17 00:25:12 2023 armv7l GNU/Linux

root@turris:/srv/home/root# cat /etc/os-release
NAME="TurrisOS"
VERSION="6.2.4"
ID="turrisos"
ID_LIKE="lede openwrt"
PRETTY_NAME="TurrisOS 6.2.4"
VERSION_ID="6.2.4"
HOME_URL="https://www.turris.cz/"
BUG_URL="https://gitlab.nic.cz/groups/turris/-/issues/"
SUPPORT_URL="https://www.turris.cz/support/"
BUILD_ID="r16824+125-0d4a0250df"
OPENWRT_BOARD="mvebu/cortexa9"
OPENWRT_ARCH="arm_cortex-a9_vfpv3-d16"
OPENWRT_TAINTS="busybox"
OPENWRT_DEVICE_MANUFACTURER="CZ.NIC"
OPENWRT_DEVICE_MANUFACTURER_URL="https://www.turris.cz/"
OPENWRT_DEVICE_PRODUCT="Turris Omnia"
OPENWRT_DEVICE_REVISION="v0"
OPENWRT_RELEASE="TurrisOS 6.2.4 0d4a0250df17a9f7b0fff720fa3224b45d5b0841"

root@turris:/srv/home/root# lspci -v -s 02:00.0
02:00.0 Unclassified device [0002]: MEDIATEK Corp. MT7915E 802.11ax PCI Express Wireless Network Adapter (prog-if 80)
        Subsystem: MEDIATEK Corp. MT7915E 802.11ax PCI Express Wireless Network Adapter
        Physical Slot: 2
        Flags: bus master, fast devsel, latency 0, IRQ 97
        Memory at e0400000 (64-bit, prefetchable) [size=1M]
        Memory at e0500000 (64-bit, prefetchable) [size=16K]
        Memory at e0504000 (64-bit, prefetchable) [size=4K]
        Capabilities: [80] Express Endpoint, MSI 00
        Capabilities: [e0] MSI: Enable+ Count=1/32 Maskable+ 64bit+
        Capabilities: [f8] Power Management version 3
        Capabilities: [100] Vendor Specific Information: ID=1556 Rev=1 Len=008 <?>
        Capabilities: [108] Latency Tolerance Reporting
        Capabilities: [110] L1 PM Substates
        Capabilities: [200] Advanced Error Reporting
        Kernel driver in use: mt7915e
lspci: Unable to load libkmod resources: error -12
webknjaz commented 1 year ago

I haven't seen this on my Turris Omnia (TurrisOS 6.3.0 c874aa40c22bec81643af57413c3f9ad613aeced r16833+124-c874aa40c2). But I do have a few TOTOLINK X5000R devices set up to run OpenWRT 21.02.5 (snapshot r16688-fa9a932fdb) as dumb APs and they exhibit the same problem. This also seems to make the devices flakily inaccessible over the wired network too — the HTTP access times out. I managed to connect over SSH and noticed load average being close to 5.0. Using top, I saw that hostapd and dawn are in the D state which is concerning. Also, [mt76-tx phy0] remains in the SW state...

hami89 commented 1 year ago

Exact same behavior experienced with my Linksys e8450 (OpenWrt 22.03.3 r20028-43d71ad93e / LuCI openwrt-22.03 branch git-23.069.50944-cd8bea9) running as router of a 3 unit setup with WIFI roaming (other two units are same hardware, but running as dumb APs). Like @webknjaz wrote, SSH access is working and hostapd and dawn are indeed in D state.

rany2 commented 1 year ago

I know correlation does not imply causation so this might be a freak coincidence but I only started facing this issue within a few hours of booting up after deploying 802.11r and DAWN, before that change it was working reliably for more than week. After uninstalling DAWN, the issue seems gone. I'm not sure what to make of this information but it might help the developers to see what DAWN is doing that causes the chip to freak out.

Edit: P.S. I'm on SNAPSHOT. Edit2: I was able to have this issue occur more than 3 times with DAWN installed back-to-back, once DAWN was removed the issue stopped occurring.

Brain2000 commented 1 year ago

@rany2 Aha... Dawn ... That's a really good point.

Based on observation, I've been leaning towards something being wrong with multicast, particularly mDNS. iPhones cause crashes when a lot of them are roaming. I can manually cause a crash with a laptop in under 10 seconds by sending a large amount of mDNS packets and then either turning on airplane mode or roam (if I stay connected and stop the multicast stream, everything remains fine).

I am going to disable Dawn and see what happens. Clearly Dawn should not cause an issue, and switching it from mDNS to broadcast might be a temporary workaround until the underlying issue can be found.

rany2 commented 1 year ago

@Brain2000 DAWN could very well be the issue, it actively sends beacon requests to STAs via a ubus hostapd interface and passively listens to beacons from STAs. I don't think it's MDNS related.

Of course, DAWN isn't the source but it helped shed a light on some bug in the firmware or kernel driver. Whatever it is, the router I had was working flawlessly up until I had setup roaming and DAWN. DAWN being uninstalled seems to have gotten rid of the issue.

stweil commented 1 year ago

My routers get these timeouts without DAWN, but I run Freifunk Gluon with a wireless mesh. I get the timeouts much sooner on routers with heavy wireless traffic.

hami89 commented 1 year ago

My 3-piece system (3 Linksys e8450 - 1 as router + 2 as dumb APs with DAWN) worked flawlessly for almost three weeks, then last week it experienced this issue again. During those three weeks, I had guests, so we had several computers and phones roaming around the house, streaming media etc. So I see no connection between the actual load or number of clients and this effect. I did not see any problems during that period, surprisingly, the system failed when only I was home.

I could reconnect to the network through one of the dumb APs, so I was able to connect to the router (controller). I could log into luci, but the page load timed out every time, but I could get in SSH. Again, hostapd and dawn were in D state (checked in top).

I checked logs and tried to restart networking, without success, so after 5 mins I restarted the whole system as it occurred during a company call and I had to get back,

Anyone interested, I have a dump of logs (some comments and the network restart attempt added in between). Errors in log started to show up around Wed Apr 19 11:13:12 2023. openwrt-logs-prepared.txt

rany2 commented 1 year ago

@hami89 Interesting. On my end I've noticed that this issues comes up only with 802.11r and DAWN setup. I was reliably getting triggering it with 802.11r and DAWN setup. DAWN without 802.11r doesn't have an issue. It's solely when 802.11r is setup alongside DAWN that this issue gets triggered extremely reliably.

No idea what to make of this information :/

rany2 commented 1 year ago

At any rate I'm getting this issue in under 2 hours and sometimes within 30 minutes of bootup, so it seems a different situation than mine.