search

openwrt / mt76

mac80211 driver for MediaTek MT76x0e, MT76x2e, MT7603, MT7615, MT7628 and MT7688
731 stars 334 forks source link

mt7615e: WIFI crashes under MESH with VHT80 and above #743

Open kaijajan opened 1 year ago

kaijajan commented 1 year ago

I encountered the following crashes using MESH with mt7615e(redmi ac2100 * 2)

environment: redmi ac2100-1: wired, WIFI-AC1 redmi ac2100-2: wireless, WIFI-AC2, ac mesh connected to redmi ac2100-1

The MESH works fine with 40Mhz but the throughput is bad. Once switch to 80Mhz, performance gets better of course, but the MESH would crash if connected to the WIFI-AC2 (message occurred on redmi ac2100-1, and nothing on redmi ac2100-2)

Both redmi ac2100 are running openwrt 22.03.03 (also tried the latest snapshot, same issue), and crashes occurred immediately on transmission connected to the WIFI AC-2. And nothing happens if connected to the WIFI-AC1.

Once the issue occurred, reboots are required for both routers.

[  362.770025] ------------[ cut here ]------------
[  362.774692] WARNING: CPU: 2 PID: 30 at target-mipsel_24kc_musl/linux-ramips_mt7621/mt76-2022-09-06-d7054646/mt7615/pci_mac.c:161 mt7615_dma_reset+0x270/0x290 [mt7615e]
[  362.789632] Timeout waiting for MCU reset state 8
[  362.794493] Modules linked in: pppoe ppp_async nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet batman_adv pppox ppp_generic nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack mt7615e mt7615_common mt7603e mt76_connac_lib mt76 mac80211 cfg80211 slhc nfnetlink nf_reject_ipv6 nf_reject_ipv4 nf_log_ipv6 nf_log_ipv4 nf_log_common nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c hwmon crc_ccitt compat sha256_generic libsha256 seqiv jitterentropy_rng drbg hmac cmac leds_gpio gpio_button_hotplug crc32c_generic
[  362.859608] CPU: 2 PID: 30 Comm: kworker/u8:1 Not tainted 5.10.161 #0
[  362.866062] Workqueue: mt76 mt7615_mac_reset_work [mt7615e]
[  362.871617] Stack : 809e0000 825eb680 80d11cfc 80083590 80735b58 80800000 80cf8000 80730000
[  362.879966]         00000009 825ea040 00000000 80083730 807fee27 00000001 80d11cb8 11419477
[  362.888316]         00000000 00000000 80735b58 80d11b40 ffffefff 00000000 ffffffea 00000000
[  362.896659]         80d11b4c 00000151 80804828 ffffffff 00000000 00000001 00000000 80730000
[  362.905007]         00000009 825ea040 00000000 825eb680 00000018 803f36e8 00000008 809c0008
[  362.913354]         ...
[  362.915794] Call Trace:
[  362.918250] [<800080e0>] show_stack+0x30/0x100
[  362.922703] [<803717f0>] dump_stack+0x9c/0xcc
[  362.927061] [<8003007c>] __warn+0xc0/0x12c
[  362.931144] [<80030174>] warn_slowpath_fmt+0x8c/0xac
[  362.936096] [<827b95f4>] mt7615_dma_reset+0x270/0x290 [mt7615e]
[  362.942020] [<80075c40>] autoremove_wake_function+0x0/0x58
[  362.947478]
[  362.949155] ---[ end trace 6632f3f93c576c1a ]---
[  393.490042] ------------[ cut here ]------------
[  393.494711] WARNING: CPU: 2 PID: 30 at target-mipsel_24kc_musl/linux-ramips_mt7621/mt76-2022-09-06-d7054646/mt7615/pci_mac.c:161 mt7615_dma_reset+0x270/0x290 [mt7615e]
[  393.509628] Timeout waiting for MCU reset state 20
[  393.514426] Modules linked in: pppoe ppp_async nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet batman_adv pppox ppp_generic nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack mt7615e mt7615_common mt7603e mt76_connac_lib mt76 mac80211 cfg80211 slhc nfnetlink nf_reject_ipv6 nf_reject_ipv4 nf_log_ipv6 nf_log_ipv4 nf_log_common nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c hwmon crc_ccitt compat sha256_generic libsha256 seqiv jitterentropy_rng drbg hmac cmac leds_gpio gpio_button_hotplug crc32c_generic
[  393.579535] CPU: 2 PID: 30 Comm: kworker/u8:1 Tainted: G        W         5.10.161 #0
[  393.587365] Workqueue: mt76 mt7615_mac_reset_work [mt7615e]
[  393.592918] Stack : 809e0000 825eb680 80d11cfc 80083590 80735b58 80800000 80cf8000 80730000
[  393.601271]         00000009 825ea040 00000000 80083730 807fee27 00000001 80d11cb8 11419477
[  393.609616]         00000000 00000000 80735b58 80d11b40 ffffefff 00000000 ffffffea 00000000
[  393.617959]         80d11b4c 00000166 80804828 ffffffff 00000000 00000001 00000000 80730000
[  393.626306]         00000009 825ea040 00000000 825eb680 00000018 803f36e8 00000008 809c0008
[  393.634655]         ...
[  393.637098] Call Trace:
[  393.639554] [<800080e0>] show_stack+0x30/0x100
[  393.644006] [<803717f0>] dump_stack+0x9c/0xcc
[  393.648364] [<8003007c>] __warn+0xc0/0x12c
[  393.652448] [<80030174>] warn_slowpath_fmt+0x8c/0xac
[  393.657400] [<827b95f4>] mt7615_dma_reset+0x270/0x290 [mt7615e]
[  393.663329] [<80075c40>] autoremove_wake_function+0x0/0x58
[  393.668787]
[  393.670449] ---[ end trace 6632f3f93c576c1b ]---
[  413.970042] mt7615e 0000:01:00.0: Message 000049ed (seq 3) timeout
[  434.450040] mt7615e 0000:01:00.0: Message 000049ed (seq 4) timeout
[  454.930041] mt7615e 0000:01:00.0: Message 000049ed (seq 5) timeout
[  481.170038] mt7615e 0000:01:00.0: Message 000049ed (seq 6) timeout
[  507.410038] mt7615e 0000:01:00.0: Message 000049ed (seq 7) timeout
[  533.650032] mt7615e 0000:01:00.0: Message 000049ed (seq 8) timeout
[  559.890028] mt7615e 0000:01:00.0: Message 000049ed (seq 9) timeout
[  586.130045] mt7615e 0000:01:00.0: Message 000049ed (seq 10) timeout
[  612.370031] mt7615e 0000:01:00.0: Message 000049ed (seq 11) timeout
[  638.610023] mt7615e 0000:01:00.0: Message 000049ed (seq 12) timeout
[  664.850017] mt7615e 0000:01:00.0: Message 000049ed (seq 13) timeout
[  691.090028] mt7615e 0000:01:00.0: Message 000049ed (seq 14) timeout
[  717.330012] mt7615e 0000:01:00.0: Message 000049ed (seq 15) timeout
[  743.570010] mt7615e 0000:01:00.0: Message 000049ed (seq 1) timeout
[  769.810007] mt7615e 0000:01:00.0: Message 000049ed (seq 2) timeout
[  796.050004] mt7615e 0000:01:00.0: Message 000049ed (seq 3) timeout
[  822.290016] mt7615e 0000:01:00.0: Message 000049ed (seq 4) timeout
lukasz1992 commented 1 year ago

Use snapshot and try again

kaijajan commented 1 year ago

Thx for the reply.

Just tested the latest snapshot (commit: 9639ef2a5b20a912916c8b9b0dd8a37cd8f6beb5) mt76 driver version: (package/kernel/mt76/Makefile)

PKG_SOURCE_URL:=https://github.com/openwrt/mt76
PKG_SOURCE_PROTO:=git
PKG_SOURCE_DATE:=2023-02-02
PKG_SOURCE_VERSION:=b2360d59747c6fed2b65bc1c3563c10593c83f3e
PKG_MIRROR_HASH:=f85f00fa8f549de348a115f92c4f0235e413ed924669aece390c7570882526dd

The output of dmesg was different, but crashes that occurred under VHT80 were the same. And the result was captured in WIFI-AC1, (sometimes WIFI-AC2, not always the same)

result:

[  572.042801] ------------[ cut here ]------------
[  572.047671] WARNING: CPU: 0 PID: 761 at backports-6.1-rc8/net/mac80211/rx.c:5197 ieee80211_rx_list+0x654/0xedc [mac80211]
[  572.058634] Rate marked as a VHT rate but data is invalid: MCS: 47, NSS: 3
[  572.065729] Modules linked in: pppoe ppp_async nft_fib_inet nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet batman_adv pppox ppp_generic nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_objref nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_counter nft_chain_nat nf_tables nf_nat nf_flow_table nf_conntrack mt7615e mt7615_common mt7603e mt76_connac_lib mt76 mac80211 cfg80211 slhc nfnetlink nf_reject_ipv6 nf_reject_ipv4 nf_log_ipv6 nf_log_ipv4 nf_log_common nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c hwmon crc_ccitt compat sha256_generic libsha256 seqiv jitterentropy_rng drbg hmac cmac leds_gpio gpio_button_hotplug crc32c_generic
[  572.130972] CPU: 0 PID: 761 Comm: napi/phy1-9 Not tainted 5.10.167 #0
[  572.137420] Stack : 809e0000 00000001 84216aa8 800826ec 3ffff158 00e71f59 00000000 00000000
[  572.145773]         8267bbf4 809c0000 8073387c 824b89d0 807fae47 00000001 8267bb98 00e71f59
[  572.154119]         00000000 00000000 8073387c 8267ba38 ffffefff 00000000 ffffffea 00000000
[  572.162463]         8267ba44 0000015a 80800920 ffffffff 00000000 00000001 00000000 80730000
[  572.170806]         00000009 00000000 82c3f840 00000001 00000018 803f39d8 00000000 809c0000
[  572.179152]         ...
[  572.181591] Call Trace:
[  572.184052] [<80008068>] show_stack+0x30/0x100
[  572.188502] [<80371ff4>] dump_stack+0xa0/0xd0
[  572.192857] [<8002f4e4>] __warn+0x9c/0x124
[  572.196938] [<8002f5f8>] warn_slowpath_fmt+0x8c/0xac
[  572.201992] [<827358e0>] ieee80211_rx_list+0x654/0xedc [mac80211]
[  572.208122] [<82545328>] mt76_rx_complete+0x1c4/0x298 [mt76]
[  572.213771] [<82545714>] mt76_rx_poll_complete+0x318/0x650 [mt76]
[  572.219849] [<825423d8>] mt76_dma_rx_poll+0x274/0x4dc [mt76]
[  572.225496] [<827b4684>] mt7615_unregister_device+0x27c/0x334 [mt7615e]
[  572.232080]
[  572.233827] ---[ end trace 00e4484491ba3682 ]---
[  595.082810] mt7615e 0000:01:00.0: Message 000049ed (seq 14) timeout
[  621.322550] mt7615e 0000:01:00.0: Message 000049ed (seq 15) timeout
[  647.561583] mt7615e 0000:01:00.0: Message 000049ed (seq 1) timeout
[  673.800221] mt7615e 0000:01:00.0: Message 000049ed (seq 2) timeout
[  700.039054] mt7615e 0000:01:00.0: Message 000049ed (seq 3) timeout