nbd168
commented
4 years ago Could you please bisect this? Thanks
Open dangowrt opened 4 years ago
nbd168
commented
4 years ago Could you please bisect this? Thanks
dangowrt
commented
4 years ago Turns out to be unrelated to recent commits but was just due to the fast connectivity I enjoyed in a specific environment. It reliably occurs when RX goes beyond ~35MBit/s, sitting right next to the AP, then even with disable_ht=1 and disable_vht=1 802.11a 54MBit/s data rates would kill it. It then ends up in a reset-loop (see logs above), even unloading mt76x2e kernel module doesn't help at this point, only reboot does. Probably Linux doesn't control PCI hardware reset but it's done by firmware.
nbd168
commented
4 years ago The reset loop should be fixed now, please test
pfjgeraedts
commented
4 years ago I had similar experience as dangowrt. Now from OpenWrt master 0f8c806eb8 (that includes your fix) the reset loops seem to have turned into series of resets (that stop when data transfer is interrupted); I may have encountered a single reset loop, but I cannot confirm this. Some details: I have a remote setup with two MT7612E (one in an x86_32 and one in an x86_64), both running the above OpenWrt state (same 32bit image for both). The x86_64 is setup as AP and the x86_32 as STA. Only when disabling (V)HT modes data transfer is steady (at ~20 Mbit/s as reported by iperf). Higher modes result in a series of resets and (<=) Kbit/s rates. After a series of resets, disabling (V)HT also results in Kbit/s rates; rebooting does help. This behavior occurs in both directions. As a cross check I connected the STA with another AP (ath9k) at HT40 (Netgear WNDR3800 running OpenWrt 19.07.1; ath79). When STA is running iperf as server, reception rates is again at Kbit/s, but when running as client transmission rates is at ~190 Mbit/s. Promising! Please let me know how I can be of assistance to further debug this. Thanks.
pfjgeraedts
commented
4 years ago After updating to OpenWrt master fd0cc72d9c (that includes upto and including mt76 master 85c516081338) the series of resets seem to have turned into a single reset followed by a kernel panic.
Again I connected the STA with another AP (ath9k) at HT40 (Netgear WNDR3800 now running at OpenWrt 19.07.2; ath79). When STA is running iperf as client, transmission is still fine. When STA is running iperf as server, the following kernel messages result.
[ 75.652197] mt76x2e 0000:02:00.0: MCU message 31 (seq 11) timed out [ 75.653745] BUG: kernel NULL pointer dereference, address: 0000000c [ 75.655316] #PF: supervisor read access in kernel mode [ 75.656124] #PF: error_code(0x0000) - not-present page [ 75.656124] pdpt = 0000000000000000 pde = f000eef3f000eef3 [ 75.656124] Oops: 0000 [#1] SMP NOPTI [ 75.656124] CPU: 0 PID: 962 Comm: kworker/u4:6 Not tainted 5.4.45 #0 [ 75.656124] Hardware name: MICRO-STAR INTERNATIONAL CO., LTD U90/U100/U90/U100, BIOS 4.6.3 12/01/2009 [ 75.656124] Workqueue: phy0 mt76x02_wdt_work [mt76x02_lib] [ 75.656124] EIP: hrtimer_active+0xb/0x50 [ 75.656124] Code: 6f 00 29 c6 8b 43 20 19 d7 8b 14 24 8b 00 e8 1c 4c 6f 00 89 f0 89 fa 83 c4 04 5b 5e 5f 5d c3 55 89 e5 8d 74 26 00 90 8b 50 20 <8b> 4a 0c f6 c1 01 75 1d 80 78 24 00 75 27 39 42 10 74 22 39 4a 0c [ 75.656124] EAX: f50f2860 EBX: f50f2860 ECX: f6204038 EDX: 00000000 [ 75.656124] ESI: 00000000 EDI: f5f8c9c0 EBP: f5da9ee0 ESP: f5da9ee0 [ 75.656124] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00010286 [ 75.656124] CR0: 80050033 CR2: 0000000c CR3: 01bf2000 CR4: 000006f0 [ 75.656124] Call Trace: [ 75.656124] hrtimer_try_to_cancel+0x11/0x80 [ 75.656124] hrtimer_cancel+0xd/0x20 [ 75.656124] napi_disable+0x5c/0x70 [ 75.656124] mt76x02_wdt_work+0x136/0x520 [mt76x02_lib] [ 75.656124] ? mt76x2_register_device+0x2a0/0x2f0 [mt76x2e] [ 75.656124] process_one_work+0x1b2/0x2c0 [ 75.656124] worker_thread+0x37/0x400 [ 75.656124] kthread+0xe6/0x100 [ 75.656124] ? process_one_work+0x2c0/0x2c0 [ 75.656124] ? kthread_park+0x90/0x90 [ 75.656124] ret_from_fork+0x19/0x38 [ 75.656124] Modules linked in: pppoe ppp_async mt76x0u mt76x0e mt76x0_common iptable_nat xt_state xt_nat xt_conntrack xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD pppox ppp_generic nf_nat nf_flow_table_hw nf_flow_table nf_conntrack_rtcache nf_conntrack mt7915e mt76x2u mt76x2e mt76x2_common mt76x02_usb mt76x02_lib mt7663u mt7615e mt7615_common mt7603e mt7601u mt76_usb mt76 mac80211 ipt_REJECT cfg80211 xt_time xt_tcpudp xt_multiport xt_mark xt_mac xt_limit xt_comment xt_TCPMSS xt_LOG via_velocity via_rhine slhc sis900 r8169 pcnet32 nf_reject_ipv4 nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 ne2k_pci iptable_mangle iptable_filter ip_tables forcedeth e100 crc_ccitt compat 8390 8139too nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 3c59x e1000 vfat fat nls_utf8 nls_iso8859_1 nls_cp437 natsemi button_hotplug tg3 ptp realtek pps_core mii libphy [ 75.656124] CR2: 000000000000000c [ 75.656124] ---[ end trace bc2b9ea75c64c356 ]--- [ 75.656124] EIP: hrtimer_active+0xb/0x50 [ 75.656124] Code: 6f 00 29 c6 8b 43 20 19 d7 8b 14 24 8b 00 e8 1c 4c 6f 00 89 f0 89 fa 83 c4 04 5b 5e 5f 5d c3 55 89 e5 8d 74 26 00 90 8b 50 20 <8b> 4a 0c f6 c1 01 75 1d 80 78 24 00 75 27 39 42 10 74 22 39 4a 0c [ 75.656124] EAX: f50f2860 EBX: f50f2860 ECX: f6204038 EDX: 00000000 [ 75.656124] ESI: 00000000 EDI: f5f8c9c0 EBP: f5da9ee0 ESP: f5da9ee0 [ 75.656124] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068 EFLAGS: 00010286 [ 75.656124] CR0: 80050033 CR2: 0000000c CR3: 01bf2000 CR4: 000006f0 [ 75.656124] Kernel panic - not syncing: Fatal exception [ 75.656124] Kernel Offset: disabled [ 75.656124] Rebooting in 3 seconds.. [ 75.656124] ACPI MEMORY or I/O RESET_REG.
pfjgeraedts
commented
3 years ago After updating to OpenWrt master 4682d4d770 (that includes upto and including mt76 master b22977c2727d) the series of resets seem to have returned; I can even trigger it with (V)HT modes disabled now.
It is as if the series of resets get triggered when the incoming data rate exceeds a certain threshold; The outgoing data rate seems to be unaffected; i.e. I now see the same as what dangowrt already described on Feb, 24.
Also, the MT7612U seems unaffected, while both chips seem to run the same firmware; e.g. when I let the (x86_64) AP switch from the MT7612E to the MT7612U and let the (x86_32) STA run iperf as client transmission rate is up to ~310 Mbit/s!*
It could well be that all this is strictly PCI interface related, like dangowrt already hinted at.
Please let me know when there is anything I can do to help solve this.
*Although the earlier reported ~190 Mbit/s is a steady flow, this ~310 Mbit/s is not; my working assumption is that the accompanying incoming data rate exceeds the described threshold in this latter case.
dangowrt
commented
3 years ago I'm also still seeing this with recent mt76 and developed another theory which I will try to pursue once I have measurement tools for that around: mPCIe of laptops build for other WiFi modules might not provide enough power. according to stackexchange quoting the mPCIe spec: 1100 mA (2750 mA) normal (peak) current on 3.3V. That's quite a lot and certainly more than those centrino modules typically found in laptops would draw... Intuitively I'd also rather expect that to affect TX first of all, but I'm not into the electronic details here and just wanted to share that thought... It'd be interesting to try mt76 on x86 systems with supposedly more spec conforming mPCIe slots (APU board or such) and see if we can reproduce it there as well.
dangowrt
commented
3 years ago In recent versions, resets still happen but the card then in most cases subsequently recovers (at least a bit) and functions again after the reset, just with much lower transfer rates and more frequent repetitions of that reset...
pfjgeraedts
commented
3 years ago Indeed good to keep options open and yes, I would also not expect RX to draw most power. Three laptops and one mPCIe card would also lead me to suspect the card and not the laptops.
If I recall correctly ath10k chips/drivers early on had PCI related quirks. Maybe this case is something similar? I am not aware of what detail can be observed from the kernel side when it comes to the state of the PCI bus. Maybe enough to figure out what is going on?*
I am also curious if the only difference between the MT7612E and MT7612U is the host interface.
*My background in IC design unfortunately limits me here, as I have not yet been able to find any serious time to spend on low-level Linux stuff and gain sufficient experience.
Previously wifi was usable in station mode with disable_ht=1 and disable_vht=1 set. Now, after recent commits (i'm on fd892bc033fbfad7b1a5bcba1432709a74327a1e), it crashes immediately after connecting once any traffic goes over the link:
Running:
The same hardware works fine with the iwlwifi card which originally shipped with it and also ath9k-based mPCIe modules work flawlessly.