search

openwrt / openwrt

This repository is a mirror of https://git.openwrt.org/openwrt/openwrt.git It is for reference only and is not active for check-ins. We will continue to accept Pull Requests here. They will be merged via staging trees then into openwrt.git.
Other
19.66k stars 10.26k forks source link

[MT7530/MT7621] RX speed progressively gets worse with time #12914

Closed rany2 closed 1 year ago

rany2 commented 1 year ago

Describe the bug

Issue occurs with all versions after OpenWRT 21.02

Over time, the Ethernet performance gets progressively worse until the situation is unbearable. Sometimes it starts off extremely poorly from the get-go but other times it has a good and healthy start of 800Mbit/s only to degrade to ~480Mbit/s and worse.

The dmesg (even with CONFIG_DEVICE_DEBUG and CONFIG_PCI_DEBUG) are identical to each other from when it starts off healthy and when it starts off poorly. Completely power cycling the device does not make a difference either, the issue occurs seemingly at random. In both cases even if I was fortunate enough to have it working from startup, it will eventually break again and have abysmal performance.

An increase in the number of ERR in /proc/interrupts is also observed between OpenWRT 21.02 and subsequent versions: https://github.com/openwrt/openwrt/issues/12914#issuecomment-1596320132

The temperature does not appear to be related, a measurement shows ~50 C so it is not throttling itself it seems.

NOTE: when I say "but other times it has a good" this only occurs maybe about 1 in 10 times but as previously stated, it doesn't matter; it will eventually regress typically under an hour.

NOTE 2: the device is a dumb AP and no other services besides hostapd, dawn, and uhttpd are running.

NOTE 3: VLAN filtering is not used.

OpenWrt version

r23376-bc3af3b858

OpenWrt target/subtarget

ramips/mt7621

Device

ASUS RT-AX53U

Image kind

Self-built image

Steps to reproduce

  1. Run iperf3 over extended period of time
  2. Watch performance start off decentish at ~800Mbit/s (or poorly if you weren't lucky at bootup)
  3. Try again in a few minutes/hours
  4. Note difference in bitrate

Actual behaviour

No response

Expected behaviour

Performance shouldn't just drop out of nowhere.

Additional info

No response

Diffconfig

No response

Terms

brada4 commented 1 year ago

Do you see any error counters rising in ethtool -S eth0 or ifconfig eth0 ? Like save both outputs, do test until slow and compare (diff) with new outputs.

rany2 commented 1 year ago

@brada4 

root@asus:~# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr FC:34:97:0B:39:70  
          inet6 addr: fe80::fe34:97ff:fe0b:3970/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:2026  Metric:1
          RX packets:9540015 errors:0 dropped:0 overruns:0 frame:0
          TX packets:971968 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:14714527840 (13.7 GiB)  TX bytes:243472624 (232.1 MiB)
          Interrupt:21 

root@asus:~# ethtool -S eth0
NIC statistics:
     tx_bytes: 243477840
     tx_packets: 971989
     tx_skip: 0
     tx_collisions: 0
     rx_bytes: 14714538001
     rx_packets: 9540035
     rx_overflow: 0
     rx_fcs_errors: 0
     rx_short_errors: 0
     rx_long_errors: 0
     rx_checksum_errors: 0
     rx_flow_control_packets: 0
     rx_xdp_redirect: 0
     rx_xdp_pass: 0
     rx_xdp_drop: 0
     rx_xdp_tx: 0
     rx_xdp_tx_errors: 0
     tx_xdp_xmit: 0
     tx_xdp_xmit_errors: 0
     p06_TxDrop: 0
     p06_TxCrcErr: 0
     p06_TxUnicast: 9536051
     p06_TxMulticast: 3285
     p06_TxBroadcast: 699
     p06_TxCollision: 0
     p06_TxSingleCollision: 0
     p06_TxMultipleCollision: 0
     p06_TxDeferred: 0
     p06_TxLateCollision: 0
     p06_TxExcessiveCollistion: 0
     p06_TxPause: 0
     p06_TxPktSz64: 0
     p06_TxPktSz65To127: 157697
     p06_TxPktSz128To255: 104018
     p06_TxPktSz256To511: 241250
     p06_TxPktSz512To1023: 24126
     p06_Tx1024ToMax: 9012944
     p06_TxBytes: 14752698141
     p06_RxDrop: 0
     p06_RxFiltering: 14
     p06_RxUnicast: 967368
     p06_RxMulticast: 3974
     p06_RxBroadcast: 647
     p06_RxAlignErr: 0
     p06_RxCrcErr: 0
     p06_RxUnderSizeErr: 0
     p06_RxFragErr: 0
     p06_RxOverSzErr: 0
     p06_RxJabberErr: 0
     p06_RxPause: 11518
     p06_RxPktSz64: 11518
     p06_RxPktSz65To127: 732209
     p06_RxPktSz128To255: 94775
     p06_RxPktSz256To511: 25461
     p06_RxPktSz512To1023: 16643
     p06_RxPktSz1024ToMax: 102901
     p06_RxBytes: 244214992
     p06_RxCtrlDrop: 0
     p06_RxIngressDrop: 0
     p06_RxArlDrop: 0
rany2 commented 1 year ago

Maybe the ethtool stats are wrong? There is no way there are no packets dropped.

rany2 commented 1 year ago

Here are the ethtool stats for the only port connected to the device:

root@asus:~# ethtool -S lan2
NIC statistics:
     tx_packets: 1197734
     tx_bytes: 251891696
     rx_packets: 14841316
     rx_bytes: 24801706385
     TxDrop: 0
     TxCrcErr: 0
     TxUnicast: 1241843
     TxMulticast: 2125
     TxBroadcast: 334
     TxCollision: 0
     TxSingleCollision: 0
     TxMultipleCollision: 0
     TxDeferred: 0
     TxLateCollision: 0
     TxExcessiveCollistion: 0
     TxPause: 0
     TxPktSz64: 7712
     TxPktSz65To127: 995898
     TxPktSz128To255: 95430
     TxPktSz256To511: 24690
     TxPktSz512To1023: 17069
     Tx1024ToMax: 103503
     TxBytes: 260450409
     RxDrop: 90436
     RxFiltering: 0
     RxUnicast: 14927696
     RxMulticast: 3358
     RxBroadcast: 699
     RxAlignErr: 0
     RxCrcErr: 0
     RxUnderSizeErr: 0
     RxFragErr: 0
     RxOverSzErr: 0
     RxJabberErr: 0
     RxPause: 0
     RxPktSz64: 9533
     RxPktSz65To127: 257892
     RxPktSz128To255: 108847
     RxPktSz256To511: 241675
     RxPktSz512To1023: 27653
     RxPktSz1024ToMax: 14286153
     RxBytes: 25206069331
     RxCtrlDrop: 0
     RxIngressDrop: 90436
     RxArlDrop: 0
rany2 commented 1 year ago

When checking the lan2 DSA port's ethtool stats I could confirm that the RX dropped stats went from 109398 to 118852 while doing a 10 second iperf test. So it seems like there a lot of failures when it reaches 480Mbit/s. Rx drop counter does not increase when not under load.

brada4 commented 1 year ago

That seems like flow control in the switch overreacting. In principle there shoud be some drops if you approach CPU gigabit port from 2 other gigabit ports via switch that should induce slowing down TCP connections and sending flow control packets to clients supporting that, but not in the clean environment. Can you try installing irqbalance and/or enabling network packet steering? Unlikely to help but who knows.

rany2 commented 1 year ago

I should have noted this but I tried that. This is with irqbalance and packet steering enabled. It occurred with those two disabled as well.

brada4 commented 1 year ago

Lets disable all flow control in path: ethtool -a eth0 ethtool -A eth0 auto off tx off rx off also lanX port(s) involved. TCP should gracefully survive small drop rate on connection and slow down transmission per connection, not inflicting damage on low speed connections around.

rany2 commented 1 year ago

I just checked and I have realized that none of my devices support sending pause frames but somehow this is the only device that registers that it has received something on the system port? (Edit: might be a red herring though)

brada4 commented 1 year ago

Switch ASIC may drop few packets from all ports receiving pause frame from invisible system ports. Most enterprise switches do not support any flow control at gigabit.

rany2 commented 1 year ago

Lets disable all flow control in path: ethtool -a eth0 ethtool -A eth0 auto off tx off rx off also lanX port(s) involved. TCP should gracefully survive small drop rate on connection and slow down transmission per connection, not inflicting damage on low speed connections around.

No luck with your suggestion, no impact unfortunately

ptpt52 commented 1 year ago

how to run the test?

iperf3 client --- lan1[Router]wan --- iperf3 server

hardware offload enable?

rany2 commented 1 year ago

@ptpt52 directly via router, and there is no hardware offloading enabled.

Edit: performance between switch ports is OK, getting gigabit speeds there

rany2 commented 1 year ago

Something I tried was adding a delay after gpio reset because this what OEM driver did, but it does not help; but the fact that it sometimes works on boot up only to degrade with time tells me something is broken in initialization

Edit: also try to bring down mac5 and 6 before sw reset like mt7531 as the OEM driver did that for mt7530 but it didn't help either.

rany2 commented 1 year ago

I just noticed that https://github.com/openwrt/openwrt/blob/main/target/linux/generic/pending-6.1/723-net-mt7531-ensure-all-MACs-are-powered-down-before-r.patch is wrong. It should be PMCR_FORCE_LNK and that change impacts mt7530_setup not mt7531_setup so patch name is wrong as well.

rany2 commented 1 year ago

I just noticed that https://github.com/openwrt/openwrt/blob/main/target/linux/generic/pending-6.1/723-net-mt7531-ensure-all-MACs-are-powered-down-before-r.patch is wrong. It should be PMCR_FORCE_LNK and that change impacts mt7530_setup not mt7531_setup so patch name is wrong as well.

OEM driver uses PMCR_FORCE_MODE actually

rany2 commented 1 year ago

it doesn't make a difference anyway, issue remains

brada4 commented 1 year ago

Ethrool -k offluads?

rany2 commented 1 year ago 
# ethtool -k eth0 | grep -v ': off'
Features for eth0:
rx-checksumming: on
tx-checksumming: on
    tx-checksum-ipv4: on
    tx-checksum-ipv6: on
scatter-gather: on
    tx-scatter-gather: on
tcp-segmentation-offload: on
    tx-tcp-segmentation: on
    tx-tcp-ecn-segmentation: on
    tx-tcp-mangleid-segmentation: on
    tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
tx-vlan-offload: on
hw-tc-offload: on

(When I said "not using hardware offloading" I was referring to flow offloading, just fyi)

brada4 commented 1 year ago

(i got that)

Disable offloads one by one (starting from bottom) and do quick measurement if ot magically starts working at some point.

rany2 commented 1 year ago

yeah no luck with that, issue remains..

rany2 commented 1 year ago

@dangowrt @nbd168 When under load for longer periods of time, this sometimes ends up happening. Could this help with understanding the issue?

[  197.095778] Unhandled kernel unaligned access[#1]:
[  197.100617] CPU: 3 PID: 271 Comm: napi/mtk_eth-6 Not tainted 5.15.117 #0
[  197.107309] $ 0   : 00000000 00000001 00000122 80b7ca9c
[  197.112543] $ 4   : fffffffe 00000000 81027bb8 00000000
[  197.117774] $ 8   : 0061d6ef 00000040 00000039 8155e3c0
[  197.122996] $12   : 00000b97 000034fc 00000000 00000000
[  197.128220] $16   : 80b1dc20 00000000 80b2c620 00150015
[  197.133442] $20   : 81027bb8 80b7ca98 80150015 8141df00
[  197.138669] $24   : 00000000 8032e3b8                  
[  197.143893] $28   : 81d7c000 81d7dcb8 00000000 801b1140
[  197.149115] Hi    : 00000000
[  197.151981] Lo    : 00000041
[  197.154844] epc   : 801b1374 ___slab_alloc.constprop.0+0x530/0x790
[  197.161032] ra    : 801b1140 ___slab_alloc.constprop.0+0x2fc/0x790
[  197.167196] Status: 1100fc02 KERNEL EXL 
[  197.171113] Cause : 40800014 (ExcCode 05)
[  197.175104] BadVA : 00000122
[  197.177967] PrId  : 0001992f (MIPS 1004Kc)
[  197.182041] Modules linked in: wireguard mt7915e mt76_connac_lib mt76 mac80211 libchacha20poly1305 cfg80211 poly1305_mips libcurve25519_generic libcrc32c hwmon compat cls_flower chacha_mips act_vlan cls_bpf act_bpf sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact ledtrig_usbport ip_tunnel vxlan udp_tunnel ip6_udp_tunnel sha512_generic seqiv jitterentropy_rng drbg hmac cmac leds_gpio xhci_plat_hcd xhci_pci xhci_mtk_hcd xhci_hcd gpio_button_hotplug usbcore nls_base usb_common crc32c_generic
[  197.232644] Process napi/mtk_eth-6 (pid: 271, threadinfo=769a9b6c, task=444f8b41, tls=00000000)
[  197.241319] Stack : 00000000 fffff7ff ff7f5530 81cea840 00000000 00000015 81416308 00000000
[  197.249681]         81416300 00000001 844e63c0 81d7dcf0 81d7dce8 81d7dce8 85bf8fd4 00000000
[  197.258039]         80150015 00000920 0000081e 00000920 00000a20 8141df00 00000a20 00000000
[  197.266393]         00000000 8089a3f0 808a0000 81d248b8 81d2c000 801b1aac 8155e210 00000a20
[  197.274754]         000005ee 00000013 00000a20 801b1f94 0000005f 81d248b8 81d2c000 800a4b74
[  197.283113]         ...
[  197.285556] Call Trace:
[  197.287986] [<801b1374>] ___slab_alloc.constprop.0+0x530/0x790
[  197.293821] [<801b1aac>] __slab_alloc.constprop.0.isra.0+0x3c/0x90
[  197.299990] [<801b1f94>] kmem_cache_alloc+0x494/0x578
[  197.305031] [<80519438>] build_skb+0x34/0x190
[  197.309380] [<804d9bf0>] mtk_napi_rx+0x8c8/0xeb0
[  197.313996] [<8053e6f4>] __napi_poll+0x70/0x1f8
[  197.318545] [<8053e924>] napi_threaded_poll+0xa8/0x108
[  197.323674] [<80053d64>] kthread+0x17c/0x1a0
[  197.327946] [<800034cc>] ret_from_kernel_thread+0x14/0x1c

[  197.334821] Code: 8e420008  8e430004  ac620004 <ac430000> 8fa30020  24020100  ae420004  24020122  ae420008 

[  197.346125] ---[ end trace bfba77fb6de4beaa ]---
[  197.350802] Kernel panic - not syncing: Fatal exception in interrupt

and

[   80.071102] CPU 2 Unable to handle kernel paging request at virtual address 744d2094, epc == 801b1be8, ra == 80519438                              
[   80.081760] Oops[#1]:                                                                                                                              
[   80.084050] CPU: 2 PID: 274 Comm: napi/mtk_eth-6 Not tainted 5.15.117 #0                   
[   80.090728] $ 0   : 00000000 00000001 744d2094 4d749420                                                                                            
[   80.095963] $ 4   : ac80cabf 00000a20 00000001 00000000                                                                                            
[   80.101184] $ 8   : 00091472 00000040 00000039 8155ffc0                                                                                            
[   80.106410] $12   : 00000b97 000034fc 00000000 00000000                                      
[   80.111632] $16   : 00000a20 8141da00 00000a20 00000000                                                      
[   80.116860] $20   : 744d203c 8089a3f0 808a0000 814418b8
[   80.122082] $24   : 00000000 8032e3b8                              
[   80.127307] $28   : 81dfe000 81dffd48 81d6e000 80519438                                                                                                                                                                       
[   80.132530] Hi    : 00000000                                                                                                                                                                  
[   80.135397] Lo    : 00000041                                            
[   80.138259] epc   : 801b1be8 kmem_cache_alloc+0xe8/0x578        
[   80.143587] ra    : 80519438 build_skb+0x34/0x190                                            
[   80.148294] Status: 1100fc03 KERNEL EXL IE                                                   
[   80.152477] Cause : 40800008 (ExcCode 02)                                                                                                                                                                                     
[   80.156470] BadVA : 744d2094                                                                                                                                                                  
[   80.159332] PrId  : 0001992f (MIPS 1004Kc)                                                                                                                                                    
[   80.163408] Modules linked in: wireguard mt7915e mt76_connac_lib mt76 mac80211 libchacha20poly1305 cfg80211 poly1305_mips libcurve25519_generic libcrc32c hwmon compat cls_flower chacha_mips act_vlan cls_bpf act_bpf sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 
cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact ledtrig_usbport ip_tunnel vxlan udp_tunnel ip6_udp_tunnel sha512_generic seqiv jitterentropy_rng drbg hmac cmac leds_gpio xhci_plat_hcd xhci_pci xhci_mtk_hcd xhci_hcd gpio_button_hotplug us
bcore nls_base usb_common crc32c_generic                                                                                                                                                                                         
[   80.214013] Process napi/mtk_eth-6 (pid: 274, threadinfo=c621d061, task=ce97ab3a, tls=00000000)                                                                                                                                                                            
[   80.222697] Stack : 000001b2 814418b8 81d6e000 800a4b74 814416a0 00000960 826d98e0 0000000f                                                                                                                                                                                
[   80.231062]         8270e420 826d98e0 000001b2 80519438 ff7ea510 8053f6b4 ffffaa16 00000031                                                                                                                                                                                
[   80.239420]         814416a0 a2f0bb20 000005ee 0000000f 8270e420 804d9bf0 808a0000 80591c00                                                                                                                                                                                
[   80.247776]         a23764ee 00000012 00000000 00000000 00000040 ff7ea510 000058f2 c5ee0000                                                                                                                                   
[   80.256137]         000058f2 8270e3e0 000006c8 814416a0 00000040 026d9920 0270e420 80a90000                                                                                                                                                                                
[   80.264500]         ...                                                                                                                                                                                                                                                    
[   80.266950] Call Trace:                                                                                                                                                                                                                                                    
[   80.269380] [<801b1be8>] kmem_cache_alloc+0xe8/0x578                                                                                
[   80.274348] [<80519438>] build_skb+0x34/0x190                                                                                       
[   80.278695] [<804d9bf0>] mtk_napi_rx+0x8c8/0xeb0                                                                                                                                                                                                                           
[   80.283307] [<8053e6f4>] __napi_poll+0x70/0x1f8                                                                                                                                                                                                                            
[   80.287860] [<8053e924>] napi_threaded_poll+0xa8/0x108                                                                              
[   80.292986] [<80053d64>] kthread+0x17c/0x1a0                                                                                        
[   80.297264] [<800034cc>] ret_from_kernel_thread+0x14/0x1c                                                                                                                                                                                                                  

[   80.304137] Code: 8e240078  02821021  7c0218a0 <8c420000> 00231c02  00821026  00433826  41656000  30a50001                                                                                                                                                                 

[   80.315460] ---[ end trace 4067ba48270f4645 ]---
rany2 commented 1 year ago

If it's useful:

(gdb) l *mtk_napi_rx+0x8c8
0x9298 is in mtk_napi_rx (drivers/net/ethernet/mediatek/mtk_eth_soc.c:2336).
2331    
2332                dma_unmap_single(eth->dma_dev, trxd.rxd1,
2333                         ring->buf_size, DMA_FROM_DEVICE);
2334    
2335                skb = build_skb(data, ring->frag_size);
2336                if (unlikely(!skb)) {
2337                    netdev->stats.rx_dropped++;
2338                    skb_free_frag(data);
2339                    goto skip_rx;
2340                }
rany2 commented 1 year ago

Nevermind it's a known issue: https://github.com/openwrt/openwrt/issues/9561

rany2 commented 1 year ago

I can't trigger those two kernel panics anymore, not sure what was up with those :/

brada4 commented 1 year ago

Unaligned access is most likely miscompilation ie to do with compiler parameters. Second is failing kmem allocation typically due to oom. In ideal world driver could gracefully continue on such failure like drop packet, but doe not seem the case.

Openwrt does not have slab stats by default, with them you could be able to identify if net skb pool uses too much or some other memory pool grow to all memory. Things like slabtop, /proc/slabinfo etc.

rany2 commented 1 year ago

@brada4 I doubt it has to do with oom, the device has 256MB of RAM and these are the only services running:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.1  0.5   1972  1328 ?        Ss   19:34   0:05 /sbin/procd
ubus       544  0.0  0.4   1536  1072 ?        S    19:35   0:02 /sbin/ubusd
root       545  0.0  0.2   1064   588 ttyS0    Ss+  19:35   0:00 /sbin/askfirst /usr/libexec/login.sh
root       580  0.0  0.3   1208   792 ?        S    19:35   0:00 /sbin/urngd
root       626  0.0  0.6   2076  1572 ?        S    19:35   0:01 /usr/sbin/haveged -F -w 1024 -d 32 -i 32 -v 1
logd       931  0.0  0.3   1488   924 ?        S    19:35   0:01 /sbin/logd -S 64
root       990  0.0  0.8   3460  2156 ?        S    19:35   0:00 /sbin/rpcd -s /var/run/ubus/ubus.sock -t 30
root      1135  0.0  0.4   1280  1028 ?        S    19:35   0:00 /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -s -g -p 22 -K 300 -T 3
root      1186  0.0  0.3   2916   988 ?        S    19:35   0:00 /sbin/ujail -t 5 -n hostapd -U network -G network -C /etc/capabilities/wpad.json -c -- /usr/sbin/hostapd -s -g /var/run/hostapd/global
root      1187  0.0  0.3   2916   988 ?        S    19:35   0:00 /sbin/ujail -t 5 -n wpa_supplicant -U network -G network -C /etc/capabilities/wpad.json -c -- /usr/sbin/wpa_supplicant -n -s -g /var/run/wpa_supplicant/global
network   1211  1.3  2.1   7036  5340 ?        S    19:35   1:01 /usr/sbin/hostapd -s -g /var/run/hostapd/global
network   1212  0.0  0.8   6588  2156 ?        S    19:35   0:00 /usr/sbin/wpa_supplicant -n -s -g /var/run/wpa_supplicant/global
root      1251  0.0  0.5   2176  1448 ?        S    19:35   0:02 /sbin/netifd
root      1520  0.0  0.3   1648   904 ?        S    19:35   0:00 /usr/sbin/uhttpd -f -h /www -r asus -x /cgi-bin -t 60 -T 30 -k 20 -A 1 -n 3 -N 100 -R -p 0.0.0.0:80 -p [::]:80
root      1703  0.0  0.4   2916  1204 ?        S    19:35   0:00 /sbin/ujail -t 5 -n umdns -S /etc/seccomp/umdns.json -u -l -- /usr/sbin/umdns
root      1712  0.0  0.4   1856  1216 ?        S    19:35   0:00 /usr/sbin/umdns
root      1810  2.6  0.8   3328  2108 ?        SL   19:35   1:57 /usr/sbin/dawn
root      1873  0.0  0.3   1356   864 ?        Sl   19:35   0:01 /usr/sbin/irqbalance -f -c 2 -t 10
root      2338  0.0  0.5   2916  1256 ?        S    19:35   0:00 /sbin/ujail -t 5 -n ntpd -U ntp -G ntp -C /etc/capabilities/ntpd.json -c -u -r /bin/ubus -r /usr/bin/env -r /usr/bin/jshn -r /usr/sbin/ntpd-hotplug -r /usr/share/libubox/jshn.sh -- /usr/sbin/ntpd -n -N -S /usr/sbin/ntpd-hotplug -p 10.146.109.1
ntp       2369  0.0  0.4   1380  1016 ?        S    19:35   0:00 /usr/sbin/ntpd -n -N -S /usr/sbin/ntpd-hotplug -p 10.146.109.1
root      3237  0.7  0.4   1300  1008 ?        S    20:49   0:00 /usr/sbin/dropbear -F -P /var/run/dropbear.1.pid -s -g -p 22 -K 300 -T 3 -2 9
root      3238  0.0  0.4   1384  1076 pts/0    Ss   20:49   0:00 -ash
root      3276  0.0  0.3   1864   952 pts/0    R+   20:50   0:00 ps aux
root      3277  0.0  0.3   1380   976 pts/0    S+   20:50   0:00 grep -Ev \[\S+\]$

and this is free -h:


 free -h
              total        used        free      shared  buff/cache   available
Mem:          243Mi        55Mi       165Mi       0.0Ki        22Mi       152Mi
Swap:            0B          0B          0B
``
brada4 commented 1 year ago

It is in different place, catch kmem pool growing wild, like a runaway process could be detected with your used tools. I don't have clear recipe to add support for zoneinfo in a single rebuild, but it still will be chasing ghosts

rany2 commented 1 year ago

UPDATE: OpenWRT 21.02 does not have this issue, all subsequent versions do.

rany2 commented 1 year ago

Another observation, on 21.02 the ERR counter in /proc/interrupts is under 50 whereas on SNAPSHOT/23.05/22.03 it almost exceeds a million.

SNAPSHOT:

           CPU0       CPU1       CPU2       CPU3       
  8:   11560538   11548657   11624111   11519204  MIPS GIC Local   1  timer
  9:    6188256          0          0          0  MIPS GIC  63  IPI call
 10:          0   13486634          0          0  MIPS GIC  64  IPI call
 11:          0          0   11192002          0  MIPS GIC  65  IPI call
 12:          0          0          0    6396570  MIPS GIC  66  IPI call
 13:       8029          0          0          0  MIPS GIC  67  IPI resched
 14:          0      40426          0          0  MIPS GIC  68  IPI resched
 15:          0          0      54612          0  MIPS GIC  69  IPI resched
 16:          0          0          0       8949  MIPS GIC  70  IPI resched
 17:          0          0          0          0  MIPS GIC  19  1e000600.gpio-bank0, 1e000600.gpio-bank1, 1e000600.gpio-bank2
 19:         14          0          0          0  MIPS GIC  33  ttyS0
 20:          0          0          0          0  MIPS GIC  29  xhci-hcd:usb1
 21:    7802624          0          0          0  MIPS GIC  10  1e100000.ethernet
 22:    1809817          0          0          0  MIPS GIC  11  PCIe PME, aerdrv, mt7915e-hif
 23:   15054549          0          0          0  MIPS GIC  31  PCIe PME, aerdrv, mt7915e
 24:          3          0          0          0  MIPS GIC  30  mt7530
 25:          0          0          0          0    mt7530   1  mt7530-0:01
 26:          0          0          0          0    mt7530   2  mt7530-0:02
 27:          3          0          0          0    mt7530   3  mt7530-0:03
ERR:     998120

22.02:

           CPU0       CPU1       CPU2       CPU3       
  8:      44547      44511      44450      44505  MIPS GIC Local   1  timer
  9:      24594          0          0          0  MIPS GIC  63  IPI call
 10:          0      18399          0          0  MIPS GIC  64  IPI call
 11:          0          0      91160          0  MIPS GIC  65  IPI call
 12:          0          0          0      29323  MIPS GIC  66  IPI call
 13:       9580          0          0          0  MIPS GIC  67  IPI resched
 14:          0      26259          0          0  MIPS GIC  68  IPI resched
 15:          0          0      26170          0  MIPS GIC  69  IPI resched
 16:          0          0          0      39845  MIPS GIC  70  IPI resched
 17:          0          0          0          0  MIPS GIC  19  1e000600.gpio-bank0, 1e000600.gpio-bank1, 1e000600.gpio-bank2
 19:         14          0          0          0  MIPS GIC  33  ttyS0
 20:          0          0          0          0  MIPS GIC  29  xhci-hcd:usb1
 21:     220579          0          0          0  MIPS GIC  10  1e100000.ethernet
 23:      11272          0          0          0  MIPS GIC  11  mt7915e-hif
 24:      32708          0          0          0  MIPS GIC  31  mt7915e
 26:          0          0          0          0  1e000600.gpio  16  keys
 27:          0          0          0          0  1e000600.gpio  15  keys
ERR:         27
ThiloteE commented 1 year ago

Looks like a job for https://git-scm.com/docs/git-bisect

rany2 commented 1 year ago

@ThiloteE I am aware but there is no way I'm doing that here. Just too many changes....

rany2 commented 1 year ago

What I could say is that this started from the very first rc for 22.03, so 22.03 was always broken basically.

rany2 commented 1 year ago

Also I should mention that a bisect will be extremely time consuming because as stated in the original post, this doesn't occur all the time. Sometimes it starts off poorly at bootup other times the boot is solid and it only degrades later. So the testing will take AGES!

DragonBluep commented 1 year ago

@ptpt52 directly via router, and there is no hardware offloading enabled.

Edit: performance between switch ports is OK, getting gigabit speeds there

Running iperf3 on router side is not the right way. There has been a lot of discussion here: https://forum.openwrt.org/t/mt7621-ethernet-performance-below-expectations/119901

By the way, the main change from 21.x to 22.x was to replace iptables with nftables, which seems to have led to significant performance degradation.

rany2 commented 1 year ago

@DragonBluep I do not have any firewall installed:

cfg80211              288282  4 mt7915e,mt76_connac_lib,mt76,mac80211
chacha_mips             5531  1 libchacha20poly1305
compat                  1602  2 mac80211,cfg80211
crc_ccitt               1806  0
gpio_button_hotplug     6834  0
hwmon                   7810  1 mt7915e
ip_tunnel              14022  0
leds_gpio               3090  0
ledtrig_usbport         2850  0
libchacha20poly1305     4678  1 wireguard
libcurve25519_generic   10454  1 wireguard
mac80211              547992  3 mt7915e,mt76_connac_lib,mt76
mt76                   48038  2 mt7915e,mt76_connac_lib
mt76_connac_lib        26464  1 mt7915e
mt7915e               101296  0
nls_base                5626  1 usbcore
poly1305_mips           2415  1 libchacha20poly1305
udp_tunnel              2937  2 wireguard,vxlan
usb_common              3166  3 xhci_plat_hcd,xhci_hcd,usbcore
usbcore               150574  5 ledtrig_usbport,xhci_plat_hcd,xhci_pci,xhci_mtk,xhci_hcd
vxlan                  37170  0
wireguard              53095  0
xhci_hcd              120200  3 xhci_plat_hcd,xhci_pci,xhci_mtk
xhci_mtk                5010  0
xhci_pci                3890  0
xhci_plat_hcd           5842  0

Regarding iperf, I do understand that however the network is in fact slower both via router and another PC on the same network. Also it doesn't mean that the perf would just drop out of nowhere.

As previously stated, this is a dumb AP with the bare minimum.

Spudz76 commented 1 year ago

Also of note 22.02 is fine and 22.03 is not whilst both are nftables, if that was even involved.

And to clarify it never drops packets it's just slower? Is the RTT on ICMP ping noticeably higher during this also? Do small packets such as ICMP ECHO or DNS seem unaffected while streaming/max-MTU packets have trouble? I wonder if there is a magic/poison packet condition that triggers the problem, leading to apparent randomness but if such packet-of-death were figured out it could be reproduced to assist in bisect.

Probably the biggest clue is the interrupt ERR count, something basic system level driver must have changed. I note the three mt7530-0: devices are only in the snapshot output. What are the diffs between tag 22.02 and 22.03 for the relevant target/platform files, and could those be a smaller set to bisect? Doesn't seem like it would be in common parts or there would be problems with many other targets.

Spudz76 commented 1 year ago

(seems weird to have a MTU:2026 btw)

Spudz76 commented 1 year ago

Here are some musings from when the mt7530 DSA driver became a thing, noting that there are quirks where packets of max-size just get silently lost... hmmmmmm (also later on down the thread, there is debate about how interrupts might/should/could work... hmmmmmmmmmmmmm)

I wonder if issue disappears by setting MTU:1500 or 1492 or less, or gets worse due to fragging. Still, 2026 seems wrong, unless things know how to do jumbos, and then why not 9000.

Spudz76 commented 1 year ago

The router I'm currently using, I don't note any problems but also have millions of interrupts ERR. Have not tried to replicate with iperf. I do sometimes/rarely get a slowdown with respect to Internet speeds but I have NAT and nftables stuff going and a bunch of other junk using CPU.

Also of note my MTU is 1504, but that other thread seemed to say it will tell you >1500 is illegal if you try to set it. But it's definitely not 2026.

root@wonky:~# uptime
 05:33:36 up 40 days,  4:56,  load average: 2.03, 1.98, 1.99
root@wonky:~# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
  7:          0          0          0          0      MIPS   7  timer
  8:  347379294  347379276  347379267  347379261  MIPS GIC Local   1  timer
  9: 1013590807          0          0          0  MIPS GIC  63  IPI call
 10:          0 1315474950          0          0  MIPS GIC  64  IPI call
 11:          0          0 3254889250          0  MIPS GIC  65  IPI call
 12:          0          0          0 2764087623  MIPS GIC  66  IPI call
 13:   24159280          0          0          0  MIPS GIC  67  IPI resched
 14:          0 3038645992          0          0  MIPS GIC  68  IPI resched
 15:          0          0  122056870          0  MIPS GIC  69  IPI resched
 16:          0          0          0  498250549  MIPS GIC  70  IPI resched
 17:          0          0          0          0  MIPS GIC  19  1e000600.gpio-bank0, 1e000600.gpio-bank1, 1e000600.gpio-bank2
 19:         12          0          0          0  MIPS GIC  33  ttyS0
 20: 3797738205          0          0          0  MIPS GIC  29  xhci-hcd:usb1
 21:  592178944          0          0          0  MIPS GIC  10  1e100000.ethernet
 22:        563          0  771886119          0  MIPS GIC  11  mt7615e
 23:       2015          0          0  118339328  MIPS GIC  31  mt7615e
 25:          3          0          0          0  MIPS GIC  30  mt7530
 26:          3          0          0          0    mt7530   1  mt7530-0:01
 27:          0          0          0          0    mt7530   2  mt7530-0:02
 28:          0          0          0          0    mt7530   3  mt7530-0:03
 29:          0          0          0          0    mt7530   4  mt7530-0:04
 30:          0          0          0          0  1e000600.gpio  17  keys
 31:          0          0          0          0  1e000600.gpio  12  keys
ERR:   31408381
root@wonky:~# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 4E:54:55:25:57:68
          inet6 addr: fe80::4c54:55ff:fe25:5768/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1504  Metric:1
          RX packets:2675228781 errors:0 dropped:166 overruns:0 frame:0
          TX packets:5202119236 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:334482353348 (311.5 GiB)  TX bytes:7008190302937 (6.3 TiB)
          Interrupt:21
rany2 commented 1 year ago

@Spudz76 I had this issue with jumbo frames enabled and disabled, that mt7530 router in particular needs jumbo frames for WireGuard/VXLAN interfaces (so it ends up with the standard MTU of 1500 not 1450). I think this misconception that mt7530 has issues with jumbo frames occurred because the datasheet provided invalid information about jumbo frames and mt7530, they claimed it supported 9k when it actually supported 2048 MAX.

P.S. the reason your eth0 has a weird MTU of 1504 is because it needs an extra 4 bytes for DSA tag but the ports themselves would have the "actual" MTU.

rany2 commented 1 year ago

I wonder if issue disappears by setting MTU:1500 or 1492 or less, or gets worse due to fragging. Still, 2026 seems wrong, unless things know how to do jumbos, and then why not 9000.

It does work fine with 2030 as MTU on OpenWRT 21.02, that's not the issue.

Probably the biggest clue is the interrupt ERR count, something basic system level driver must have changed. I note the three mt7530-0: devices are only in the snapshot output. What are the diffs between tag 22.02 and 22.03 for the relevant target/platform files, and could those be a smaller set to bisect? Doesn't seem like it would be in common parts or there would be problems with many other targets.

OpenWRT 22.03 had a massive refactor for mt7621's drivers and used a lot of the now upstreamed drivers. So this is most likely the case, however I have no idea which component might cause this issue. Perhaps @paraka could help and provide suggestions? He was responsible for most of that effort on that front

paraka commented 1 year ago

I don't really know but @arinc9 is the expert in mt7530 so maybe he could help.

rany2 commented 1 year ago

So I guess you think increase in errors in /proc/interrupts is related to mt7530 and not the platform drivers?

paraka commented 1 year ago

So I guess you think increase in errors in /proc/interrupts is related to mt7530 and not the platform drivers?

I only meant that I have never touched by myself nothing related with ethernet or mt7530 for these SoCs and @arinc9 is normally doing that. So I was only pinging him for help :) Interrupts for mt7530 were enabled recently also and there was not support before.

brada4 commented 1 year ago

ERR seems proportionate to uptime (i.e. timer) , probably some unhandled semi-hidden device on SoC. comparable to x86 SMI or so. Can you check if it grows on idle device (ssh connection allowed) same as on busy one, like for 1h

Spudz76 commented 1 year ago

On a very very idle copy of my main router. This unit just sits around in between test flashes. It is running a different build, 5.15 kernel vs 5.10 and some other stacked previews of patches and other hacks (hwcrypto device for example shows up in this dump, with some interrupts from having done the bootup tests)

But ERR is zero on this one. Either 5.15 fixes something (snapshot should be 5.15 anyway by now?) or it needs traffic.

Also of note there is one more mt7530 interrupt compared to the OP device. Guess the Linksys EA7300v1 has one more (physical) port vs the ASUS RT-AX53U?

MTU stuff makes sense I guess, I also have no knowledge of wireguard. And am still of the old best practice of avoiding jumbos at all costs because everything never supports them right. :)

root@wonky2:~# uptime
 11:07:29 up 113 days, 18:00,  load average: 0.00, 0.00, 0.00
root@wonky2:~# cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3
  8:  982805130  982805139  982805130  982805123  MIPS GIC Local   1  timer
  9:  464644681          0          0          0  MIPS GIC  63  IPI call
 10:          0 1541131021          0          0  MIPS GIC  64  IPI call
 11:          0          0 1041173388          0  MIPS GIC  65  IPI call
 12:          0          0          0  961117599  MIPS GIC  66  IPI call
 13:      41601          0          0          0  MIPS GIC  67  IPI resched
 14:          0      42745          0          0  MIPS GIC  68  IPI resched
 15:          0          0      42075          0  MIPS GIC  69  IPI resched
 16:          0          0          0      51776  MIPS GIC  70  IPI resched
 17:          0          0          0          0  MIPS GIC  19  1e000600.gpio-bank0, 1e000600.gpio-bank1, 1e000600.gpio-bank2
 19:        114          0          0          0  MIPS GIC  33  ttyS0
 20:          0          0          0          0  MIPS GIC  29  xhci-hcd:usb1
 21:      11866          0          0          0  MIPS GIC  26  1e004000.crypto
 22:   34449992          0          0          0  MIPS GIC  10  1e100000.ethernet
 23:          5          0          0          0  MIPS GIC  30  mt7530
 24:          5          0          0          0    mt7530   1  mt7530-0:01
 25:          0          0          0          0    mt7530   2  mt7530-0:02
 26:          0          0          0          0    mt7530   3  mt7530-0:03
 27:          0          0          0          0    mt7530   4  mt7530-0:04
 28:          0          0          0          0  1e000600.gpio  17  keys
 29:          0          0          0          0  1e000600.gpio  12  keys
 30:         18          0          0          0  MIPS GIC  11  mt7615e
 31:         18          0          0          0  MIPS GIC  31  mt7615e
ERR:          0
brada4 commented 1 year ago

That is within variety of SoC configuration at production line.

Do you get err with wifi down? seems case with no-err router?

arinc9 commented 1 year ago

I just noticed that https://github.com/openwrt/openwrt/blob/main/target/linux/generic/pending-6.1/723-net-mt7531-ensure-all-MACs-are-powered-down-before-r.patch is wrong. It should be PMCR_FORCE_LNK and that change impacts mt7530_setup not mt7531_setup so patch name is wrong as well.

I'm a maintainer of the MT7530 DSA subdriver. Everything said above is utterly false. The PMCR_FORCE_LNK bit only exists on the MT7530 switch. The bit that has got the same function on the MT7531 switch is 31 and is called MT7531_FORCE_LNK. The SYS_CTRL_PHY_RST bit on the 0x7000 register doesn't exist on the MT7531 switch. Because of this, it's pointed out on the MT7531 Reference Manual for Development Board rev 1.0 to set all the MACs down using the MT7531_FORCE_LNK bit before setting SYS_CTRL_SW_RST and SYS_CTRL_REG_RST, hence the reason for the patch linked above.

If there's anything remotely wrong here, it's the unnecessary setting of the SYS_CTRL_PHY_RST bit for the MT7531 switch. I already have a patch series in the works that addresses this. https://github.com/arinc9/linux/commit/4d9a2781ce232f1fa53afc975e22345e768ee1d4

I suggest you read a bit more before jumping to conclusions.

I suspect the regression reported here is caused by the changes on the MediaTek ethernet driver. I don't see a reason to investigate it as these SoCs were not intended to do IP packet forwarding at CPU let alone dealing with the iperf3 load, be it on the generating or the receiving side. The packet processing engine (PPE, marketed as Network Accelerator on the MediaTek website) is supposed to deal with forwarding packets.

Enable hardware offloading, which lets the PPE deal with forwarding packets, then try a download or upload test between two switch ports, each on separate networks. You should consistently get 933 Mbps.

If you really want to find the commit that causes the regression, the bisection should be done on the Linux repository, not OpenWrt. It should take about 10 tests to pinpoint the kernel minor version that introduces the regression.

You can either somehow make OpenWrt compile the Linux version of your choice, or just boot vanilla Linux with Buildroot as the filesystem using mainline U-Boot which can boot this without any issues.

I will try the latter on my MT7621 board, see if I also come across this regression.

rany2 commented 1 year ago

Enable hardware offloading, which lets the PPE deal with forwarding packets, then try a download or upload test between two switch ports, each on separate networks. You should consistently get 933 Mbps.

Does this mean that I need to have nft-offload installed? I do not have any firewall installed. See https://github.com/openwrt/openwrt/issues/12914#issuecomment-1597908625