mediatek mt7915e firmware instability

szyper-sakrona commented 7 months ago

Hello, I've tested nearly all your releases for my WSM20 machines - but current releases has latest mediatek firmware (mt7915e) from 202310xxx. I've noticed that it generates problems with wifi - timeouts and wifi is not working normally. I've uploaded older firmware from 20230418xxx - and after 1 day it is working good. I'll be informing is that better for longer periods. Maybe it is more safe to get back to this firmware on that routers. Regards

ptpt52 commented 7 months ago

I would revert to older firmware for mt7915e in next release.

szyper-sakrona commented 6 months ago

After testing for a few days - something wrong is with usteer and these firmwares...I do not know, but after some hours there is these symptoms (dmesg dump):

[ 4027.942671] mt7915e 0000:02:00.0: Message 000025ed (seq 11) timeout [ 4027.948975] phy1-ap0: failed to set key (0, cc:42:10:45:bd:dc) to hardware (- 145) [ 4048.422139] mt7915e 0000:02:00.0: Message 00005aed (seq 12) timeout [ 4068.901634] mt7915e 0000:02:00.0: Message 00005aed (seq 13) timeout [ 4089.381003] mt7915e 0000:02:00.0: Message 000025ed (seq 14) timeout [ 4109.860433] mt7915e 0000:02:00.0: Message 00005aed (seq 15) timeout [ 4130.339856] mt7915e 0000:02:00.0: Message 00005aed (seq 1) timeout .......................................................................................................................................................... [ 9475.495869] phy0-ap0: HW problem - can not stop rx aggregation for 6a:98:b9:6 c:72:e6 tid 0 [ 9475.504241] phy0-ap0: HW problem - can not stop rx aggregation for 6a:98:b9:6 c:72:e6 tid 6 [ 9475.562797] ------------[ cut here ]------------ [ 9475.567482] WARNING: CPU: 2 PID: 1856 at /mnt/Data/Sources/openwrt/x-wrt/tmpf s/target-mipsel_24kc_musl/linux-ramips_mt7621/mac80211-regular/backports-6.6.15/ net/mac80211/agg-tx.c:407 ___ieee80211_stop_tx_ba_session+0x2ac/0x2cc [mac80211] [ 9475.588675] Modules linked in: pppoe ppp_async wireguard pptp pppox ppp_mppe ppp_generic nft_fib_inet mt7915e mt76_connac_lib mt76 mac80211 libchacha20poly13 05 ipt_REJECT cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_state xt_rece nt xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_helper xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_TCPMSS xt_REDIRECT xt_MASQUERADE xt_LOG xt_HL xt_DSCP xt_DNETMAP xt_DHCPMAC xt_CLASSIFY ts_fsm ts_bm tcp_scalable tcp_bbr slhc sch_cake poly1305_mips nft_tproxy nft_so cket nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject_bridge nft_rejec t nft_redir nft_quota nft_queue nft_objref nft_numgen nft_nat nft_meta_bridge nf t_masq nft_log nft_limit nft_hash nft_fwd_netdev nft_fib_ipv6 nft_fib_ipv4 nft_f ib nft_dup_netdev nft_ct nft_counter nft_compat nft_chain_nat nfnetlink_queue nf _tproxy_ipv6 nf_tproxy_ipv4 nf_tables nf_socket_ipv6 nf_socket_ipv4 nf_reject_ip v4 nf_nat_tftp nf_nat_snmp_basic nf_nat_sip [ 9475.589192] nf_nat_pptp nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda nf_l og_syslog nf_dup_netdev nf_conntrack_tftp nf_conntrack_snmp nf_conntracksip nf conntrack_pptp nf_conntrack_netlink nf_conntrack_irc nf_conntrack_h323 nf_conntr ack_ftp nf_conntrack_broadcast nf_conntrack_bridge ts_kmp nf_conntrack_amanda nf _conncount macvlan libcurve25519_generic libcrc32c ipvlan iptable_nat iptable_ma ngle iptable_filter ipt_ECN ip_tables hwmon crc_ccitt compat_xtables compat chac ha_mips br_netfilter asn1_decoder natflow natcap nf_nat nf_conntrack sch_tbf sch _ingress sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact cryptodev xt_set ip_set_list_set ip_se t_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip _set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ipset hash_ipport ip_set_hash_ipmark ip_set_hash_ipmac ip_set_hash_ip ip_set_bitmap_po rt ip_set_bitmap_ipmac ip_set_bitmap_ip [ 9475.676690] ip_set nfnetlink jool_siit jool nf_defrag_ipv4 jool_common ip6ta ble_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 bondin g tls ip6_gre ip_gre gre nat46 nf_defrag_ipv6 sit ip6tunnel tunnel6 tunnel4 ip tunnel udp_diag tcp_diag raw_diag inet_diag tun ovpn_dco_v2 udp_tunnel ip6_udp_t unnel nls_utf8 nls_iso8859_1 nls_base crypto_user algif_skcipher algif_rng algif _hash algif_aead af_alg sha512_generic sha256_generic libsha256 sha1_generic seq iv jitterentropy_rng drbg md5 crypto_hw_eip93 hmac ecb des_generic libdes cmac a uthencesn authenc arc4 leds_gpio softdog gpio_button_hotplug crc32c_generic [ 9475.819065] CPU: 2 PID: 1856 Comm: hostapd Not tainted 5.15.148 #0

and wifi is not working normally... Any chances that maybe you can migrate this device to for ex. new 6.x kernels change "hostap" version?

ptpt52 commented 6 months ago

try latest version now.

szyper-sakrona commented 6 months ago

I temporary disable usteer on all my APs - maybe the firmware is "innocent" and there is hostapd+usteer conflict?

szyper-sakrona commented 6 months ago

Dear ptpt52, the problem is described here https://github.com/openwrt/openwrt/issues/11931 Is there any chance to introduce 6.1 kernel to ramips builds? maybe it could resolve errors - the latest "stable" builds was late 2023y. Thanks!

ptpt52 commented 6 months ago

the lastest version is now kernel 6.1

godshades commented 6 months ago

@ptpt52 wifi unstable, sometime wifi disconnect, sometime router highload cpu i'm using 2x cr6608 with mesh: router 1 have internet, router 2 not (can't access lan too), after reboot router 1, both have network, idk why

szyper-sakrona commented 6 months ago

latest build also have seq timeouts - wifi unstable - I think the hostapd build is broken from some months, maybe latest patches could fix this cohabitation.

ptpt52 commented 6 months ago

next release would update to latest mt76 driver, then try again when next release is ready

szyper-sakrona commented 6 months ago

now I'm got back to official 23.05.02 version with older software - testing different combinations. We will see!

szyper-sakrona commented 6 months ago

Dear Ptpt52 - could you make a test version of Xwrt with these hostapd patches from this repo?: https://github.com/cmonroe/feed-wifi-master/tree/smartrg-master/hostapd Anyway I try latest version - I notice that it is connected with AX clients, usteer, hostapd - the firmware is innocent. Thanks

ptpt52 commented 6 months ago

x-wrt new release is now ready.

Dear Ptpt52 - could you make a test version of Xwrt with these hostapd patches from this repo?:

It looks like not that easy to build agaist with that hostapd

szyper-sakrona commented 6 months ago

Hello, latest update seems to works good, also with latest mediatek firmware 7915e (date 20231215). Thanks! Regards!

szyper-sakrona commented 4 months ago

Hello Ptpt52, after testing various configurations - your kernel's version builds and mediatek firmwares I found only one combination which is surprising stable. I used it with DAWN (have several AP with roaming). This is the best stable (30+ days) one: X-WRT 24.01_b202403222222, Mantic with latest CMonroe version mt76 - mt7915e from 2024040X (https://github.com/cmonroe/feed-wifi-master/tree/smartrg-master/mt76/files/firmware) Anything newer gives also kernel "...Message 00005aed (seq .....) timeout" errors after a half day or less/more. Interesting discussion is here https://github.com/openwrt/mt76/issues/690 What do You thing about this specific combination? Regards I didn't test latest builds - is there any chance that latest version could fix mt76/hostapd problems? Thx

x-wrt / build-release

mediatek mt7915e firmware instability #43

After testing for a few days - something wrong is with usteer and these firmwares...I do not know, but after some hours there is these symptoms (dmesg dump):