greearb / ath10k-ct

Stand-alone ath10k driver based on Candela Technologies Linux kernel.
111 stars 40 forks source link

Crash inside ath10k from what I can tell - using latest beta CT firmware binaries #143

Open atiensivu opened 4 years ago

atiensivu commented 4 years ago

Netgear 7800 - OpenWRT - Build info below crash log:

[354147.746549] ------------[ cut here ]------------ [354147.746661] WARNING: CPU: 0 PID: 9 at backports-5.7-rc3-1/net/mac80211/sta_info.c:1929 ieee80211_sta_update_pending_airtime+0x1f8/0x1fc [mac80211] [354147.750291] STA a4:d9:31:00:56:37 AC 2 txq pending airtime underflow: 4294966496, 800 [354147.750293] Modules linked in: xt_connlimit pppoe ppp_async nf_conncount iptable_nat ath10k_pci ath10k_core ath xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT pppox ppp_generic nf_nat nf_flow_table_hw nf_flow_table nf_conntrack_rtcache nf_conntrack_netlink nf_conntrack mac80211 ipt_REJECT ebtable_nat ebtable_filter ebtable_broute cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY wireguard slhc sch_cake nlmon nfnetlink_queue nfnetlink_log nf_reject_ipv4 nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among ebt_802_3 crc_ccitt compat sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit [354147.763399] act_mirred ledtrig_usbport ledtrig_heartbeat xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ifb ip6_udp_tunnel udp_tunnel netlink_diag leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_dwc3 ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod ehci_hcd gpio_button_hotplug [354147.900989] CPU: 0 PID: 9 Comm: ksoftirqd/0 Not tainted 5.4.46 #0 [354147.923141] Hardware name: Generic DT based system [354147.929141] [] (unwind_backtrace) from [] (show_stack+0x14/0x20) [354147.933920] [] (show_stack) from [] (dump_stack+0x94/0xa8) [354147.941907] [] (dump_stack) from [] (warn+0xb4/0xd0) [354147.949017] [] (warn) from [] (warn_slowpath_fmt+0x80/0x90) [354147.956008] [] (warn_slowpath_fmt) from [] (ieee80211_sta_update_pending_airtime+0x1f8/0x1fc [mac80211]) [354147.963718] [] (ieee80211_sta_update_pending_airtime [mac80211]) from [] (ieee80211_report_low_ack+0x254/0x50c [mac80211]) [354147.975047] [] (ieee80211_report_low_ack [mac80211]) from [] (ieee80211_free_txskb+0x14/0x2c [mac80211]) [354147.987733] [] (ieee80211_free_txskb [mac80211]) from [] (ath10k_txrx_tx_unref+0x608/0x738 [ath10k_core]) [354147.999168] [] (ath10k_txrx_tx_unref [ath10k_core]) from [] (ath10k_htt_t2h_msg_handler+0xe6c/0x1288 [ath10k_core]) [354148.010421] [] (ath10k_htt_t2h_msg_handler [ath10k_core]) from [] (ath10k_pci_htt_rx_cb+0x178/0x230 [ath10k_pci]) [354148.022845] [] (ath10k_pci_htt_rx_cb [ath10k_pci]) from [] (ath10k_ce_per_engine_service+0x9c/0x10c [ath10k_core]) [354148.034741] [] (ath10k_ce_per_engine_service [ath10k_core]) from [] (ath10k_ce_per_engine_service_any+0x80/0xd8 [ath10k_core]) [354148.046791] [] (ath10k_ce_per_engine_service_any [ath10k_core]) from [] (ath10k_pci_napi_poll+0x54/0x15c [ath10k_pci]) [354148.059972] [] (ath10k_pci_napi_poll [ath10k_pci]) from [] (net_rx_action+0x118/0x374) [354148.072461] [] (net_rx_action) from [] (__do_softirq+0x130/0x2d4) [354148.082179] [] (__do_softirq) from [] (run_ksoftirqd+0x38/0x4c) [354148.090164] [] (run_ksoftirqd) from [] (smpboot_thread_fn+0xfc/0x1c8) [354148.098067] [] (smpboot_thread_fn) from [] (kthread+0x160/0x164) [354148.106135] [] (kthread) from [] (ret_from_fork+0x14/0x2c) [354148.114030] Exception stack(0xdd465fb0 to 0xdd465ff8) [354148.121152] 5fa0: 00000000 00000000 00000000 00000000 [354148.126369] 5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [354148.134611] 5fe0: 00000000 00000000 00000000 00000000 00000013 00000000 [354148.142905] ---[ end trace 34d2bd0d6810903a ]---

[ 15.370966] ath10k_pci 0000:01:00.0: assign IRQ: got 35 [ 15.370992] ath10k 5.1 driver, optimized for CT firmware, probing pci device: 0x46. [ 15.371750] ath10k_pci 0000:01:00.0: enabling device (0140 -> 0142) [ 15.377527] ath10k_pci 0000:01:00.0: enabling bus mastering [ 15.378031] ath10k_pci 0000:01:00.0: pci irq msi oper_irq_mode 2 irq_mode 0 reset_mode 0 [ 15.966252] ath10k_pci 0000:01:00.0: qca9984/qca9994 hw1.0 target 0x01000000 chip_id 0x00000000 sub 168c:cafe [ 15.966285] ath10k_pci 0000:01:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0 [ 15.976767] ath10k_pci 0000:01:00.0: firmware ver 10.4b-ct-9984-fH-013-4ab470999 api 5 features mfp,peer-flow-ctrl,txstatus-noack,wmi-10.x-CT,ratemask-CT,regdump-CT,txrate-CT,flush-all-CT,pingpong-CT,ch-regs-CT,nop-CT,htt-mgt-CT,set-special-CT,tx-rc-CT,cust-stats-CT,txrate2-CT,beacon-cb-CT,wmi-block-ack-CT,wmi-bcn-rc-CT crc32 34b2045a [ 18.306064] ath10k_pci 0000:01:00.0: board_file api 2 bmi_id 0:1 crc32 85498734 [ 24.146989] ath10k_pci 0000:01:00.0: unsupported HTC service id: 1536 [ 24.148025] ath10k_pci 0000:01:00.0: 10.4 wmi init: vdevs: 16 peers: 48 tid: 96 [ 24.152495] ath10k_pci 0000:01:00.0: msdu-desc: 2500 skid: 32 [ 24.235422] ath10k_pci 0000:01:00.0: wmi print 'P 48/48 V 16 K 144 PH 176 T 186 msdu-desc: 2500 sw-crypt: 0 ct-sta: 0' [ 24.236265] ath10k_pci 0000:01:00.0: wmi print 'free: 84872 iram: 13412 sram: 11224' [ 24.524857] ath10k_pci 0000:01:00.0: htt-ver 2.2 wmi-op 6 htt-op 4 cal pre-cal-file max-sta 32 raw 0 hwcrypto 1

Build: OpenWrt SNAPSHOT r13600-9a477b833a / LuCI Master git-20.168.54087-84a0b68

I can try to actively debug/bisect if interested, as I sure am.

greearb commented 4 years ago

This may be a mac80211 air-time-fairness issue, not sure it is related to ath10k at all. Is system unstable in any way you can notice? Plz post this to openwrt mailing list, though we can keep the bug open here as well.

atiensivu commented 4 years ago

This may be a mac80211 air-time-fairness issue, not sure it is related to ath10k at all. Is system unstable in any way you can notice? Plz post this to openwrt mailing list, though we can keep the bug open here as well.

Good deal. Sent to the devel list.

Normal temperatures, known good hardware

Behavior: After the crash, wireless was spotty/dropping packets, from what I could tell - came back to normal after reboot.

Outside of that oops, it has been a great device - so thanks for keeping this fork updated. :)

atiensivu commented 4 years ago

I'll be sending this to the devel list - my original post is currently stuck in moderation due to the title? Weird.

It did crash again today though:

[ 64.751260] ath10k_pci 0000:01:00.0: Invalid peer id 0 or peer stats buffer, peer: 337a071d sta: 00000000 [ 118.146127] ath10k_pci 0001:01:00.0: Invalid VHT mcs 15 peer stats [ 121.499106] TCP: request_sock_TCP: Possible SYN flooding on port 5000. Sending cookies. Check SNMP counters. [37713.544457] ath10k_pci 0001:01:00.0: wmi: fixing invalid VHT TX rate code 0xff [48665.154047] ath10k_pci 0000:01:00.0: htt tx: fixing invalid VHT TX rate code 0xff [58071.693437] ------------[ cut here ]------------ [58071.693548] WARNING: CPU: 0 PID: 9 at backports-5.7-rc3-1/net/mac80211/sta_info.c:1929 ieee80211_sta_update_pending_airtime+0x1f8/0x1fc [mac80211] [58071.697136] STA 50:de:06:5f:c5:7c AC 2 txq pending airtime underflow: 4294967236, 60 [58071.697138] Modules linked in: xt_connlimit pppoe ppp_async nf_conncount iptable_nat ath10k_pci ath10k_core ath xt_state xt_nat xt_helper xt_conntrack xt_connmark xt_connbytes xt_REDIRECT xt_MASQUERADE xt_FLOWOFFLOAD xt_CT pppox ppp_generic nf_nat nf_flow_table_hw nf_flow_table nf_conntrack_rtcache nf_conntrack_netlink nf_conntrack mac80211 ipt_REJECT ebtable_nat ebtable_filter ebtable_broute cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY wireguard slhc sch_cake nlmon nfnetlink_queue nfnetlink_log nf_reject_ipv4 nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 iptable_raw iptable_mangle iptable_filter ipt_ECN ip_tables ebtables ebt_vlan ebt_stp ebt_redirect ebt_pkttype ebt_mark_m ebt_mark ebt_limit ebt_among ebt_802_3 crc_ccitt compat sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_tcindex cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit

[58071.710221] act_mirred ledtrig_usbport ledtrig_heartbeat xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink nf_log_ipv6 nf_log_common ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ifb ip6_udp_tunnel udp_tunnel netlink_diag leds_gpio xhci_plat_hcd xhci_pci xhci_hcd dwc3 dwc3_qcom ohci_platform ohci_hcd phy_qcom_dwc3 ahci fsl_mph_dr_of ehci_platform ehci_fsl sd_mod ahci_platform libahci_platform libahci libata scsi_mod ehci_hcd gpio_button_hotplug

[58071.847090] CPU: 0 PID: 9 Comm: ksoftirqd/0 Not tainted 5.4.46 #0 [58071.869248] Hardware name: Generic DT based system [58071.875510] [] (unwind_backtrace) from [] (show_stack+0x14/0x20) [58071.880200] [] (show_stack) from [] (dump_stack+0x94/0xa8) [58071.888100] [] (dump_stack) from [] (warn+0xb4/0xd0) [58071.895124] [] (warn) from [] (warn_slowpath_fmt+0x80/0x90) [58071.902018] [] (warn_slowpath_fmt) from [] (ieee80211_sta_update_pending_airtime+0x1f8/0x1fc [mac80211]) [58071.909650] [] (ieee80211_sta_update_pending_airtime [mac80211]) from [] (ieee80211_report_low_ack+0x254/0x50c [mac80211]) [58071.920894] [] (ieee80211_report_low_ack [mac80211]) from [] (ieee80211_free_txskb+0x14/0x2c [mac80211]) [58071.933492] [] (ieee80211_free_txskb [mac80211]) from [] (ath10k_txrx_tx_unref+0x608/0x738 [ath10k_core]) [58071.944840] [] (ath10k_txrx_tx_unref [ath10k_core]) from [] (ath10k_htt_t2h_msg_handler+0xe6c/0x1288 [ath10k_core]) [58071.956008] [] (ath10k_htt_t2h_msg_handler [ath10k_core]) from [] (ath10k_pci_htt_rx_cb+0x178/0x230 [ath10k_pci]) [58071.967996] [] (ath10k_pci_htt_rx_cb [ath10k_pci]) from [] (ath10k_ce_per_engine_service+0x9c/0x10c [ath10k_core]) [58071.980153] [] (ath10k_ce_per_engine_service [ath10k_core]) from [] (ath10k_ce_per_engine_service_any+0x80/0xd8 [ath10k_core]) [58071.992116] [] (ath10k_ce_per_engine_service_any [ath10k_core]) from [] (ath10k_pci_napi_poll+0x54/0x15c [ath10k_pci]) [58072.005211] [] (ath10k_pci_napi_poll [ath10k_pci]) from [] (net_rx_action+0x118/0x374) [58072.017612] [] (net_rx_action) from [] (__do_softirq+0x130/0x2d4) [58072.027244] [] (__do_softirq) from [] (run_ksoftirqd+0x38/0x4c) [58072.035142] [] (run_ksoftirqd) from [] (smpboot_thread_fn+0xfc/0x1c8) [58072.042612] [] (smpboot_thread_fn) from [] (kthread+0x160/0x164) [58072.050939] [] (kthread) from [] (ret_from_fork+0x14/0x2c) [58072.058748] Exception stack(0xdd465fb0 to 0xdd465ff8) [58072.065782] 5fa0: 00000000 00000000 00000000 00000000 [58072.070912] 5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [58072.079068] 5fe0: 00000000 00000000 00000000 00000000 00000013 00000000 [58072.087300] ---[ end trace 1945af53b469d02c ]--- [80810.438090] ath10k_pci 0001:01:00.0: Spurious quick kickout for STA `30:23:03:55:b3:0d``

If this is airtime fairness again, I can delete these messages. Just trying to help out where I can.

greearb commented 4 years ago

I think this indicates atf issue:

58071.902018] [] (warn_slowpath_fmt) from [] (ieee80211_sta_update_pending_airtime+0x1f8/0x1fc [mac80211]) [58071.909650] [] (ieee80211_sta_update_pending_airtime [mac80211]) from [] (ieee80211_report_low_ack+0x254/0x50c