search

greearb / ath10k-ct

Stand-alone ath10k driver based on Candela Technologies Linux kernel.
111 stars 41 forks source link

Firmware crash on UniFi AC Mesh #178

Closed kroeckx closed 3 years ago

kroeckx commented 3 years ago

I've recently upgraded my UniFi AC Mesh to OpenWRT 21.02 snapshot r15831-c74df745fd

After running a while, I had the following in dmesg:

[    0.000000] Linux version 5.4.99 (builder@buildhost) (gcc version 8.4.0 (OpenWrt GCC 8.4.0 r15831-c74df745fd)) #0 Fri Feb 19 12:19:59 2021
[    0.000000] printk: bootconsole [early0] enabled
[    0.000000] CPU0 revision is: 00019750 (MIPS 74Kc)
[    0.000000] MIPS: machine is Ubiquiti UniFi AC Mesh
[    0.000000] SoC: Qualcomm Atheros QCA956X ver 1 rev 0
[    0.000000] Initrd not found or empty - disabling initrd
[    0.000000] Primary instruction cache 64kB, VIPT, 4-way, linesize 32 bytes.
[    0.000000] Primary data cache 32kB, 4-way, VIPT, cache aliases, linesize 32 bytes
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x0000000007ffffff]
[    0.000000] On node 0 totalpages: 32768
[    0.000000]   Normal zone: 288 pages used for memmap
[    0.000000]   Normal zone: 0 pages reserved
[    0.000000]   Normal zone: 32768 pages, LIFO batch:7
[    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[    0.000000] pcpu-alloc: [0] 0 
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 32480
[    0.000000] Kernel command line: console=ttyS0,115200n8 rootfstype=squashfs,jffs2
[    0.000000] Dentry cache hash table entries: 16384 (order: 4, 65536 bytes, linear)
[    0.000000] Inode-cache hash table entries: 8192 (order: 3, 32768 bytes, linear)
[    0.000000] Writing ErrCtl register=00000000
[    0.000000] Readback ErrCtl register=00000000
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Memory: 122248K/131072K available (5223K kernel code, 189K rwdata, 680K rodata, 1196K init, 205K bss, 8824K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] NR_IRQS: 51
[    0.000000] random: get_random_bytes called from 0x806559fc with crng_init=0
[    0.000000] CPU clock: 775.000 MHz
[    0.000000] clocksource: MIPS: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 4932285024 ns
[    0.000007] sched_clock: 32 bits at 387MHz, resolution 2ns, wraps every 5541893118ns
[    0.008227] Calibrating delay loop... 385.02 BogoMIPS (lpj=770048)
[    0.046713] pid_max: default: 32768 minimum: 301
[    0.051712] Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.059427] Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.071402] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.081711] futex hash table entries: 256 (order: -1, 3072 bytes, linear)
[    0.088988] pinctrl core: initialized pinctrl subsystem
[    0.096879] NET: Registered protocol family 16
[    0.124232] workqueue: max_active 576 requested for napi_workq is out of range, clamping between 1 and 512
[    0.137800] clocksource: Switched to clocksource MIPS
[    0.144258] NET: Registered protocol family 2
[    0.149699] tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 4096 bytes, linear)
[    0.158577] TCP established hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.166665] TCP bind hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.174104] TCP: Hash tables configured (established 1024 bind 1024)
[    0.180928] UDP hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.187864] UDP-Lite hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.195504] NET: Registered protocol family 1
[    0.200158] PCI: CLS 0 bytes, default 32
[    0.207852] workingset: timestamp_bits=14 max_order=15 bucket_order=1
[    0.220863] squashfs: version 4.0 (2009/01/31) Phillip Lougher
[    0.227050] jffs2: version 2.2 (NAND) (SUMMARY) (LZMA) (RTIME) (CMODE_PRIORITY) (c) 2001-2006 Red Hat, Inc.
[    0.250194] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[    0.261419] pinctrl-single 1804002c.pinmux: 544 pins, size 68
[    0.268498] Serial: 8250/16550 driver, 16 ports, IRQ sharing enabled
[    0.277381] printk: console [ttyS0] disabled
[    0.281968] 18020000.uart: ttyS0 at MMIO 0x18020000 (irq = 9, base_baud = 1562500) is a 16550A
[    0.291073] printk: console [ttyS0] enabled
[    0.300130] printk: bootconsole [early0] disabled
[    0.319582] spi-nor spi0.0: mx25l12805d (16384 Kbytes)
[    0.324990] 7 fixed-partitions partitions found on MTD device spi0.0
[    0.331564] Creating 7 MTD partitions on "spi0.0":
[    0.336534] 0x000000000000-0x000000060000 : "u-boot"
[    0.342574] 0x000000060000-0x000000070000 : "u-boot-env"
[    0.348951] 0x000000070000-0x000000800000 : "firmware"
[    0.358429] 2 uimage-fw partitions found on MTD device firmware
[    0.364586] Creating 2 MTD partitions on "firmware":
[    0.369731] 0x000000000000-0x000000200000 : "kernel"
[    0.375683] 0x000000200000-0x000000790000 : "rootfs"
[    0.381672] mtd: device 4 (rootfs) set to be root filesystem
[    0.389274] 1 squashfs-split partitions found on MTD device rootfs
[    0.395728] 0x000000550000-0x000000790000 : "rootfs_data"
[    0.402244] 0x000000800000-0x000000f90000 : "kernel1"
[    0.408336] 0x000000f90000-0x000000fb0000 : "bs"
[    0.414053] 0x000000fb0000-0x000000ff0000 : "cfg"
[    0.419807] 0x000000ff0000-0x000001000000 : "art"
[    0.427397] libphy: Fixed MDIO Bus: probed
[    1.082165] libphy: ag71xx_mdio: probed
[    1.088221] ag71xx 19000000.eth: connected to PHY at mdio.0:04 [uid=004dd074, driver=Atheros 8031 ethernet]
[    1.098896] eth0: Atheros AG71xx at 0xb9000000, irq 4, mode: sgmii
[    1.105624] i2c /dev entries driver
[    1.111241] NET: Registered protocol family 10
[    1.121067] Segment Routing with IPv6
[    1.125003] NET: Registered protocol family 17
[    1.129686] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[    1.143085] 8021q: 802.1Q VLAN Support v1.8
[    1.148242] PCI host bridge /ahb/pcie-controller@18250000 ranges:
[    1.154601]  MEM 0x0000000012000000..0x0000000013ffffff
[    1.160007]   IO 0x0000000000000000..0x0000000000000000
[    1.165564] PCI host bridge to bus 0000:00
[    1.169833] pci_bus 0000:00: root bus resource [mem 0x12000000-0x13ffffff]
[    1.176939] pci_bus 0000:00: root bus resource [io  0x0000]
[    1.182700] pci_bus 0000:00: root bus resource [??? 0x00000000 flags 0x0]
[    1.189709] random: fast init done
[    1.193227] pci_bus 0000:00: No busn resource found for root bus, will use [bus 00-ff]
[    1.201454] pci 0000:00:00.0: [168c:003c] type 00 class 0x028000
[    1.207715] pci 0000:00:00.0: reg 0x10: [mem 0x00000000-0x001fffff 64bit]
[    1.214780] pci 0000:00:00.0: reg 0x30: [mem 0x00000000-0x0000ffff pref]
[    1.221795] pci 0000:00:00.0: supports D1
[    1.225949] pci 0000:00:00.0: PME# supported from D0 D1 D3hot
[    1.232724] pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to 00
[    1.239606] pci 0000:00:00.0: BAR 0: assigned [mem 0x12000000-0x121fffff 64bit]
[    1.247184] pci 0000:00:00.0: BAR 6: assigned [mem 0x12200000-0x1220ffff pref]
[    1.255573] hctosys: unable to open rtc device (rtc0)
[    1.266252] VFS: Mounted root (squashfs filesystem) readonly on device 31:4.
[    1.280117] Freeing unused kernel memory: 1196K
[    1.284818] This architecture does not have kernel memory protection.
[    1.291470] Run /sbin/init as init process
[    1.862623] init: Console is alive
[    1.866419] init: - watchdog -
[    2.857184] kmodloader: loading kernel modules from /etc/modules-boot.d/*
[    3.015036] kmodloader: done loading kernel modules from /etc/modules-boot.d/*
[    3.033310] init: - preinit -
[    4.294215] random: jshn: uninitialized urandom read (4 bytes read)
[    4.560794] random: jshn: uninitialized urandom read (4 bytes read)
[    4.592655] random: jshn: uninitialized urandom read (4 bytes read)
[    8.847392] eth0: link up (1000Mbps/Full duplex)
[    8.855261] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[    9.037724] jffs2: notice: (542) jffs2_build_xattr_subsystem: complete building xattr subsystem, 23 of xdatum (22 unchecked, 1 orphan) and 31 of xref (1 dead, 0 orphan) found.
[    9.055884] mount_root: switching to jffs2 overlay
[    9.066974] overlayfs: upper fs does not support tmpfile.
[    9.095642] urandom-seed: Seeding with /etc/urandom.seed
[    9.178757] eth0: link down
[    9.195156] procd: - early -
[    9.198297] procd: - watchdog -
[    9.829918] procd: - watchdog -
[    9.835709] procd: - ubus -
[    9.898006] urandom_read: 5 callbacks suppressed
[    9.898012] random: ubusd: uninitialized urandom read (4 bytes read)
[    9.912653] random: ubusd: uninitialized urandom read (4 bytes read)
[    9.926659] procd: - init -
[   10.911514] kmodloader: loading kernel modules from /etc/modules.d/*
[   11.125637] Loading modules backported from Linux version v5.10.16-0-gde53befa79cf
[   11.133522] Backport generated by backports.git v5.10.16-1-0-g21d2a1d2
[   11.194232] xt_time: kernel timezone is -0000
[   11.202290] urngd: v1.0.2 started.
[   11.427909] PPP generic driver version 2.4.2
[   11.442529] NET: Registered protocol family 24
[   11.522192] ath10k 5.10 driver, optimized for CT firmware, probing pci device: 0x3c.
[   11.543995] ath10k_pci 0000:00:00.0: enabling device (0000 -> 0002)
[   11.550665] ath10k_pci 0000:00:00.0: pci irq legacy oper_irq_mode 1 irq_mode 0 reset_mode 0
[   11.568739] random: crng init done
[   14.940713] ath10k_pci 0000:00:00.0: qca988x hw2.0 target 0x4100016c chip_id 0x043222ff sub 0000:0000
[   14.950297] ath10k_pci 0000:00:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0
[   14.962522] ath10k_pci 0000:00:00.0: firmware ver 10.1-ct-8x-__fW-022-ecad3248 api 2 features wmi-10.x,has-wmi-mgmt-tx,mfp,txstatus-noack,wmi-10.x-CT,ratemask-CT,txrate-CT,get-temp-CT,tx-rc-CT,cust-stats-CT,retry-gt2-CT,txrate2-CT,beacon-cb-CT,wmi-block-ack-CT crc32 3e4cf97f
[   15.332980] ath10k_pci 0000:00:00.0: board_file api 1 bmi_id N/A crc32 bebc7c08
[   16.276621] ath10k_pci 0000:00:00.0: 10.1 wmi init: vdevs: 16  peers: 127  tid: 256
[   16.294559] ath10k_pci 0000:00:00.0: wmi print 'P 128 V 8 T 410'
[   16.302986] ath10k_pci 0000:00:00.0: wmi print 'msdu-desc: 1424  sw-crypt: 0 ct-sta: 0'
[   16.311281] ath10k_pci 0000:00:00.0: wmi print 'alloc rem: 24984 iram: 38672'
[   16.373192] ath10k_pci 0000:00:00.0: htt-ver 2.1 wmi-op 2 htt-op 2 cal file max-sta 128 raw 0 hwcrypto 1
[   16.388582] ath10k_pci 0000:00:00.0: NOTE:  Firmware DBGLOG output disabled in debug_mask: 0x10000000
[   16.501593] ath: EEPROM regdomain sanitized
[   16.501604] ath: EEPROM regdomain: 0x64
[   16.501607] ath: EEPROM indicates we should expect a direct regpair map
[   16.501625] ath: Country alpha2 being used: 00
[   16.501628] ath: Regpair used: 0x64
[   16.592352] ath: EEPROM regdomain sanitized
[   16.592363] ath: EEPROM regdomain: 0x64
[   16.592366] ath: EEPROM indicates we should expect a direct regpair map
[   16.592385] ath: Country alpha2 being used: 00
[   16.592388] ath: Regpair used: 0x64
[   16.603334] ieee80211 phy1: Selected rate control algorithm 'minstrel_ht'
[   16.605183] ieee80211 phy1: Atheros AR9561 Rev:0 mem=0xb8100000, irq=2
[   16.682113] kmodloader: done loading kernel modules from /etc/modules.d/*
[   29.947988] br-lan: port 1(eth0) entered blocking state
[   29.953455] br-lan: port 1(eth0) entered disabled state
[   29.959141] device eth0 entered promiscuous mode
[   31.663992] ath: EEPROM regdomain: 0x8038
[   31.668183] ath: EEPROM indicates we should expect a country code
[   31.674492] ath: doing EEPROM country->regdmn map search
[   31.679980] ath: country maps to regdmn code: 0x37
[   31.684927] ath: Country alpha2 being used: BE
[   31.689512] ath: Regpair used: 0x37
[   31.693117] ath: regdomain 0x8038 dynamically updated by user
[   31.699120] ath: EEPROM regdomain: 0x8038
[   31.703274] ath: EEPROM indicates we should expect a country code
[   31.709566] ath: doing EEPROM country->regdmn map search
[   31.715050] ath: country maps to regdmn code: 0x37
[   31.719994] ath: Country alpha2 being used: BE
[   31.724582] ath: Regpair used: 0x37
[   31.728185] ath: regdomain 0x8038 dynamically updated by user
[   33.073375] eth0: link up (1000Mbps/Full duplex)
[   33.088770] br-lan: port 1(eth0) entered blocking state
[   33.094230] br-lan: port 1(eth0) entered forwarding state
[   33.129855] IPv6: ADDRCONF(NETDEV_CHANGE): br-lan: link becomes ready
[   33.662123] br-lan: port 2(wlan1) entered blocking state
[   33.667658] br-lan: port 2(wlan1) entered disabled state
[   33.673438] device wlan1 entered promiscuous mode
[   33.678533] br-lan: port 2(wlan1) entered blocking state
[   33.684059] br-lan: port 2(wlan1) entered forwarding state
[   35.011074] ath10k_pci 0000:00:00.0: 10.1 wmi init: vdevs: 16  peers: 127  tid: 256
[   35.029040] ath10k_pci 0000:00:00.0: wmi print 'P 128 V 8 T 410'
[   35.035766] ath10k_pci 0000:00:00.0: wmi print 'msdu-desc: 1424  sw-crypt: 0 ct-sta: 0'
[   35.044098] ath10k_pci 0000:00:00.0: wmi print 'alloc rem: 24984 iram: 38672'
[   35.132691] ath10k_pci 0000:00:00.0: pdev param 0 not supported by firmware
[   35.148132] ath10k_pci 0000:00:00.0: rts threshold -1
[   35.154588] br-lan: port 2(wlan1) entered disabled state
[   35.172307] br-lan: port 3(wlan0) entered blocking state
[   35.177868] br-lan: port 3(wlan0) entered disabled state
[   35.183665] device wlan0 entered promiscuous mode
[   42.559616] IPv6: ADDRCONF(NETDEV_CHANGE): wlan1: link becomes ready
[   42.566423] br-lan: port 2(wlan1) entered blocking state
[   42.571953] br-lan: port 2(wlan1) entered forwarding state
[   53.000506] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[   53.007328] br-lan: port 3(wlan0) entered blocking state
[   53.012872] br-lan: port 3(wlan0) entered forwarding state
[ 1715.191720] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1715.294116] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1715.396516] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1715.498905] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1715.601318] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1715.703702] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1715.806115] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1715.908510] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1716.010907] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1716.113290] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1716.215688] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1716.318109] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1716.420502] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1716.522896] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1716.625301] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1716.727684] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1716.830098] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1716.932500] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1717.034895] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1717.137283] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1717.239690] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1717.342114] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1717.444500] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1717.546894] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1717.649293] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1717.751693] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1717.854094] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1717.956495] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1718.058892] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1718.161286] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1718.263681] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1718.271341] ath10k_pci 0000:00:00.0: Cannot communicate with firmware, previous wmi cmds: 36904:353754 36954:353752 36904:353749 36952:353584, jiffies: 354560, attempting to fake crash and restart firmware, dev-flags: 0x42
[ 1718.291750] ath10k_pci 0000:00:00.0: failed to transmit management frame via WMI: -11
[ 1718.300015] ath10k_pci 0000:00:00.0: failed to send pdev bss chan info request: -143
[ 1718.308461] ath10k_pci 0000:00:00.0: failed to send wmi nop: -143
[ 1718.314811] ath10k_pci 0000:00:00.0: could not request stats (type -268435456 ret -143 specifier 1)
[ 1718.325136] ath10k_pci 0000:00:00.0: failed to set beacon mode for vdev 0: -143
[ 1718.332778] ath10k_pci 0000:00:00.0: failed to set dtim period for vdev 0: -143
[ 1718.340655] ath10k_pci 0000:00:00.0: firmware crashed! (guid a52d4b9b-9667-488b-ab50-d1d73889fafc)
[ 1718.350008] ath10k_pci 0000:00:00.0: qca988x hw2.0 target 0x4100016c chip_id 0x043222ff sub 0000:0000
[ 1718.359543] ath10k_pci 0000:00:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0
[ 1718.368055] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1718.379465] ath10k_pci 0000:00:00.0: firmware ver 10.1-ct-8x-__fW-022-ecad3248 api 2 features wmi-10.x,has-wmi-mgmt-tx,mfp,txstatus-noack,wmi-10.x-CT,ratemask-CT,regdump-CT,txrate-CT,flush-all-CT,pingpong-CT,ch-regs-CT,nop-CT,set-special-CT,get-temp-CT,tx-rc-CT,cust-stats-CT,retry-gt2-CT,txrate2-CT,beacon-cb-CT,wmi-block-ack-CT crc32 3e4cf97f
[ 1718.410832] ath10k_pci 0000:00:00.0: board_file api 1 bmi_id N/A crc32 bebc7c08
[ 1718.418398] ath10k_pci 0000:00:00.0: htt-ver 2.1 wmi-op 2 htt-op 2 cal file max-sta 128 raw 0 hwcrypto 1
[ 1718.428286] ath10k_pci 0000:00:00.0: firmware register dump:
[ 1718.434145] ath10k_pci 0000:00:00.0: [00]: 0x00940750 0x00400C00 0x00980000 0x009AF0DC
[ 1718.442379] ath10k_pci 0000:00:00.0: [04]: 0x009AF148 0x009AF264 0x00941B6C 0x00941B20
[ 1718.450590] ath10k_pci 0000:00:00.0: [08]: 0x00941B00 0x009423A4 0x009422DC 0x009422C4
[ 1718.458779] ath10k_pci 0000:00:00.0: [12]: 0x00941B90 0x009423F4 0x009423D4 0x009423DC
[ 1718.466967] ath10k_pci 0000:00:00.0: [16]: 0x009423E4 0x009423EC 0x00942888 0x00942520
[ 1718.475167] ath10k_pci 0000:00:00.0: [20]: 0x009424FC 0x0094241C 0x00942540 0x00942498
[ 1718.483381] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1718.491007] ath10k_pci 0000:00:00.0: [24]: 0x00942450 0x0094250C 0x009424D0 0x00942750
[ 1718.499190] ath10k_pci 0000:00:00.0: [28]: 0x00942798 0x00942844 0x009428C4 0x00942594
[ 1718.507384] ath10k_pci 0000:00:00.0: [32]: 0x0094258C 0x00942C28 0x00942F4C 0x00942F7C
[ 1718.515575] ath10k_pci 0000:00:00.0: [36]: 0x00942F90 0x00942FE4 0x00942FF8 0x00943040
[ 1718.523758] ath10k_pci 0000:00:00.0: [40]: 0x0094077C 0x00940790 0x00943084 0x00942F00
[ 1718.531945] ath10k_pci 0000:00:00.0: [44]: 0x009AF290 0x00942A80 0x00942D40 0x00942D64
[ 1718.540127] ath10k_pci 0000:00:00.0: [48]: 0x00942D78 0x00942D9C 0x00957E18 0x00957E28
[ 1718.548338] ath10k_pci 0000:00:00.0: [52]: 0x00957E20 0x00940788 0x009430EC 0x00000000
[ 1718.556549] ath10k_pci 0000:00:00.0: [56]: 0x00000000 0x00000000 0x00000000 0x00000000
[ 1718.564731] ath10k_pci 0000:00:00.0: Copy Engine register dump:
[ 1718.570890] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1718.578520] ath10k_pci 0000:00:00.0: [00]: 0x00057400   8   8   3   3
[ 1718.585189] ath10k_pci 0000:00:00.0: [01]: 0x00057800  13  13 111 112
[ 1718.591857] ath10k_pci 0000:00:00.0: [02]: 0x00057c00   8   8   7   8
[ 1718.598526] ath10k_pci 0000:00:00.0: [03]: 0x00058000   0   0   0   0
[ 1718.605192] ath10k_pci 0000:00:00.0: [04]: 0x00058400 2781 2781 152 112
[ 1718.612035] ath10k_pci 0000:00:00.0: [05]: 0x00058800   0   0 447 448
[ 1718.618718] ath10k_pci 0000:00:00.0: [06]: 0x00058c00  19  19  19  19
[ 1718.625388] ath10k_pci 0000:00:00.0: [07]: 0x00059000   0   0   0   0
[ 1718.632136] ath10k_pci 0000:00:00.0: debug log header, dbuf: 0x411ac0  dropped: 0
[ 1718.639942] ath10k_pci 0000:00:00.0: [0] next: 0x411aa8 buf: 0x40fefc sz: 1500 len: 416 count: 26 free: 0
[ 1718.649941] ath10k_pci 0000:00:00.0: ath10k_pci ATH10K_DBG_BUFFER:
[ 1718.656389] ath10k: [0000]: A9031B00 204CFC0B 0A000000 0B000000 BE031B00 204CFC0B 0A000000 0B000000
[ 1718.665737] ath10k: [0008]: D2031B00 204CFC0B 0A000000 0B000000 E7031B00 204CFC0B 0A000000 0B000000
[ 1718.675125] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1718.682749] ath10k: [0016]: FB031B00 204CFC0B 0A000000 0B000000 10041B00 204CFC0B 0A000000 0B000000
[ 1718.692106] ath10k: [0024]: 24041B00 204CFC0B 0A000000 0B000000 39041B00 204CFC0B 0A000000 0B000000
[ 1718.701521] ath10k: [0032]: 4D041B00 204CFC0B 0A000000 0B000000 62041B00 204CFC0B 0A000000 0B000000
[ 1718.710887] ath10k: [0040]: 76041B00 204CFC0B 0A000000 0B000000 8B041B00 204CFC0B 0A000000 0B000000
[ 1718.720246] ath10k: [0048]: 9F041B00 204CFC0B 0A000000 0B000000 B4041B00 204CFC0B 0A000000 0B000000
[ 1718.729593] ath10k: [0056]: C8041B00 204CFC0B 0A000000 0B000000 DD041B00 204CFC0B 0A000000 0B000000
[ 1718.738944] ath10k: [0064]: F1041B00 204CFC0B 0A000000 0B000000 06051B00 204CFC0B 0A000000 0B000000
[ 1718.748324] ath10k: [0072]: 1A051B00 204CFC0B 0A000000 0B000000 2F051B00 204CFC0B 0A000000 0B000000
[ 1718.757698] ath10k: [0080]: 43051B00 204CFC0B 0A000000 0B000000 58051B00 204CFC0B 0A000000 0B000000
[ 1718.767054] ath10k: [0088]: 6C051B00 204CFC0B 0A000000 0B000000 81051B00 204CFC0B 0A000000 0B000000
[ 1718.776443] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 1718.784071] ath10k: [0096]: 95051B00 204CFC0B 0A000000 0B000000 AA051B00 204CFC0B 0A000000 0B000000
[ 1718.793416] ath10k_pci 0000:00:00.0: ATH10K_END
[ 1718.798181] ath10k_pci 0000:00:00.0: [1] next: 0x411ac0 buf: 0x40f90c sz: 1500 len: 0 count: 0 free: 0
[ 1718.813280] ath10k_pci 0000:00:00.0: failed to set cts protection for vdev 0: -143
[ 1718.821166] ath10k_pci 0000:00:00.0: failed to recalculate rts/cts prot for vdev 0: -143
[ 1718.829586] ath10k_pci 0000:00:00.0: failed to set preamble for vdev 0: -143
[ 1718.836901] ath10k_pci 0000:00:00.0: failed to set mgmt tx rate -143
[ 1718.845082] ath10k_pci 0000:00:00.0: removing peer, cleanup-all, deleting: peer efa6cb31 vdev: 0 addr: b4:fb:e4:75:27:0b 
[ 1718.960197] ieee80211 phy0: Hardware restart was requested
[ 1719.902509] ath10k_pci 0000:00:00.0: 10.1 wmi init: vdevs: 16  peers: 127  tid: 256
[ 1719.920583] ath10k_pci 0000:00:00.0: wmi print 'P 128 V 8 T 410'
[ 1719.926830] ath10k_pci 0000:00:00.0: wmi print 'msdu-desc: 1424  sw-crypt: 0 ct-sta: 0'
[ 1719.935389] ath10k_pci 0000:00:00.0: wmi print 'alloc rem: 24984 iram: 38672'
[ 1720.007886] ath10k_pci 0000:00:00.0: pdev param 0 not supported by firmware
[ 1720.015365] ath10k_pci 0000:00:00.0: set-coverage-class, phyclk: 88  value: 0
[ 1720.023696] ath10k_pci 0000:00:00.0: rts threshold -1
[ 1720.037872] ath10k_pci 0000:00:00.0: device successfully recovered
[ 3699.295662] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3699.398073] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3699.500476] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3699.602872] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3699.705259] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3699.807663] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3699.910059] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3700.012456] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3700.114863] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3700.217260] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3700.319651] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3700.422055] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3700.524458] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3700.626843] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3700.729238] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3700.831641] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3700.934042] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3701.036432] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3701.138841] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3701.241238] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3701.343634] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3701.446033] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3701.548436] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3701.650836] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3701.753240] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3701.855643] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3701.958035] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3702.060443] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3702.162839] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3702.265225] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3702.367645] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3702.470052] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3702.572459] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3702.674843] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3702.764066] ath10k_pci 0000:00:00.0: Cannot communicate with firmware, previous wmi cmds: 40859:849904 36904:849785 36904:849780 36904:849775, jiffies: 850688, attempting to fake crash and restart firmware, dev-flags: 0x42
[ 3702.784521] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3702.792166] ath10k_pci 0000:00:00.0: failed to send wmi nop: -11
[ 3702.798403] ath10k_pci 0000:00:00.0: could not request stats (type -268435456 ret -143 specifier 1)
[ 3702.808429] ath10k_pci 0000:00:00.0: failed to send pdev bss chan info request: -143
[ 3702.817181] ath10k_pci 0000:00:00.0: failed to set beacon mode for vdev 0: -143
[ 3702.824813] ath10k_pci 0000:00:00.0: failed to set dtim period for vdev 0: -143
[ 3702.832894] ath10k_pci 0000:00:00.0: failed to set cts protection for vdev 0: -143
[ 3702.840804] ath10k_pci 0000:00:00.0: failed to recalculate rts/cts prot for vdev 0: -143
[ 3702.849185] ath10k_pci 0000:00:00.0: failed to set preamble for vdev 0: -143
[ 3702.856532] ath10k_pci 0000:00:00.0: failed to set mgmt tx rate -143
[ 3702.863467] ath10k_pci 0000:00:00.0: firmware crashed! (guid b490e93b-c62d-4dbe-99ee-545f21075462)
[ 3702.872805] ath10k_pci 0000:00:00.0: qca988x hw2.0 target 0x4100016c chip_id 0x043222ff sub 0000:0000
[ 3702.882397] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3702.890025] ath10k_pci 0000:00:00.0: kconfig debug 0 debugfs 1 tracing 0 dfs 1 testmode 0
[ 3702.902267] ath10k_pci 0000:00:00.0: firmware ver 10.1-ct-8x-__fW-022-ecad3248 api 2 features wmi-10.x,has-wmi-mgmt-tx,mfp,txstatus-noack,wmi-10.x-CT,ratemask-CT,regdump-CT,txrate-CT,flush-all-CT,pingpong-CT,ch-regs-CT,nop-CT,set-special-CT,get-temp-CT,tx-rc-CT,cust-stats-CT,retry-gt2-CT,txrate2-CT,beacon-cb-CT,wmi-block-ack-CT crc32 3e4cf97f
[ 3702.933641] ath10k_pci 0000:00:00.0: board_file api 1 bmi_id N/A crc32 bebc7c08
[ 3702.941210] ath10k_pci 0000:00:00.0: htt-ver 2.1 wmi-op 2 htt-op 2 cal file max-sta 128 raw 0 hwcrypto 1
[ 3702.951094] ath10k_pci 0000:00:00.0: firmware register dump:
[ 3702.956996] ath10k_pci 0000:00:00.0: [00]: 0x00940750 0x00400C00 0x00980000 0x009AF0DC
[ 3702.965206] ath10k_pci 0000:00:00.0: [04]: 0x009AF148 0x009AF264 0x00941B6C 0x00941B20
[ 3702.973396] ath10k_pci 0000:00:00.0: [08]: 0x00941B00 0x009423A4 0x009422DC 0x009422C4
[ 3702.981658] ath10k_pci 0000:00:00.0: [12]: 0x00941B90 0x009423F4 0x009423D4 0x009423DC
[ 3702.989869] ath10k_pci 0000:00:00.0: [16]: 0x009423E4 0x009423EC 0x00942888 0x00942520
[ 3702.998107] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3703.005727] ath10k_pci 0000:00:00.0: [20]: 0x009424FC 0x0094241C 0x00942540 0x00942498
[ 3703.013926] ath10k_pci 0000:00:00.0: [24]: 0x00942450 0x0094250C 0x009424D0 0x00942750
[ 3703.022138] ath10k_pci 0000:00:00.0: [28]: 0x00942798 0x00942844 0x009428C4 0x00942594
[ 3703.030331] ath10k_pci 0000:00:00.0: [32]: 0x0094258C 0x00942C28 0x00942F4C 0x00942F7C
[ 3703.038526] ath10k_pci 0000:00:00.0: [36]: 0x00942F90 0x00942FE4 0x00942FF8 0x00943040
[ 3703.046714] ath10k_pci 0000:00:00.0: [40]: 0x0094077C 0x00940790 0x00943084 0x00942F00
[ 3703.054896] ath10k_pci 0000:00:00.0: [44]: 0x009AF290 0x00942A80 0x00942D40 0x00942D64
[ 3703.063124] ath10k_pci 0000:00:00.0: [48]: 0x00942D78 0x00942D9C 0x00957E18 0x00957E28
[ 3703.071332] ath10k_pci 0000:00:00.0: [52]: 0x00957E20 0x00940788 0x009430EC 0x00000000
[ 3703.079518] ath10k_pci 0000:00:00.0: [56]: 0x00000000 0x00000000 0x00000000 0x00000000
[ 3703.087738] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3703.095359] ath10k_pci 0000:00:00.0: Copy Engine register dump:
[ 3703.101495] ath10k_pci 0000:00:00.0: [00]: 0x00057400   8   8   3   3
[ 3703.108161] ath10k_pci 0000:00:00.0: [01]: 0x00057800  11  11 109 110
[ 3703.114829] ath10k_pci 0000:00:00.0: [02]: 0x00057c00  31  31  30  31
[ 3703.121495] ath10k_pci 0000:00:00.0: [03]: 0x00058000   4   4   4   4
[ 3703.128158] ath10k_pci 0000:00:00.0: [04]: 0x00058400   3   3  43   3
[ 3703.134827] ath10k_pci 0000:00:00.0: [05]: 0x00058800  18  18  17  18
[ 3703.141492] ath10k_pci 0000:00:00.0: [06]: 0x00058c00  27  27  27  27
[ 3703.148160] ath10k_pci 0000:00:00.0: [07]: 0x00059000   0   0   0   0
[ 3703.154900] ath10k_pci 0000:00:00.0: debug log header, dbuf: 0x411ac0  dropped: 0
[ 3703.162743] ath10k_pci 0000:00:00.0: [0] next: 0x411aa8 buf: 0x40fefc sz: 1500 len: 144 count: 9 free: 0
[ 3703.172642] ath10k_pci 0000:00:00.0: ath10k_pci ATH10K_DBG_BUFFER:
[ 3703.179045] ath10k: [0000]: 27073A00 204CFC0B 0A000000 0B000000 3B073A00 204CFC0B 0A000000 0B000000
[ 3703.188435] ath10k_pci 0000:00:00.0: SWBA overrun on vdev 0, skipped old beacon
[ 3703.196056] ath10k: [0008]: 50073A00 204CFC0B 0A000000 0B000000 64073A00 204CFC0B 0A000000 0B000000
[ 3703.205562] ath10k: [0016]: 79073A00 204CFC0B 0A000000 0B000000 8D073A00 204CFC0B 0A000000 0B000000
[ 3703.214952] ath10k: [0024]: A2073A00 204CFC0B 0A000000 0B000000 B6073A00 204CFC0B 0A000000 0B000000
[ 3703.224308] ath10k: [0032]: CB073A00 204CFC0B 0A000000 0B000000
[ 3703.230431] ath10k_pci 0000:00:00.0: ATH10K_END
[ 3703.235144] ath10k_pci 0000:00:00.0: [1] next: 0x411ac0 buf: 0x40f90c sz: 1500 len: 0 count: 0 free: 0
[ 3703.250372] ath10k_pci 0000:00:00.0: removing peer, cleanup-all, deleting: peer da376718 vdev: 0 addr: b4:fb:e4:75:27:0b 
[ 3703.363733] ieee80211 phy0: Hardware restart was requested
[ 3704.305809] ath10k_pci 0000:00:00.0: 10.1 wmi init: vdevs: 16  peers: 127  tid: 256
[ 3704.323776] ath10k_pci 0000:00:00.0: wmi print 'P 128 V 8 T 410'
[ 3704.330254] ath10k_pci 0000:00:00.0: wmi print 'msdu-desc: 1424  sw-crypt: 0 ct-sta: 0'
[ 3704.338570] ath10k_pci 0000:00:00.0: wmi print 'alloc rem: 24984 iram: 38672'
[ 3704.411044] ath10k_pci 0000:00:00.0: pdev param 0 not supported by firmware
[ 3704.426473] ath10k_pci 0000:00:00.0: rts threshold -1
[ 3704.440636] ath10k_pci 0000:00:00.0: device successfully recovered
SimonIremonger commented 3 years ago

I can confirm getting much the same errors, firmware crash, reset, etc... using a BT-HomeHub-v5-Type-A with the same OpenWRT 21.02SNAPSHOT with same ath10k firmware CT as well. I also have older logs from running older 'master' build of OpenWRT from January without issue I believe. I would like to provide more info / collect more debugging from console(serial)-log and syslog, if I know what debugging to turn on and what to get // what would be worthwhile....

greearb commented 3 years ago

I cannot make progress on firmware crashes that are for this reason: [ 3702.764066] ath10k_pci 0000:00:00.0: Cannot communicate with firmware, previous wmi cmds: 40859:849904 36904:849785 36904:849780 36904:849775, jiffies: 850688, attempting to fake crash and restart firmware, dev-flags: 0x42 The firmware stopped responding to the driver, so driver fakes a crash to recover. That is likely as good as I can do. In case those wmi-cmds are always the same in your crashes, just possibly they provide a clue, but I have dug through that code for weeks at a time with not much progress over the years...

kroeckx commented 3 years ago

Note that my dmesg contains 2 such crashes:

ath10k_pci 0000:00:00.0: Cannot communicate with firmware, previous wmi cmds: 36904:353754 36954:353752 36904:353749 36952:353584, jiffies: 354560, attempting to fake crash and restart firmware, dev-flags: 0x42
ath10k_pci 0000:00:00.0: Cannot communicate with firmware, previous wmi cmds: 40859:849904 36904:849785 36904:849780 36904:849775, jiffies: 850688, attempting to fake crash and restart firmware, dev-flags: 0x42

They don't seem to have anything in common.

I've now switched that device to non-ct, and it works without problems so far.

kroeckx commented 3 years ago

Should any firmware version work with the driver I have? If @SimonIremonger can say which firmware version worked I can try to do a bisect, but I guess it will be slow since it's takes time to reproduce.

kroeckx commented 3 years ago

I might have misunderstand what's in this git repo. I now think it's only the driver, not the firmware.

greearb commented 3 years ago

The wave-1 CT firmware is based on a pretty old upstream firmware, and all of it is closed source. If QCA firmware works better for you then, that is probably best. The ath10k-ct driver should work fine with upstream QCA firmware, but the stock ath10k driver may work fine for you as well.

klukonin commented 3 years ago

@greearb I see a possible approach to switch ath10k-ct beacon mode to "BURST". This mode should (I guess) avoid some timing or queuing issues in firmware.

SimonIremonger commented 3 years ago

Informal reports indicate issues here occurring into OpenWRT 21.02 rc (imminent release...) versions, informed users needing to switch to non -ct firmware. Will try to add some serial logging and debugging/monitoring to my ath10k-containing gadgets...

greearb commented 3 years ago

The -ct firmware for wave-2 is completely different beast from the wave-1 -ct firmware. Please be specific about what platform(s) and/or chipsets seem to have issues or not.

CodeFetch commented 3 years ago

@greearb From what I know the UniFi AC Mesh has a rebranded QCA9886. Isn't that wave 2? Is it possible the wrong firmware was chosen for it or something?

greearb commented 3 years ago

9886 is wave-2. 9888 aka 988x is wave-1.

greearb commented 3 years ago

The original bug report was about wave-1, radio times out, and I don't know how to fix. The driver did recover the system after the radio became un-responsive. Closing this bug