Closed tangsoft closed 8 years ago
Here again. CPU stall?
BusyBox v1.24.1 (2015-11-30 21:05:54 CST) built-in shell (ash)
| |.-----.-----.-----.| | | |.----.| | | - || | -| || | | || || | |_____|| |___||||____||| |__| |__| W I R E L E S S F R E E D O M
DESIGNATED DRIVER (Bleeding Edge, r47680)
~# dmesg
[ 29.396954] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready
[ 29.403489] br-lan: port 3(wlan0) entered forwarding state
[ 29.409046] br-lan: port 3(wlan0) entered forwarding state
[ 29.611755] br-lan: port 1(eth0) entered forwarding state
[ 29.811666] br-lan: port 4(wlan1) entered forwarding state
[ 31.401936] br-lan: port 3(wlan0) entered forwarding state
[ 33.116216] pppoe-wan: renamed from ppp0
[ 706.750659] ieee80211 phy1: failed to reallocate TX buffer
[ 706.756273] ieee80211 phy0: failed to reallocate TX buffer
[ 714.051196] ieee80211 phy1: failed to reallocate TX buffer
[ 714.056817] ieee80211 phy0: failed to reallocate TX buffer
[ 736.721747] ieee80211 phy1: failed to reallocate TX buffer
[ 736.727335] ieee80211 phy0: failed to reallocate TX buffer
[ 758.640292] ieee80211 phy1: failed to reallocate TX buffer
[ 758.645855] ieee80211 phy0: failed to reallocate TX buffer
[ 759.140003] ieee80211 phy1: failed to reallocate TX buffer
[ 1174.742847] Unable to handle kernel paging request at virtual address 0000e1d0
[ 1174.750218] pgd = ca264000
[ 1174.753206] [0000e1d0] pgd=0d934831, pte=00000000, *ppte=00000000
[ 1174.759586] Internal error: Oops: 80000007 [#1] SMP ARM
[ 1174.764838] Modules linked in: pppoe ppp_async iptable_nat pppox ppp_generic nf_nat_ipv4 nf_conntrack_ipv6 nf_conntrack_ipv4 ipt_REJECT ipt_MASQUERADE armada_thermal xt_time xt_tcpudp xt_state xt_socket xt_nat xt_multiport xt_mark xt_mac xt_limit xt_id xt_conntrack xt_comment xt_TPROXY xt_TCPMSS xt_REDIRECT xt_LOG xt_CT thermal_sys slhc nf_reject_ipv4 nf_nat_masquerade_ipv4 nf_nat nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_raw iptable_mangle iptable_filter ip_tables crc_ccitt hid evdev input_core tmp421 pwm_fan hwmon i2c_mv64xxx i2c_dev mwlwifi mac80211 cfg80211 compat ledtrig_usbdev xt_set ip_set_list_set ip_set_hash_netiface ip_set_hash_netport ip_set_hash_netnet ip_set_hash_net ip_set_hash_netportnet ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_raw ip6table_mangle ip6table_filter ip6_tables x_tables tun nls_utf8 nls_cp437 regmap_spi regmap_mmio ecb usb_storage leds_tlc59116 regmap_i2c i2c_core xhci_plat_hcd xhci_pci xhci_hcd sata_mv libata ehci_orion ehci_platform ehci_hcd sd_mod scsi_mod gpio_button_hotplug ext4 jbd2 mbcache usbcore nls_base usb_common aead crypto_hash
[ 1174.883886] CPU: 1 PID: 1955 Comm: transmission-da Not tainted 3.18.23 #1
[ 1174.890716] task: cb39a800 ti: ca18c000 task.ti: ca18c000
[ 1174.896172] PC is at 0xe1d0
[ 1174.899003] LR is at inet_sendmsg+0xa8/0xb4
[ 1174.903223] pc : [<0000e1d0>] lr : [
I believe this is related to #44
I did not get any reports related to stall or memory leaking except for you. Can you let me know your test environment?
I check in version 10.3.0.5-20151208. Please use this version to do test. You can check version information via "cat /sys/kernel/debug/ieee80211/phy0/mwlwifi/info". This version is almost the same as previous latest commit except for changing spinlock to mutex for firmware command. I mark a formal release version, so you can make sure which version of driver is tested by you. Thanks.
Well, thank you, @yuhhaurlin, a 2T USB 2.5' hard disk attached to my wrt1900ac v1, /dev/sda1 10G mounted as extroot, /dev/sda2 as swap, the rest as /dev/sda3 for storage of transmissionbt downloading; firmware self compiled from trunk version with mod fs-ext4,block mount, bridge, tun and udp tunnel. Movie streaming on both my iPhone6 and Ipad4 by app named INFUSE over wireless 5G. 2.4G is terriblely slow and can not offer 1080P streaming. I experienced random jitters and sometimes wrt1900ac crashed and rebooted. The 5G performance via my late 2013 MacBook Pro is about 30mbytes per second. But stability is a big concern. Let me know if you need any else information.
Can you test the latest one I commit, so you can check driver version?
Here is the mwlwifi version currently used, give me some time to test the one you mentioned.
~# cat /sys/kernel/debug/ieee80211/phy0/mwlwifi/info
driver name: mwlwifi chip type: 88W8864 hw version: 7 driver version: 10.3.0.14 firmware version: 0x07020806 mac address: 00:25:9c:13:28:f5 2g: enable 5g: disable antenna: 4 4 irq number: 87 iobase0: d1000000 iobase1: d1200000 tx limit: 768 rx limit: 64 ap macid support: 0000ffff sta macid support: 00010000 macid used: 00000001 qe trigger number: 2515366 mfg mode: false
@yuhhaurlin, build 47680'is a build that contains the kernel panic fix, as uploaded by Kaloz to trunk. @tangsoft puzzle is that the version number didn't change the last few fixes. Just this morning yuhhaurlin uploaded some new fixes though and upgraded the driver to .15, with a date included. That should make each iteration of the .15 driver unique and aide in troubleshooting. As he mentioned above, might want to use the that latest driver, before doing any new testing.
BTW: if I understand above info correctly you've disabled the 5ghz interface completely? Not many here would do that. Maybe that houses a clue?
Ok, I am compiling the firmware with new .15 driver.
Just realized the 5G died itself over night. It is not disabled by me. PS: Just double checked, it is so weird, even the info shows 5G is disabled, but I am still able to connect to it.
@tangsoft: sorry my fault. This is just the info for interface phy0. Check phy1 and it should show it just the other way around. I wasn't fully awake yet :-)
@johnnysl, np, thanks for you guys hard work. my box got updated with .15 drive and same trunk version 47680. i will update with the test progress.
~# cat /sys/kernel/debug/ieee80211/phy0/mwlwifi/info
driver name: mwlwifi chip type: 88W8864 hw version: 7 driver version: 10.3.0.15-20151208 firmware version: 0x07020806 mac address: 00:25:9c:13:28:f5 2g: enable 5g: disable antenna: 4 4 irq number: 87 iobase0: d1000000 iobase1: d1200000 tx limit: 768 rx limit: 64 ap macid support: 0000ffff sta macid support: 00010000 macid used: 00000001 qe trigger number: 76251 mfg mode: false
~# cat /sys/kernel/debug/ieee80211/phy1/mwlwifi/info
driver name: mwlwifi chip type: 88W8864 hw version: 7 driver version: 10.3.0.15-20151208 firmware version: 0x07020806 mac address: 00:25:9c:13:28:f6 2g: disable 5g: enable antenna: 4 4 irq number: 88 iobase0: d1400000 iobase1: d1600000 tx limit: 768 rx limit: 64 ap macid support: 0000ffff sta macid support: 00010000 macid used: 00000001 qe trigger number: 76461 mfg mode: false
@yuhhaurlin @johnnysl So far so good with .15 under trunk version 47680. no dmesg errors any more, DLNA streaming is much more smooth as well. Thanks. I will observe and test for more hours.
Hi, I also have this issue, wrt1900ac v1 dmesg errors:
[201482.696427] ieee80211 phy1: failed to reallocate TX buffer
[201482.696477] ieee80211 phy0: failed to reallocate TX buffer
[201661.212525] ieee80211 phy1: failed to reallocate TX buffer
[201661.212574] ieee80211 phy0: failed to reallocate TX buffer
[251419.025524] ieee80211 phy0: check ba result error 1
[251419.030655] ieee80211 phy0: ampdu start error code: -22
[251419.065606] ieee80211 phy0: check ba result error 1
[251419.070680] ieee80211 phy0: ampdu start error code: -22
It takes about 3 days usage to get these error messages, and it seems to be triggered easily when streaming videos. I am using commit 352efe4bdc34204cc528ac9a979bb0032a6996e0 as kaloz commits before.
Here goes driver info
root@OpenWrt:~# cat /sys/kernel/debug/ieee80211/phy1/mwlwifi/info
driver name: mwlwifi
chip type: 88W8864
hw version: 7
driver version: 10.3.0.14
firmware version: 0x07020806
mac address: xxxxxxxxxxxx
2g: disable
5g: enable
antenna: 4 4
irq number: 88
iobase0: d1800000
iobase1: d1a00000
tx limit: 768
rx limit: 64
ap macid support: 0000ffff
sta macid support: 00010000
macid used: 00000001
qe trigger number: 13613611
mfg mode: false
root@OpenWrt:~# cat /sys/kernel/debug/ieee80211/phy0/mwlwifi/info
driver name: mwlwifi
chip type: 88W8864
hw version: 7
driver version: 10.3.0.14
firmware version: 0x07020806
mac address: xxxxxxxxxxxx
2g: enable
5g: disable
antenna: 4 4
irq number: 87
iobase0: d1400000
iobase1: d1600000
tx limit: 768
rx limit: 64
ap macid support: 0000ffff
sta macid support: 00010000
macid used: 00000001
qe trigger number: 13613853
mfg mode: false
I will try the latest commit and report later.
BTW, I also get these dmesg errors after long time getting failed to reallocate TX buffer
and ampdu/ba errors, but I don't know whether it was triggered by mwlwifi or not, only for reference.
[171885.686636] logd: page allocation failure: order:1, mode:0x204020
[171885.692865] CPU: 0 PID: 1179 Comm: logd Not tainted 3.18.23 #1
[171885.698809] Backtrace:
[171885.701387] [<c001c3ec>] (dump_backtrace) from [<c001c608>] (show_stack+0x18/0x1c)
[171885.709078] r7:00000001 r6:00000000 r5:c048d9dc r4:00000000
[171885.714932] [<c001c5f0>] (show_stack) from [<c01974a4>] (dump_stack+0x88/0x98)
[171885.722293] [<c019741c>] (dump_stack) from [<c0090364>] (warn_alloc_failed+0x118/0x13c)
[171885.730414] r5:00204020 r4:c0481008
[171885.734148] [<c0090250>] (warn_alloc_failed) from [<c0092608>] (__alloc_pages_nodemask+0x6dc/0x780)
[171885.743320] r3:00000000 r2:00000000
[171885.747027] r9:00000000 r8:00000001 r7:00000000 r6:00000000 r5:c0485400 r4:00000000
[171885.754990] [<c0091f2c>] (__alloc_pages_nodemask) from [<c00c05cc>] (cache_alloc_refill+0x30c/0x578)
[171885.764249] r10:c0483950 r9:00000000 r8:00000000 r7:cf8b4740 r6:00000240 r5:cfa70ec0
[171885.772291] r4:00000020
[171885.774958] [<c00c02c0>] (cache_alloc_refill) from [<c00c09a8>] (kmem_cache_alloc+0x78/0xdc)
[171885.783521] r10:cd2b4bc0 r9:cd2b4bc0 r8:c0481008 r7:ce0fa640 r6:cf8b4740 r5:a0000113
[171885.791543] r4:00000020
[171885.794219] [<c00c0930>] (kmem_cache_alloc) from [<c0285cf0>] (sk_prot_alloc+0x2c/0xec)
[171885.802346] r7:ce0fa640 r6:00000020 r5:cf8b4740 r4:c04a08e0
[171885.808191] [<c0285cc4>] (sk_prot_alloc) from [<c02860a0>] (sk_clone_lock+0x20/0x1d0)
[171885.816144] r7:ce0fa640 r6:cd2b4bc0 r5:cd1b21c0 r4:ce0fa640
[171885.821991] [<c0286080>] (sk_clone_lock) from [<c02d3c24>] (inet_csk_clone_lock+0x18/0x84)
[171885.830380] r6:cd1b21c0 r5:cd1b21c0 r4:ce0fa640
[171885.835179] [<c02d3c0c>] (inet_csk_clone_lock) from [<c02eb754>] (tcp_create_openreq_child+0x1c/0x3c4)
[171885.844612] r5:cd1b21c0 r4:ce0fa640
[171885.848337] [<c02eb738>] (tcp_create_openreq_child) from [<c02e8940>] (tcp_v4_syn_recv_sock+0x54/0x284)
[171885.857856] r7:00000000 r6:cd1b21c0 r5:ce0fa640 r4:ce0fa640
[171885.863710] [<c02e88ec>] (tcp_v4_syn_recv_sock) from [<c02ebddc>] (tcp_check_req+0x2e0/0x5a8)
[171885.872361] r10:cd2b4bc0 r9:b0401850 r8:cd1b21c0 r7:c0481008 r6:cdc07dec r5:c02e88ec
[171885.880383] r4:ce0fa640
[171885.883055] [<c02ebafc>] (tcp_check_req) from [<c02e8cec>] (tcp_v4_do_rcv+0x17c/0x384)
[171885.891087] r10:0000006c r9:cdc07dec r8:cdc07d80 r7:cdc07dd8 r6:cd2b4bc0 r5:c0481008
[171885.899118] r4:ce0fa640
[171885.901776] [<c02e8b70>] (tcp_v4_do_rcv) from [<c02eadf0>] (tcp_v4_rcv+0x3c4/0x770)
[171885.909554] r10:00001824 r9:0000c8e2 r8:cd2b4bc0 r7:cdc07dd8 r6:00000000 r5:c049eb80
[171885.917588] r4:ce0fa640
[171885.920247] [<c02eaa2c>] (tcp_v4_rcv) from [<c02c7380>] (ip_local_deliver_finish+0xcc/0x1d8)
[171885.928809] r10:00000000 r9:c0482348 r8:cd05c800 r7:00000000 r6:c0483204 r5:ce0fa640
[171885.936844] r4:c04244bc
[171885.939504] [<c02c72b4>] (ip_local_deliver_finish) from [<c02c7970>] (ip_local_deliver+0x68/0xb4)
[171885.948501] r7:c049eb80 r6:ce0fa640 r5:cdc07dd8 r4:ce0fa640
[171885.954355] [<c02c7908>] (ip_local_deliver) from [<c02c77ac>] (ip_rcv_finish+0x320/0x374)
[171885.962658] r4:ce0fa640
[171885.965319] [<c02c748c>] (ip_rcv_finish) from [<c02c7d90>] (ip_rcv+0x3d4/0x4ac)
[171885.972750] r7:c049eb80 r6:ce0fa640 r5:cdc07d80 r4:00000058
[171885.978597] [<c02c79bc>] (ip_rcv) from [<c02978ec>] (__netif_receive_skb_core+0x4d8/0x640)
[171885.986985] r8:00000008 r7:c0482358 r6:c0483788 r5:ce0fa640 r4:c02c79bc
[171885.993902] [<c0297414>] (__netif_receive_skb_core) from [<c0299798>] (__netif_receive_skb+0x74/0x88)
[171886.003248] r10:cfddaab0 r9:cfddaaa4 r8:00000040 r7:00000001 r6:cfddaa54 r5:0000000c
[171886.011270] r4:cfddaab4
[171886.013940] [<c0299724>] (__netif_receive_skb) from [<c029a27c>] (process_backlog+0x94/0x164)
[171886.022590] r5:0000000c r4:cfddaab4
[171886.026312] [<c029a1e8>] (process_backlog) from [<c0299ae4>] (net_rx_action+0x88/0x17c)
[171886.034439] r10:cfddaa48 r9:c0480100 r8:00000040 r7:0000010f r6:cfddaab4 r5:cfddaa40
[171886.042469] r4:00000001 r3:c029a1e8
[171886.046196] [<c0299a5c>] (net_rx_action) from [<c002b308>] (__do_softirq+0x10c/0x238)
[171886.054149] r10:cda68000 r9:00000100 r8:40000001 r7:c0480088 r6:c048008c r5:00000003
[171886.062171] r4:0000000c
[171886.064844] [<c002b1fc>] (__do_softirq) from [<c002b6d0>] (irq_exit+0xa0/0xb0)
[171886.072190] r10:00000897 r9:00000769 r8:cf805000 r7:00000001 r6:00000000 r5:00000000
[171886.080223] r4:c047c8f8
[171886.082895] [<c002b630>] (irq_exit) from [<c005be20>] (__handle_domain_irq+0x94/0xa4)
[171886.090840] r5:00000000 r4:c047c8f8
[171886.094572] [<c005bd8c>] (__handle_domain_irq) from [<c00086d8>] (armada_370_xp_handle_irq+0x58/0xc8)
[171886.103919] r9:00000769 r8:c0481048 r7:c04c4430 r6:cda69e40 r5:c04c4444 r4:c005ab50
[171886.111867] [<c0008680>] (armada_370_xp_handle_irq) from [<c00097e0>] (__irq_svc+0x40/0x54)
[171886.120345] Exception stack(0xcda69e40 to 0xcda69e88)
[171886.125528] 9e40: 00000bf1 00000000 c04a966c 0000622f 0000006f c04a7790 c04a525c c0489060
[171886.133845] 9e60: c04a9704 00000769 00000897 cda69f04 c04a9710 cda69e88 00293438 c005ab50
[171886.142142] 9e80: 20000013 ffffffff
[171886.145742] r10:00000897 r9:00000769 r8:c04a9704 r7:cda69e74 r6:ffffffff r5:20000013
[171886.153777] r4:c005ab50 r3:00293438
[171886.157504] [<c005a810>] (do_syslog) from [<c0115dbc>] (kmsg_read+0x34/0x5c)
[171886.164674] r10:00000000 r9:cda68000 r8:00001000 r7:cf8b3400 r6:00000000 r5:00001000
[171886.172706] r4:00a57040
[171886.175375] [<c0115d88>] (kmsg_read) from [<c010abf0>] (proc_reg_read+0x68/0x98)
[171886.182892] r5:00000000 r4:c0115d88
[171886.186618] [<c010ab88>] (proc_reg_read) from [<c00c4548>] (vfs_read+0x90/0x14c)
[171886.194136] r7:c010ab88 r6:cda69f70 r5:cf9531c0 r4:00a57040
[171886.199976] [<c00c44b8>] (vfs_read) from [<c00c4b58>] (SyS_read+0x5c/0xb8)
[171886.206972] r9:cda68000 r8:00001000 r7:00a57040 r6:cf9531c0 r5:c0481008 r4:cf9531c0
[171886.214930] [<c00c4afc>] (SyS_read) from [<c0008ec0>] (ret_fast_syscall+0x0/0x38)
[171886.222536] r8:c0009064 r7:00000003 r6:00a58050 r5:00000000 r4:00000000
[171886.229423] Mem-info:
[171886.231795] Normal per-cpu:
[171886.234701] CPU 0: hi: 90, btch: 15 usd: 14
[171886.239604] CPU 1: hi: 90, btch: 15 usd: 83
[171886.244527] active_anon:5708 inactive_anon:5789 isolated_anon:0
[171886.244527] active_file:19248 inactive_file:20678 isolated_file:0
[171886.244527] unevictable:0 dirty:0 writeback:0 unstable:0
[171886.244527] free:949 slab_reclaimable:2120 slab_unreclaimable:4510
[171886.244527] mapped:849 shmem:2917 pagetables:119 bounce:0
[171886.244527] free_cma:0
[171886.276821] Normal free:3796kB min:2016kB low:2520kB high:3024kB active_anon:22832kB inactive_anon:23156kB active_file:76992kB inactive_file:82712kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:262144kB managed:255024kB mlocked:0kB dirty:0kB writeback:0kB mapped:3396kB shmem:11668kB slab_reclaimable:8480kB slab_unreclaimable:18040kB kernel_stack:552kB pagetables:476kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[171886.319153] lowmem_reserve[]: 0 0 0
[171886.322813] Normal: 865*4kB (UMR) 9*8kB (R) 10*16kB (R) 2*32kB (R) 1*64kB (R) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3820kB
[171886.335540] 42859 total pagecache pages
[171886.339480] 7 pages in swap cache
[171886.342907] Swap cache stats: add 146, delete 139, find 0/1
[171886.348590] Free swap = 135636kB
[171886.352004] Total swap = 136188kB
[171886.361580] 65536 pages of RAM
[171886.364750] 1202 free pages
[171886.367646] 1780 reserved pages
[171886.370886] 4534 slab pages
[171886.373788] 169828 pages shared
[171886.377032] 7 pages swap cached
Being watching the box for a while, I can tell there is still something not right, let me know what I can do for troubleshooting.
Regularly from output of top command, a process named ksoftirqd consumes a large chunk of CPU and the DLNA streaming experiences a jitter.
2K cached Mem: 251280K used, 3848K free, 21408K shrd, 3088K buff, 190648K cached CPU: 0% usr 1% sys 0% nic 74% idle 7% io 0% irq 15% sirq Load average: 0.25 0.27 0.24 2/87 31494 PID PPID USER STAT VSZ %VSZ %CPU COMMAND 3 2 root SW 0 0% 1% [ksoftirqd/0] 402 2 root SW 0 0% 1% [usb-storage] 26375 1345 root S 13236 5% 1% /usr/bin/minidlna -f /tmp/minidlna.co 2053 1 root S 1560 1% 1% /usr/sbin/hostapd -P /var/run/wifi-ph 158 2 root SW 0 0% 0% [kswapd0] 1266 1 root S 1540 1% 0% /sbin/netifd 31442 2 root DW 0 0% 0% [kworker/0:0] 1839 1 root S 1560 1% 0% /usr/sbin/hostapd -P /var/run/wifi-ph 31489 31417 root R 1088 0% 0% top 1695 1266 root S 1060 0% 0% /usr/sbin/pppd nodetach ipparam wan i 7 2 root SW 0 0% 0% [rcu_sched] 11 2 root SW 0 0% 0% [ksoftirqd/1] 1345 1 root S 13272 5% 0% /usr/bin/minidlna -f /tmp/minidlna.co 1698 1 root S 12016 5% 0% /usr/sbin/asterisk 1506 1 root S 3844 2% 0% /usr/sbin/openvpn --syslog openvpn(ov 1507 1 root S 2876 1% 0% /usr/sbin/openvpn --syslog openvpn(ov 1465 1 root S 2676 1% 0% /usr/sbin/nmbd -F 1464 1 root S 2656 1% 0% /usr/sbin/smbd -F 1 0 root S 1364 1% 0% /sbin/procd 1379 1 root S 1348 1% 0% /usr/sbin/uhttpd
Overall, as far as I can tell, .15 improves the stability, less crash; but performance issue insists.
And tx buffer reallocate failure still happens but looks like less.
[25868.258656] ieee80211 phy0: failed to reallocate TX buffer [26707.461831] ieee80211 phy1: failed to reallocate TX buffer [27307.159178] ieee80211 phy0: failed to reallocate TX buffer [27307.164762] ieee80211 phy1: failed to reallocate TX buffer [27308.190824] ieee80211 phy0: failed to reallocate TX buffer [27308.196408] ieee80211 phy1: failed to reallocate TX buffer
I think memory is gone. Can you try test from ethernet to wifi to see if you still encounter this problem?
@tangsoft: can you run a cat /proc/interrupts for me?
~# cat /proc/interrupts
CPU0 CPU1
16: 7335444 7503802 armada_370_xp_irq 5 armada_370_xp_per_cpu_tick
18: 2739022 0 armada_370_xp_irq 31 mv64xxx_i2c 19: 21 0 armada_370_xp_irq 41 serial
25: 0 0 armada_370_xp_irq 45 ehci_hcd:usb1
26: 233292 0 armada_370_xp_irq 8 mvneta
27: 2388633 0 armada_370_xp_irq 10 mvneta
28: 0 0 armada_370_xp_irq 55 f10a0000.sata
29: 13890 0 armada_370_xp_irq 113 f10d0000.nand
69: 0 0 f1018140.gpio 0 gpio_keys
70: 0 0 f1018140.gpio 1 gpio_keys
87: 97999111 0 armada_370_xp_irq 59 mwlwifi
88: 100925205 0 armada_370_xp_irq 60 mwlwifi
89: 2 0 armada_370_xp_irq 51 f1060900.xor
90: 2 0 armada_370_xp_irq 52 f1060900.xor
91: 2 0 armada_370_xp_irq 94 f10f0900.xor
92: 2 0 armada_370_xp_irq 95 f10f0900.xor
93: 695048 0 armada_370_xp_msi_irq 0 xhci_hcd
IPI0: 0 0 CPU wakeup interrupts
IPI1: 0 0 Timer broadcast interrupts
IPI2: 249048 1102014 Rescheduling interrupts
IPI3: 0 0 Function call interrupts
IPI4: 28193 1033613 Single function call interrupts
IPI5: 0 0 CPU stop interrupts
IPI6: 0 0 IRQ work interrupts
IPI7: 0 0 completion interrupts
Err: 0
Sent from my iPhone
On Dec 9, 2015, at 00:30, johnnysl notifications@github.com wrote:
@tangsoft: can you run a cat /proc/interrupts for me?
— Reply to this email directly or view it on GitHub.
@tangsoft, hmm had a hunch that you might have had mwlwifi and mvneta split across the cores. This actually looks ok. Never mind in this case
@yuhhaurlin: what do you mean with "memory is gone" could you explain a bit more?
Free memory is gone.
So a memory leak? Why would it be gone? Your reply raises a lot of questions with me, in trying to understand why. Sorry for any dumb questions.
From our internal stress test, we did not find this kind of problem. I check issue #44, the same test is done from ethernet to wifi. I check the reporter to do stress test to see if he encounter the same problem.
@yuhhaurlin, I don't have a DLNA streaming server on Ethernet. One thing to note, a typical scenario is attaching an USB disk and offer local DLNA service over wireless. And I have a HTPC which can play movies from samba shares of my box over Ethernet without any problem.
I think what yuhhaurlin is trying to say is that there seems to be an issue with attaching usb drive to the router and then it runs out of memory as one streams over time. So if the test could be done removing that scenario for now that would be great...no point in testing everything at once as it will be very difficult to determine what the problem is...rather test 1 bug at a time in this case being the wifi.
agree, i have issues with attaching local drives and running out of memory as well. Seems to be a general issue currently with the OpenWRT image
The current minidlna in openwrt's package repository is 1.1.4, which probably has memory leak issue, I prefer compiling latest 1.1.5 yourself if you are familiar with buildroot. The minidlna project is located on SourceForge.
So far so good on my test with latest mwlwifi commit. On Dec 9, 2015 4:03 PM, "johnnysl" notifications@github.com wrote:
agree, i have issues with attaching local drives and running out of memory as well. Seems to be a general issue currently with the OpenWRT image
— Reply to this email directly or view it on GitHub https://github.com/kaloz/mwlwifi/issues/52#issuecomment-163145337.
If this is the case, I think this issue should be closed.
I will test more hours to observe wireless performance. Please be patient. Thanks. On Dec 9, 2015 4:50 PM, "yuhhaurlin" notifications@github.com wrote:
If this is the case, I think this issue should be closed.
— Reply to this email directly or view it on GitHub https://github.com/kaloz/mwlwifi/issues/52#issuecomment-163152903.
I only get these ampdu/ba failure using commit c8165c47910169d6d73983b11743fa75941b5b76 after 21h 23m 45s,
[58339.780105] ieee80211 phy0: check ba result error 1
[58339.785100] ieee80211 phy0: ampdu start error code: -22
[58339.829785] ieee80211 phy0: check ba result error 1
[58339.834779] ieee80211 phy0: ampdu start error code: -22
[58339.869723] ieee80211 phy0: check ba result error 1
[58339.874727] ieee80211 phy0: ampdu start error code: -22
and
root@OpenWrt:~# cat /sys/kernel/debug/ieee80211/phy0/mwlwifi/info
driver name: mwlwifi
chip type: 88W8864
hw version: 7
driver version: 10.3.0.15-20151208
firmware version: 0x07020806
mac address: xxxxxxxxxx
2g: enable
5g: disable
antenna: 4 4
irq number: 87
iobase0: d1400000
iobase1: d1600000
tx limit: 768
rx limit: 64
ap macid support: 0000ffff
sta macid support: 00010000
macid used: 00000001
qe trigger number: 3852270
mfg mode: false
root@OpenWrt:~# cat /sys/kernel/debug/ieee80211/phy1/mwlwifi/info
driver name: mwlwifi
chip type: 88W8864
hw version: 7
driver version: 10.3.0.15-20151208
firmware version: 0x07020806
mac address: xxxxxxxxxx
2g: disable
5g: enable
antenna: 4 4
irq number: 88
iobase0: d1800000
iobase1: d1a00000
tx limit: 768
rx limit: 64
ap macid support: 0000ffff
sta macid support: 00010000
macid used: 00000001
qe trigger number: 3852489
mfg mode: false
root@OpenWrt:~# cat /proc/interrupts
CPU0 CPU1
16: 8570926 9214531 armada_370_xp_irq 5 armada_370_xp_per_cpu_tick
18: 1952922 0 armada_370_xp_irq 31 mv64xxx_i2c
19: 35 0 armada_370_xp_irq 41 serial
25: 0 0 armada_370_xp_irq 45 ehci_hcd:usb1
26: 779165 0 armada_370_xp_irq 8 mvneta
27: 4230368 0 armada_370_xp_irq 10 mvneta
28: 0 0 armada_370_xp_irq 55 f10a0000.sata
29: 36493 0 armada_370_xp_irq 113 f10d0000.nand
69: 0 0 f1018140.gpio 0 gpio_keys
70: 0 0 f1018140.gpio 1 gpio_keys
87: 108226545 0 armada_370_xp_irq 59 mwlwifi
88: 108329062 0 armada_370_xp_irq 60 mwlwifi
89: 2 0 armada_370_xp_irq 51 f1060900.xor
90: 2 0 armada_370_xp_irq 52 f1060900.xor
91: 2 0 armada_370_xp_irq 94 f10f0900.xor
92: 2 0 armada_370_xp_irq 95 f10f0900.xor
93: 497259 0 armada_370_xp_msi_irq 0 xhci_hcd
IPI0: 0 0 CPU wakeup interrupts
IPI1: 0 0 Timer broadcast interrupts
IPI2: 295255 986073 Rescheduling interrupts
IPI3: 0 0 Function call interrupts
IPI4: 13694 1971354 Single function call interrupts
IPI5: 0 0 CPU stop interrupts
IPI6: 0 0 IRQ work interrupts
IPI7: 0 0 completion interrupts
Err: 0
BTW, I am using ffmpeg 2.8.3 and minidlna 1.1.5 compiled by myself.
I ran a stress test on 5Ghz and these are the results. The test ran for around 40 minutes.
5Ghz Mbps Input
Ram% Used
CPU%
Core0
Core1
CPU Temperature
The reason why core1 started getting higher was I added a 2nd device to the load test.
Let me know if there are questions?? Should I also do this for 2.4Ghz?
No. It looks like mwlwifi driver is all right. The memory leaking is not happened in this driver. I close this issue.
My box is wrt1900ac v1, complied with trunk version r47680.
BusyBox v1.24.1 (2015-11-30 21:05:54 CST) built-in shell (ash)
| |.-----.-----.-----.| | | |.----.| | | - || | -| || | | || || | |_____|| |___||||____||| |__| |__| W I R E L E S S F R E E D O M
DESIGNATED DRIVER (Bleeding Edge, r47680)
2 oz. Cranberry Juice
:~# dmesg [ 60.149850] br-lan: port 3(wlan0) entered forwarding state [ 60.155432] br-lan: port 3(wlan0) entered forwarding state [ 60.321688] br-lan: port 4(wlan1) entered forwarding state [ 62.161684] br-lan: port 3(wlan0) entered forwarding state [ 63.835676] pppoe-wan: renamed from ppp0 [ 4790.518054] ieee80211 phy1: failed to reallocate TX buffer [28229.515899] ieee80211 phy1: failed to reallocate TX buffer [28229.521519] ieee80211 phy0: failed to reallocate TX buffer [47131.220998] ieee80211 phy1: failed to reallocate TX buffer [47566.593219] ieee80211 phy1: failed to reallocate TX buffer [47566.598836] ieee80211 phy0: failed to reallocate TX buffer [48676.586579] ieee80211 phy1: failed to reallocate TX buffer [48676.592216] ieee80211 phy0: failed to reallocate TX buffer [78687.969573] ieee80211 phy1: failed to reallocate TX buffer [78687.975162] ieee80211 phy0: failed to reallocate TX buffer [78747.952565] ieee80211 phy1: failed to reallocate TX buffer [78803.903644] ieee80211 phy1: failed to reallocate TX buffer [78831.803680] Unable to handle kernel paging request at virtual address b528140c [78831.811021] pgd = cbc78000 [78831.813927] [b528140c] pgd=00000000 [78831.817688] Internal error: Oops: 5 [#1] SMP ARM [78831.822332] Modules linked in: pppoe ppp_async iptable_nat pppox ppp_generic nf_nat_ipv4 nf_conntrack_ipv6 nf_conntrack_ipv4 ipt_REJECT ipt_MASQUERADE armada_thermal xt_time xt_tcpudp xt_state xt_socket xt_nat xt_multiport xt_mark xt_mac xt_limit xt_id xt_conntrack xt_comment xt_TPROXY xt_TCPMSS xt_REDIRECT xt_LOG xt_CT thermal_sys slhc nf_reject_ipv4 nf_nat_masquerade_ipv4 nf_nat nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_raw iptable_mangle iptable_filter ip_tables crc_ccitt hid evdev input_core tmp421 pwm_fan hwmon i2c_mv64xxx i2c_dev mwlwifi mac80211 cfg80211 compat ledtrig_usbdev xt_set ip_set_list_set ip_set_hash_netiface ip_set_hash_netport ip_set_hash_netnet ip_set_hash_net ip_set_hash_netportnet ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_raw ip6table_mangle ip6table_filter ip6_tables x_tables tun nls_utf8 nls_cp437 regmap_spi regmap_mmio ecb usb_storage leds_tlc59116 regmap_i2c i2c_core xhci_plat_hcd xhci_pci xhci_hcd sata_mv libata ehci_orion ehci_platform ehci_hcd sd_mod scsi_mod gpio_button_hotplug ext4 jbd2 mbcache usbcore nls_base usb_common aead crypto_hash [78831.940480] CPU: 0 PID: 19253 Comm: minidlna Not tainted 3.18.23 #1 [78831.946775] task: cb99e800 ti: cb358000 task.ti: cb358000 [78831.952213] PC is at handle_mm_fault+0xa0/0x8c8 [78831.956790] LR is at 0x0 [78831.959351] pc : [] lr : [<00000000>] psr: a0000113
[78831.959351] sp : cb359e30 ip : cb358000 fp : cb359e94
[78831.970882] r10: 00000103 r9 : cb851500 r8 : 02303000
[78831.976133] r7 : cbc78000 r6 : 00000011 r5 : cb358018 r4 : b5281000
[78831.982695] r3 : f528139c r2 : cb99e800 r1 : cb851500 r0 : cfaab3c0
[78831.989260] Flags: NzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[78831.996428] Control: 10c5387d Table: 0bc7806a DAC: 00000015
[78832.002206] Process minidlna (pid: 19253, stack limit = 0xcb358238)
[78832.008512] Stack: (0xcb359e30 to 0xcb35a000)
[78832.012897] 9e20: cb359e6c cb359e40 c00d568c c00642c4
[78832.021122] 9e40: 5664413b 12444102 5664413b 12444102 c743f880 0000a82c 00000180 cbc78088
[78832.029351] 9e60: 000000a8 cfaab3c0 cfaab3c0 cb359fb0 cb358018 cfaab3c0 00000005 cb99e800
[78832.037581] 9e80: 02303760 cb851500 cb359edc cb359e98 c00203a0 c00a6480 cb359f14 cb359ea8
[78832.045800] 9ea0: c00f08ec 00000000 cfaab3f8 000000a8 cb359ee4 00000005 00000005 c00205c8
[78832.054024] 9ec0: 02303760 cb359fb0 02303760 00033d7a cb359efc cb359ee0 c00205ec c0020288
[78832.062254] 9ee0: c046acac 00000005 c00205c8 02303760 cb359fac cb359f00 c000851c c00205d4
[78832.070484] 9f00: c4c80100 cac8c360 cb359f6c cb359f18 c00bd6b8 c00f0640 00000000 00000000
[78832.078713] 9f20: c4c80108 00000000 00000002 00000000 0000a82c 00000000 00000000 00000000
[78832.086932] 9f40: cb359f5c cb358000 bef05a18 00000000 00000000 7fffffff cb358000 00000000
[78832.095157] 9f60: cb359fa4 cb359f70 c00be17c c00bd424 ffffffff 000007ff 0000a82c 00000000
[78832.103386] 9f80: bef059e8 00000000 0000000d 000177e4 80000010 ffffffff 10c5387d 10c5387d
[78832.111616] 9fa0: 00000000 cb359fb0 c00099c0 c00084e8 7ffffffe 0000000d bef05a18 7fffffff
[78832.119845] 9fc0: 00000000 0000000d 01700000 00000000 00000001 02303760 00033d7a 00000001
[78832.128075] 9fe0: 00040f7c bef05a08 000177ec 000177e4 80000010 ffffffff 6fe35b7e a6abdef5
[78832.136282] Backtrace:
[78832.138786] from
[78832.146906] r10:cb851500 r9:02303760 r8:cb99e800 r7:00000005 r6:cfaab3c0 r5:cb358018
[78832.154851] r4:cb359fb0
[78832.157435] from
[78832.165818] r10:00033d7a r9:02303760 r8:cb359fb0 r7:02303760 r6:c00205c8 r5:00000005
[78832.173764] r4:00000005
[78832.176338] from
[78832.184639] r7:02303760 r6:c00205c8 r5:00000005 r4:c046acac
[78832.190406] from
[78832.197840] Exception stack(0xcb359fb0 to 0xcb359ff8)
[78832.202921] 9fa0: 7ffffffe 0000000d bef05a18 7fffffff
[78832.211145] 9fc0: 00000000 0000000d 01700000 00000000 00000001 02303760 00033d7a 00000001
[78832.219372] 9fe0: 00040f7c bef05a08 000177ec 000177e4 80000010 ffffffff
[78832.226015] r8:10c5387d r7:10c5387d r6:ffffffff r5:80000010 r4:000177e4
[78832.232835] Code: e7e8a658 e3c34eff e3c4400f e2444440 (e794510a)
[78832.239480] ---[ end trace eb0cf273a58a11b8 ]---
[78832.246271] Unable to handle kernel paging request at virtual address d062a01c
[78832.253536] pgd = cbc78000
[78832.256262] [d062a01c] pgd=00000000
[78832.259882] Internal error: Oops: 5 [#2] SMP ARM
[78832.264517] Modules linked in: pppoe ppp_async iptable_nat pppox ppp_generic nf_nat_ipv4 nf_conntrack_ipv6 nf_conntrack_ipv4 ipt_REJECT ipt_MASQUERADE armada_thermal xt_time xt_tcpudp xt_state xt_socket xt_nat xt_multiport xt_mark xt_mac xt_limit xt_id xt_conntrack xt_comment xt_TPROXY xt_TCPMSS xt_REDIRECT xt_LOG xt_CT thermal_sys slhc nf_reject_ipv4 nf_nat_masquerade_ipv4 nf_nat nf_log_ipv4 nf_defrag_ipv6 nf_defrag_ipv4 nf_conntrack_rtcache nf_conntrack iptable_raw iptable_mangle iptable_filter ip_tables crc_ccitt hid evdev input_core tmp421 pwm_fan hwmon i2c_mv64xxx i2c_dev mwlwifi mac80211 cfg80211 compat ledtrig_usbdev xt_set ip_set_list_set ip_set_hash_netiface ip_set_hash_netport ip_set_hash_netnet ip_set_hash_net ip_set_hash_netportnet ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common ip6table_raw ip6table_mangle ip6table_filter ip6_tables x_tables tun nls_utf8 nls_cp437 regmap_spi regmap_mmio ecb usb_storage leds_tlc59116 regmap_i2c i2c_core xhci_plat_hcd xhci_pci xhci_hcd sata_mv libata ehci_orion ehci_platform ehci_hcd sd_mod scsi_mod gpio_button_hotplug ext4 jbd2 mbcache usbcore nls_base usb_common aead crypto_hash
[78832.382655] CPU: 0 PID: 19253 Comm: minidlna Tainted: G D 3.18.23 #1
[78832.390092] task: cb99e800 ti: cb358000 task.ti: cb358000
[78832.395535] PC is at _raw_spin_lock+0x10/0x4c
[78832.399933] LR is at unmap_single_vma+0x1d0/0x61c
[78832.404665] pc : [] lr : [] psr: 40000113
[78832.404665] sp : cb359af8 ip : cb359b08 fp : cb359b04
[78832.416191] r10: 02200000 r9 : cb359bc0 r8 : 02001000
[78832.421451] r7 : 01fff000 r6 : cc4e97fc r5 : cfe8ed40 r4 : 01880000
[78832.428017] r3 : 01880000 r2 : d062a000 r1 : 00000000 r0 : d062a01c
[78832.434583] Flags: nZcv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user
[78832.441761] Control: 10c5387d Table: 0bc7806a DAC: 00000015
[78832.447545] Process minidlna (pid: 19253, stack limit = 0xcb358238)
[78832.453840] Stack: (0xcb359af8 to 0xcb35a000)
[78832.458229] 9ae0: cb359b84 cb359b08
[78832.466448] 9b00: c00a43b4 c0012484 028c2fff cbc78080 00000000 cb358008 cfaab51c cfaab510
[78832.474673] 9b20: cfaab3c0 c04688c4 c04a4a88 d062a000 028c3000 00000000 cbc78084 04aea7dd
[78832.482903] 9b40: c0465018 cb851500 c048d00c 00000000 00000000 00000000 00000000 cb851500
[78832.491132] 9b60: ffffffff 00000000 cb359bc0 00000000 c03bb8a4 cb99e800 cb359bb4 cb359b88
[78832.499362] 9b80: c00a5840 c00a41f0 00000000 cfaab3f8 0000000b cb8514a8 cfaab3c0 cb359be8
[78832.507592] 9ba0: cfaab3f8 0000000b cb359c24 cb359bb8 c00ab700 c00a581c 00000000 00000000
[78832.515810] 9bc0: cfaab3c0 00000001 00000000 00000000 ffffffff c048d01c cb359c04 00000193
[78832.524036] 9be0: 00000400 cbea7000 cfa3c098 cfa3c098 0000000b c046a81c cb359c24 cb359c08
[78832.532266] 9c00: c00665dc c00125c0 cfaab3c0 cb99e800 cb99eb60 cfaab3f8 cb359c3c cb359c28
[78832.540496] 9c20: c0024e3c c00ab638 cfaab3c0 cb99e800 cb359c6c cb359c40 c0028654 c0024e08
[78832.548725] 9c40: 00000000 cb358000 c03bb8a4 cb99e800 cb359c6c cb359c60 c00278c8 c046a81c
[78832.556944] 9c60: cb359cb4 cb359c70 c001bf48 c0028358 cb358238 0000000b cb359cb4 cb358000
[78832.565169] 9c80: 60000113 00000005 c03f070c b528140c cb359de8 00000005 cfaab3c0 cb99e800
[78832.573399] 9ca0: b528140c 00010000 cb359ccc cb359cb8 c002026c c001bc84 cb359de8 00000005
[78832.581630] 9cc0: cb359d14 cb359cd0 c0020504 c002020c 00000b50 c04608f8 00000000 00000000
[78832.589860] 9ce0: cfaab3f8 00000028 cb366c4c 00000005 00000005 c00205c8 b528140c cb359de8
[78832.598090] 9d00: cb851500 00000103 cb359d34 cb359d18 c00205ec c0020288 c046acac 00000005
[78832.606309] 9d20: c00205c8 b528140c cb359de4 cb359d38 c000851c c00205d4 00000000 c743f880
[78832.614534] 9d40: c03d5c38 c00e3c90 cb359d74 cb359d58 c026d254 c02e5840 00020000 00000001
[78832.622763] 9d60: cb359de4 c026d264 cb359d8c cb359d78 c026d298 c026d234 00020000 c00e3d0c
[78832.630993] 9d80: cb359db4 cb359d90 c00e3cfc c026d270 cb359d98 00020000 0000a82c 00000000
[78832.639221] 9da0: cbafeaf0 cb359de8 cb359de4 cb359db8 c00e40b4 c00e3c9c 00000000 c743f880
[78832.647452] 9dc0: c4c80100 c00a6514 a0000113 ffffffff cb359e1c 02303000 cb359e94 cb359de8
[78832.655671] 9de0: c00096f8 c00084e8 cfaab3c0 cb851500 cb99e800 f528139c b5281000 cb358018
[78832.663897] 9e00: 00000011 cbc78000 02303000 cb851500 00000103 cb359e94 cb358000 cb359e30
[78832.672127] 9e20: 00000000 c00a6514 a0000113 ffffffff cb359e6c cb359e40 c00d568c c00642c4
[78832.680357] 9e40: 5664413b 12444102 5664413b 12444102 c743f880 0000a82c 00000180 cbc78088
[78832.688587] 9e60: 000000a8 cfaab3c0 cfaab3c0 cb359fb0 cb358018 cfaab3c0 00000005 cb99e800
[78832.696806] 9e80: 02303760 cb851500 cb359edc cb359e98 c00203a0 c00a6480 cb359f14 cb359ea8
[78832.705032] 9ea0: c00f08ec 00000000 cfaab3f8 000000a8 cb359ee4 00000005 00000005 c00205c8
[78832.713262] 9ec0: 02303760 cb359fb0 02303760 00033d7a cb359efc cb359ee0 c00205ec c0020288
[78832.721491] 9ee0: c046acac 00000005 c00205c8 02303760 cb359fac cb359f00 c000851c c00205d4
[78832.729721] 9f00: c4c80100 cac8c360 cb359f6c cb359f18 c00bd6b8 c00f0640 00000000 00000000
[78832.737950] 9f20: c4c80108 00000000 00000002 00000000 0000a82c 00000000 00000000 00000000
[78832.746169] 9f40: cb359f5c cb358000 bef05a18 00000000 00000000 7fffffff cb358000 00000000
[78832.754384] 9f60: cb359fa4 cb359f70 c00be17c c00bd424 ffffffff 000007ff 0000a82c 00000000
[78832.762598] 9f80: bef059e8 00000000 0000000d 000177e4 80000010 ffffffff 10c5387d 10c5387d
[78832.770812] 9fa0: 00000000 cb359fb0 c00099c0 c00084e8 7ffffffe 0000000d bef05a18 7fffffff
[78832.779036] 9fc0: 00000000 0000000d 01700000 00000000 00000001 02303760 00033d7a 00000001
[78832.787268] 9fe0: 00040f7c bef05a08 000177ec 000177e4 80000010 ffffffff 6fe35b7e a6abdef5
[78832.795482] Backtrace:
[78832.797986] from
[78832.806291] from
[78832.814069] r10:cb99e800 r9:c03bb8a4 r8:00000000 r7:cb359bc0 r6:00000000 r5:ffffffff
[78832.822016] r4:cb851500
[78832.824591] from
[78832.831847] r8:0000000b r7:cfaab3f8 r6:cb359be8 r5:cfaab3c0 r4:cb8514a8
[78832.838679] from
[78832.845492] r7:cfaab3f8 r6:cb99eb60 r5:cb99e800 r4:cfaab3c0
[78832.851257] from
[78832.857991] r5:cb99e800 r4:cfaab3c0
[78832.861628] from
[78832.868189] r7:c046a81c
[78832.870768] from
[78832.878634] r10:00010000 r9:b528140c r8:cb99e800 r7:cfaab3c0 r6:00000005 r5:cb359de8
[78832.886568] r4:b528140c
[78832.889154] from
[78832.898065] r7:00000005 r4:cb359de8
[78832.901703] from
[78832.910093] r10:00000103 r9:cb851500 r8:cb359de8 r7:b528140c r6:c00205c8 r5:00000005
[78832.918037] r4:00000005
[78832.920627] from
[78832.928929] r7:b528140c r6:c00205c8 r5:00000005 r4:c046acac
[78832.934683] from
[78832.942115] Exception stack(0xcb359de8 to 0xcb359e30)
[78832.947199] 9de0: cfaab3c0 cb851500 cb99e800 f528139c b5281000 cb358018
[78832.955423] 9e00: 00000011 cbc78000 02303000 cb851500 00000103 cb359e94 cb358000 cb359e30
[78832.963647] 9e20: 00000000 c00a6514 a0000113 ffffffff
[78832.968731] r8:02303000 r7:cb359e1c r6:ffffffff r5:a0000113 r4:c00a6514
[78832.975548] from
[78832.983676] r10:cb851500 r9:02303760 r8:cb99e800 r7:00000005 r6:cfaab3c0 r5:cb358018
[78832.991634] r4:cb359fb0
[78832.994208] from
[78833.002597] r10:00033d7a r9:02303760 r8:cb359fb0 r7:02303760 r6:c00205c8 r5:00000005
[78833.010543] r4:00000005
[78833.013115] from
[78833.021416] r7:02303760 r6:c00205c8 r5:00000005 r4:c046acac
[78833.027169] from
[78833.034602] Exception stack(0xcb359fb0 to 0xcb359ff8)
[78833.039693] 9fa0: 7ffffffe 0000000d bef05a18 7fffffff
[78833.047921] 9fc0: 00000000 0000000d 01700000 00000000 00000001 02303760 00033d7a 00000001
[78833.056138] 9fe0: 00040f7c bef05a08 000177ec 000177e4 80000010 ffffffff
[78833.062784] r8:10c5387d r7:10c5387d r6:ffffffff r5:80000010 r4:000177e4
[78833.069607] Code: e1a0c00d e92dd800 e24cb004 f590f000 (e1902f9f)
[78833.075728] ---[ end trace eb0cf273a58a11b9 ]---
[78833.080371] Fixing recursive fault but reboot is needed!