Receiving spinlocks KP randomly

[stevefan1999-personal]

Just a few minutes ago I saw this from my byobu remote session:

e>/-1, .owner_cpu: 152518656                                               [0/0]                                                                               
Message from syslogd@nas at Nov  4 16:25:26 ...
 kernel:[66559.853169] BUG: spinlock lockup suspected on CPU#3, swapper/3/0    

Message from syslogd@nas at Nov  4 16:25:26 ...
 kernel:[66559.857273]  lock: 0xffffffc0f5e27e10, .magic: 00000040, .owner: <non
e>/-1, .owner_cpu: 152518656                                                   
Message from syslogd@nas at Nov  4 16:25:26 ...
 kernel:[66552.808519] BUG: spinlock bad magic on CPU#3, swapper/3/0           

Message from syslogd@nas at Nov  4 16:25:26 ...
 kernel:[66552.809119]  lock: 0xffffffc0f5e27e10, .magic: 00000040, .owner: <non
e>/-1, .owner_cpu: 152518656                                                   

Message from syslogd@nas at Nov  4 16:25:26 ...
 kernel:[66559.853169] BUG: spinlock lockup suspected on CPU#3, swapper/3/0    

Message from syslogd@nas at Nov  4 16:25:26 ...
 kernel:[66559.857273]  lock: 0xffffffc0f5e27e10, .magic: 00000040, .owner: <non
e>/-1, .owner_cpu: 152518656                                                   

And then my R64 just reached a complete lockdown. Have no idea what happened, what caused this.

[GuyNoIRQ]

I've been hitting similar on a RockPro64 running Open Media Vault 0.8.3-1141. Every time I try to initialize a raid1 volume I hit this after just a couple minutes.

RockPro_Raid_Init_kernlog_Crash.log

                _                     __   _  _   
 _ __ ___   ___| | ___ __  _ __ ___  / /_ | || |  
| '__/ _ \ / __| |/ / '_ \| '__/ _ \| '_ \| || |_ 
| | | (_) | (__|   <| |_) | | | (_) | (_) |__   _|
|_|  \___/ \___|_|\_\ .__/|_|  \___/ \___/   |_|  
                    |_|                           
Linux rockpro64 4.4.167-1213-rockchip-ayufan-g34ae07687fce #1 SMP Tue Jun 18 20:44:49 UTC 2019 aarch64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
root@rockpro64:~# mdadm --readwrite /dev/md0
root@rockpro64:~# tail -f /var/log/kern.log 
Jul  6 23:17:08 rockpro64 kernel: [   18.870817] cdn-dp fec00000.dp: Direct firmware load for rockchip/dptx.bin failed with error -2
Jul  6 23:17:14 rockpro64 kernel: [   25.226971] rk_gmac-dwmac fe300000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
Jul  6 23:17:18 rockpro64 kernel: [   29.324141] NFSD: starting 90-second grace period (net ffffff80093a2f40)
Jul  6 23:17:24 rockpro64 kernel: [   34.886722] cdn-dp fec00000.dp: Direct firmware load for rockchip/dptx.bin failed with error -2
Jul  7 00:24:13 rockpro64 kernel: [   62.621049] md: md0 switched to read-write mode.
Jul  7 00:24:13 rockpro64 kernel: [   62.639974] md: resync of RAID array md0
Jul  7 00:24:13 rockpro64 kernel: [   62.640460] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
Jul  7 00:24:13 rockpro64 kernel: [   62.641129] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
Jul  7 00:24:13 rockpro64 kernel: [   62.642201] md: using 128k window, over a total of 7813895488k.
Jul  7 00:24:17 rockpro64 kernel: [   66.891480] cdn-dp fec00000.dp: Direct firmware load for rockchip/dptx.bin failed with error -2
Jul  7 00:25:21 rockpro64 kernel: [  130.896639] cdn-dp fec00000.dp: [drm:cdn_dp_request_firmware] *ERROR* Timed out trying to load firmware

Message from syslogd@rockpro64 at Jul  7 00:27:08 ...
 kernel:[  233.001921] Internal error: : 96000210 [#1] SMP
Jul  7 00:27:08 rockpro64 kernel: [  233.001042] Unhandled fault: synchronous external abort (0x96000210) at 0xffffff80096c61b0
Jul  7 00:27:08 rockpro64 kernel: [  233.001921] Internal error: : 96000210 [#1] SMP
Jul  7 00:27:08 rockpro64 kernel: [  233.002397] Modules linked in: midgard_kbase dw_hdmi_i2s_audio rockchip_saradc ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid0 multipath linear raid1 md_mod
Jul  7 00:27:08 rockpro64 kernel: [  233.004571] CPU: 0 PID: 143 Comm: scsi_eh_1 Not tainted 4.4.167-1213-rockchip-ayufan-g34ae07687fce #1
Jul  7 00:27:08 rockpro64 kernel: [  233.005524] Hardware name: Pine64 RockPro64 (DT)
Jul  7 00:27:08 rockpro64 kernel: [  233.006007] task: ffffffc0efcb8000 task.stack: ffffffc0efcb4000
Jul  7 00:27:08 rockpro64 kernel: [  233.006631] PC is at ahci_scr_read+0x70/0x98
Jul  7 00:27:08 rockpro64 kernel: [  233.007081] LR is at sata_scr_read+0x88/0x94
Jul  7 00:27:08 rockpro64 kernel: [  233.007528] pc : [<ffffff800877cb8c>] lr : [<ffffff80087634a4>] pstate: 80000145
Jul  7 00:27:08 rockpro64 kernel: [  233.008292] sp : ffffffc0efcb7a50
Jul  7 00:27:08 rockpro64 kernel: [  233.008640] x29: ffffffc0efcb7a50 x28: 0000000000000000 
Jul  7 00:27:08 rockpro64 kernel: [  233.009211] x27: ffffffc0efc49e80 x26: 0000000000000001 
Jul  7 00:27:08 rockpro64 kernel: [  233.009781] x25: ffffffc0efc4a258 x24: ffffffc0efc48000 
Jul  7 00:27:08 rockpro64 kernel: [  233.010352] x23: 0000000000000000 x22: ffffffc0efcb7b28 
Jul  7 00:27:08 rockpro64 kernel: [  233.010923] x21: ffffffc0efc49e80 x20: ffffffc0efcb7b28 
Jul  7 00:27:08 rockpro64 kernel: [  233.011493] x19: 0000000000000001 x18: 0000000030d00800 
Jul  7 00:27:08 rockpro64 kernel: [  233.012063] x17: 0000000000000000 x16: 0000000000000000 
Jul  7 00:27:08 rockpro64 kernel: [  233.012633] x15: 0000000000000000 x14: 0000000000000000 
Jul  7 00:27:08 rockpro64 kernel: [  233.013203] x13: 0000000000000000 x12: 0000000000000000 
Jul  7 00:27:08 rockpro64 kernel: [  233.013772] x11: 0000000000000000 x10: 0000000000000000 
Jul  7 00:27:08 rockpro64 kernel: [  233.014343] x9 : 0000000000000000 x8 : ffffffc0efc4a258 
Jul  7 00:27:08 rockpro64 kernel: [  233.014914] x7 : 0000000000000000 x6 : ffffffc0efc4a2d0 
Jul  7 00:27:08 rockpro64 kernel: [  233.018973] x5 : 0000000000000000 x4 : ffffff8008d9af94 
Jul  7 00:27:08 rockpro64 kernel: [  233.022971] x3 : 00000000000001b0 x2 : ffffffc0efcb7b28 
Jul  7 00:27:08 rockpro64 kernel: [  233.026914] x1 : 0000000000000030 x0 : ffffff80096c61b0 
Jul  7 00:27:08 rockpro64 kernel: [  233.030806] 
Jul  7 00:27:08 rockpro64 kernel: [  233.030806] PC: 0xffffff800877cb0c:
Jul  7 00:27:08 rockpro64 kernel: [  233.037877] cb0c  52800013 912c1821 97fe23cd 17ffffe2 a9bd7bfd 910003fd a90153f3 f90013f5
Jul  7 00:27:08 rockpro64 kernel: [  233.042148] cb2c  aa0003f5 2a0103f3 aa1e03e0 aa0203f4 d503201f f94002a3 7100127f f959dc60
Jul  7 00:27:08 rockpro64 kernel: [  233.046485] cb4c  b9402c63 f9401801 53196063 91040063 f9400820 54000268 54000061 b9401821
Jul  7 00:27:08 rockpro64 kernel: [  233.050838] cb6c  36e80201 d00030e4 913de084 91007084 b8735881 34000161 8b21c063 8b030000
Jul  7 00:27:08 rockpro64 kernel: [  233.055199] cb8c  b9400000 d5033d9f b9000280 52800000 a94153f3 f94013f5 a8c37bfd d65f03c0
Jul  7 00:27:08 rockpro64 kernel: [  233.059567] cbac  128002a0 17fffffb a9be7bfd 910003fd f9000bf3 aa0003f3 aa1e03e0 d503201f
Jul  7 00:27:08 rockpro64 kernel: [  233.063939] cbcc  f9400260 f9400000 f959dc00 f9401800 b9400000 36400120 52801fe0 b9038660
Jul  7 00:27:08 rockpro64 kernel: [  233.068322] cbec  f0004482 90004061 912c8842 911dbc21 aa1303e0 97ff9d72 f9400bf3 a8c27bfd
Jul  7 00:27:08 rockpro64 kernel: [  233.072786] 
Jul  7 00:27:08 rockpro64 kernel: [  233.072786] LR: 0xffffff8008763424:
Jul  7 00:27:08 rockpro64 kernel: [  233.080343] 3424  a90153f3 a9025bf5 aa0003f3 2a0103f5 aa1e03e0 aa0203f6 d503201f f9400274
Jul  7 00:27:08 rockpro64 kernel: [  233.084836] 3444  d283d000 8b000280 eb00027f 54000180 f959c280 eb00027f 54000120 aa1603e2
Jul  7 00:27:08 rockpro64 kernel: [  233.089268] 3464  2a1503e1 aa1303e0 9400572a a94153f3 a9425bf5 a8c37bfd d65f03c0 aa1303e0
Jul  7 00:27:08 rockpro64 kernel: [  233.093647] 3484  97fffb19 34000100 f9400680 aa1603e2 2a1503e1 f9406c03 aa1303e0 d63f0060
Jul  7 00:27:08 rockpro64 kernel: [  233.098065] 34a4  17fffff3 12800bc0 17fffff1 a9ba7bfd 910003fd a90153f3 a9025bf5 a90363f7
Jul  7 00:27:08 rockpro64 kernel: [  233.102501] 34c4  a9046bf9 aa0103f5 aa0203f6 aa0003f7 aa1e03e0 d503201f d0005813 b94012a0
Jul  7 00:27:08 rockpro64 kernel: [  233.106912] 34e4  f940e274 a94066b8 97e6e656 8b000280 cb160001 910173a2 f100003f 52800001
Jul  7 00:27:08 rockpro64 kernel: [  233.111301] 3504  9a80a2d6 aa1703e0 97ffffc4 2a0003f5 350003c0 b9405fb4 f940e27a 12000e94
Jul  7 00:27:08 rockpro64 kernel: [  233.115658] 
Jul  7 00:27:08 rockpro64 kernel: [  233.115658] SP: 0xffffffc0efcb79d0:
Jul  7 00:27:08 rockpro64 kernel: [  233.122908] 79d0  efcb7b28 ffffffc0 00000000 00000000 efc48000 ffffffc0 efc4a258 ffffffc0
Jul  7 00:27:08 rockpro64 kernel: [  233.127166] 79f0  00000001 00000000 efc49e80 ffffffc0 00000000 00000000 efcb7a50 ffffffc0
Jul  7 00:27:08 rockpro64 kernel: [  233.131340] 7a10  087634a4 ffffff80 efcb7a50 ffffffc0 0877cb8c ffffff80 80000145 00000000
Jul  7 00:27:08 rockpro64 kernel: [  233.135428] 7a30  08cbe644 ffffff80 f2bb60a0 ffffffc0 ffffffff ffffffff 00000000 00000000
Jul  7 00:27:08 rockpro64 kernel: [  233.139426] 7a50  efcb7a80 ffffffc0 087634a4 ffffff80 efc49e80 ffffffc0 efc48000 ffffffc0
Jul  7 00:27:08 rockpro64 kernel: [  233.143368] 7a70  00000001 00000000 00000000 00000000 efcb7ab0 ffffffc0 08770374 ffffff80
Jul  7 00:27:08 rockpro64 kernel: [  233.147307] 7a90  efc4a258 ffffffc0 efc49e80 ffffffc0 efc4b858 ffffffc0 00003008 00000000
Jul  7 00:27:08 rockpro64 kernel: [  233.151252] 7ab0  efcb7b60 ffffffc0 08770b34 ffffff80 efc48000 ffffffc0 efc49e80 ffffffc0
Jul  7 00:27:08 rockpro64 kernel: [  233.155230] 
Jul  7 00:27:08 rockpro64 kernel: [  233.155230] X0: 0xffffff80096c6130:
Jul  7 00:27:08 rockpro64 kernel: [  233.161788] 6130 
Jul  7 00:27:08 rockpro64 kernel: [  233.211980] Unhandled fault: synchronous external abort (0x96000210) at 0xffffff80096c6130
Jul  7 00:27:08 rockpro64 kernel: [  237.987191] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

Message from syslogd@rockpro64 at Jul  7 00:27:47 ...
 kernel:[  240.218948] BUG: spinlock lockup suspected on CPU#0, scsi_eh_1/143

Message from syslogd@rockpro64 at Jul  7 00:27:47 ...
 kernel:[  240.222849]  lock: 0xffffff800926fdf0, .magic: dead4ead, .owner: scsi_eh_1/143, .owner_cpu: 0
Jul  7 00:27:47 rockpro64 kernel: [  240.218948] BUG: spinlock lockup suspected on CPU#0, scsi_eh_1/143
Jul  7 00:27:47 rockpro64 kernel: [  240.222849]  lock: 0xffffff800926fdf0, .magic: dead4ead, .owner: scsi_eh_1/143, .owner_cpu: 0
Jul  7 00:27:47 rockpro64 kernel: [  240.227026] CPU: 0 PID: 143 Comm: scsi_eh_1 Not tainted 4.4.167-1213-rockchip-ayufan-g34ae07687fce #1
Jul  7 00:27:47 rockpro64 kernel: [  240.231336] Hardware name: Pine64 RockPro64 (DT)
Jul  7 00:27:47 rockpro64 kernel: [  240.235282] Call trace:
Jul  7 00:27:47 rockpro64 kernel: [  240.239056] [<ffffff80080889b0>] dump_backtrace+0x0/0x220
Jul  7 00:27:47 rockpro64 kernel: [  240.243183] [<ffffff8008088bf4>] show_stack+0x24/0x30
Jul  7 00:27:47 rockpro64 kernel: [  240.247311] [<ffffff80085a2b2c>] dump_stack+0x98/0xc0
Jul  7 00:27:47 rockpro64 kernel: [  240.251429] [<ffffff8008104a24>] spin_dump+0x84/0xa4
Jul  7 00:27:47 rockpro64 kernel: [  240.255566] [<ffffff8008104bc0>] do_raw_spin_lock+0xdc/0x164
Jul  7 00:27:47 rockpro64 kernel: [  240.259802] [<ffffff8008c9862c>] _raw_spin_lock_irq+0x24/0x30
Jul  7 00:27:47 rockpro64 kernel: [  240.264071] [<ffffff8008088c38>] die+0x38/0x1ac
Jul  7 00:27:47 rockpro64 kernel: [  240.268241] [<ffffff8008088ea0>] arm64_notify_die+0x6c/0x70
Jul  7 00:27:47 rockpro64 kernel: [  240.272502] [<ffffff8008080b50>] do_mem_abort+0x9c/0xac
Jul  7 00:27:47 rockpro64 kernel: [  240.276743] Exception stack(0xffffffc0efcb7560 to 0xffffffc0efcb7690)
Jul  7 00:27:47 rockpro64 kernel: [  240.281086] 7560: ffffffc0efcb8000 0000008000000000 ffffffc0efcb7730 ffffff80085a0dc4
Jul  7 00:27:47 rockpro64 kernel: [  240.285566] 7580: 0000000000000007 0000000000000000 ffffff80096c6130 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  240.290013] 75a0: ffffff800941f2fc 0000000000000000 ffffffc0efcb75d0 ffffff800810b7f8
Jul  7 00:27:47 rockpro64 kernel: [  240.294437] 75c0: 0000000000000005 0000000000000000 ffffffc0efcb7670 ffffff800810b958
Jul  7 00:27:47 rockpro64 kernel: [  240.298807] 75e0: ffffffc0efcb76f0 ffffff8008f8816c ffffff8008f8816c ffffff8008fed582
Jul  7 00:27:47 rockpro64 kernel: [  240.303163] 7600: ffffffc0efcb77bc ffffff80096c6130 0000000000000004 0000000000000010
Jul  7 00:27:47 rockpro64 kernel: [  240.307501] 7620: 0000000000000000 ffffffc0efcb77c0 ffffffc0efcb77bc 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  240.311810] 7640: 0000000000001000 0000000000aaaaaa 0000000000000008 00000000ffffffff
Jul  7 00:27:47 rockpro64 kernel: [  240.316081] 7660: ffffffc0f29dbf10 ffffff800ba002a0 ffffff800ba002a0 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  240.320398] 7680: 0000000000000000 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  240.324397] [<ffffff80080825cc>] el1_da+0x24/0x84
Jul  7 00:27:47 rockpro64 kernel: [  240.328343] [<ffffff8008085024>] show_data.constprop.3+0xa4/0xf8
Jul  7 00:27:47 rockpro64 kernel: [  240.332391] [<ffffff80080853b0>] __show_regs+0x154/0x1a8
Jul  7 00:27:47 rockpro64 kernel: [  240.336332] [<ffffff8008088cb4>] die+0xb4/0x1ac
Jul  7 00:27:47 rockpro64 kernel: [  240.340152] [<ffffff8008088ea0>] arm64_notify_die+0x6c/0x70
Jul  7 00:27:47 rockpro64 kernel: [  240.344030] [<ffffff8008080b50>] do_mem_abort+0x9c/0xac
Jul  7 00:27:47 rockpro64 kernel: [  240.347831] Exception stack(0xffffffc0efcb7880 to 0xffffffc0efcb79b0)
Jul  7 00:27:47 rockpro64 kernel: [  240.351752] 7880: 0000000000000001 0000008000000000 ffffffc0efcb7a50 ffffff800877cb8c
Jul  7 00:27:47 rockpro64 kernel: [  240.355798] 78a0: 0000000000000007 0000000000000000 ffffff80096c61b0 ffffffc0efc4a258
Jul  7 00:27:47 rockpro64 kernel: [  240.359850] 78c0: 0000000000000001 ffffffc0efc49e80 ffffffc0efcb8000 ffffffc0efcb7a50
Jul  7 00:27:47 rockpro64 kernel: [  240.363846] 78e0: ffffff80087634a4 ffffffc0efcb7920 ffffff80080825b8 0000000060000145
Jul  7 00:27:47 rockpro64 kernel: [  240.367838] 7900: 0000000000000000 ffffffc0f298eb80 0000008000000000 ffffff80080e92b8
Jul  7 00:27:47 rockpro64 kernel: [  240.371828] 7920: ffffff80096c61b0 0000000000000030 ffffffc0efcb7b28 00000000000001b0
Jul  7 00:27:47 rockpro64 kernel: [  240.375860] 7940: ffffff8008d9af94 0000000000000000 ffffffc0efc4a2d0 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  240.379858] 7960: ffffffc0efc4a258 0000000000000000 0000000000000000 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  240.383830] 7980: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  240.387754] 79a0: 0000000000000000 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  240.391371] [<ffffff80080825cc>] el1_da+0x24/0x84
Jul  7 00:27:47 rockpro64 kernel: [  240.394929] [<ffffff80087634a4>] sata_scr_read+0x88/0x94
Jul  7 00:27:47 rockpro64 kernel: [  240.398563] [<ffffff8008770374>] ata_eh_link_autopsy+0x4c/0x740
Jul  7 00:27:47 rockpro64 kernel: [  240.402271] [<ffffff8008770b34>] ata_eh_autopsy+0xcc/0xdc
Jul  7 00:27:47 rockpro64 kernel: [  240.405862] [<ffffff8008778914>] sata_pmp_error_handler+0x34/0x7b8
Jul  7 00:27:47 rockpro64 kernel: [  240.409476] [<ffffff800877c4f8>] ahci_error_handler+0x44/0x70
Jul  7 00:27:47 rockpro64 kernel: [  240.413018] [<ffffff8008772f54>] ata_scsi_port_error_handler+0xe0/0x550
Jul  7 00:27:47 rockpro64 kernel: [  240.416597] [<ffffff8008773448>] ata_scsi_error+0x84/0xa4
Jul  7 00:27:47 rockpro64 kernel: [  240.419986] [<ffffff8008744610>] scsi_error_handler+0x130/0x6d4
Jul  7 00:27:47 rockpro64 kernel: [  240.423359] [<ffffff80080d54ac>] kthread+0xe0/0xf0
Jul  7 00:27:47 rockpro64 kernel: [  240.426620] [<ffffff8008082ef0>] ret_from_fork+0x10/0x20
Jul  7 00:27:47 rockpro64 kernel: [  276.141323] Watchdog detected hard LOCKUP on cpu 0
Jul  7 00:27:47 rockpro64 kernel: [  276.141898] ------------[ cut here ]------------
Jul  7 00:27:47 rockpro64 kernel: [  276.151042] WARNING: at kernel/watchdog.c:352
Jul  7 00:27:47 rockpro64 kernel: [  276.155862] Modules linked in: midgard_kbase dw_hdmi_i2s_audio rockchip_saradc ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid0 multipath linear raid1 md_mod
Jul  7 00:27:47 rockpro64 kernel: [  276.167535] 
Jul  7 00:27:47 rockpro64 kernel: [  276.172480] CPU: 5 PID: 0 Comm: swapper/5 Not tainted 4.4.167-1213-rockchip-ayufan-g34ae07687fce #1
Jul  7 00:27:47 rockpro64 kernel: [  276.178353] Hardware name: Pine64 RockPro64 (DT)
Jul  7 00:27:47 rockpro64 kernel: [  276.183844] task: ffffffc0f2af4600 task.stack: ffffffc0f2b0c000
Jul  7 00:27:47 rockpro64 kernel: [  276.189486] PC is at watchdog_timer_fn+0x190/0x2e8
Jul  7 00:27:47 rockpro64 kernel: [  276.194978] LR is at watchdog_timer_fn+0x190/0x2e8
Jul  7 00:27:47 rockpro64 kernel: [  276.200389] pc : [<ffffff800815631c>] lr : [<ffffff800815631c>] pstate: 600001c5
Jul  7 00:27:47 rockpro64 kernel: [  276.206136] sp : ffffffc0f7f43d00
Jul  7 00:27:47 rockpro64 kernel: [  276.211514] x29: ffffffc0f7f43d00 x28: 0000000000000000 
Jul  7 00:27:47 rockpro64 kernel: [  276.217097] x27: ffffff80092678a0 x26: ffffff800922b238 
Jul  7 00:27:47 rockpro64 kernel: [  276.222643] x25: ffffffc0f7f46250 x24: ffffffc0f2b0fda0 
Jul  7 00:27:47 rockpro64 kernel: [  276.228150] x23: 0000000000000000 x22: 00000000000000fd 
Jul  7 00:27:47 rockpro64 kernel: [  276.233638] x21: ffffff8009266000 x20: ffffff800922b2b0 
Jul  7 00:27:47 rockpro64 kernel: [  276.239072] x19: ffffff800922b000 x18: 0000000000000005 
Jul  7 00:27:47 rockpro64 kernel: [  276.244427] x17: 0000000000000000 x16: 0000000000000000 
Jul  7 00:27:47 rockpro64 kernel: [  276.249745] x15: 0000000000000000 x14: ffffff800ba006c0 
Jul  7 00:27:47 rockpro64 kernel: [  276.255025] x13: ffffff800ba006c0 x12: ffffffc0f29dac20 
Jul  7 00:27:47 rockpro64 kernel: [  276.260287] x11: 00000000ffffffff x10: 0000000000000008 
Jul  7 00:27:47 rockpro64 kernel: [  276.265567] x9 : 0000000000aaaaaa x8 : 0000000000001000 
Jul  7 00:27:47 rockpro64 kernel: [  276.270820] x7 : 0000000000000000 x6 : 0000000000000001 
Jul  7 00:27:47 rockpro64 kernel: [  276.276040] x5 : 0000000000000000 x4 : 0000000000000000 
Jul  7 00:27:47 rockpro64 kernel: [  276.281157] x3 : 00000040eed1b000 x2 : 00000040eed1b000 
Jul  7 00:27:47 rockpro64 kernel: [  276.286200] x1 : 0000000000000005 x0 : 0000000000000026 
Jul  7 00:27:47 rockpro64 kernel: [  276.291240] 
Jul  7 00:27:47 rockpro64 kernel: [  276.291240] PC: 0xffffff800815629c:
Jul  7 00:27:47 rockpro64 kernel: [  276.300567] 629c  91022694 d538d080 38746801 34000041 3834681f 97ffff7a a94153f3 52800020
Jul  7 00:27:47 rockpro64 kernel: [  276.306060] 62bc  a9425bf5 a94363f7 a9446bf9 a94573fb a8c77bfd d65f03c0 91020282 f87a6860
Jul  7 00:27:47 rockpro64 kernel: [  276.311564] 62dc  f8626865 91022294 eb05001f 54000101 38746860 35fffb80 b9403082 90007200
Jul  7 00:27:47 rockpro64 kernel: [  276.317155] 62fc  910de000 340000c2 9400dfb4 f8226860 f87c7b60 3834681f 17ffffd3 9400e03e
Jul  7 00:27:47 rockpro64 kernel: [  276.322821] 631c  d4210000 f87c7b60 52800021 38346801 17ffffcd 97fe4698 f9444ea1 36080941
Jul  7 00:27:47 rockpro64 kernel: [  276.328584] 633c  b9401341 910003e2 340008e1 531f7821 d35efc00 8b21c2c1 cb000021 b6f80841
Jul  7 00:27:47 rockpro64 kernel: [  276.334401] 635c  6b160016 54000800 91022a99 d538d080 38606b21 34000101 91024294 f8746802
Jul  7 00:27:47 rockpro64 kernel: [  276.340277] 637c  d5384101 eb01005f 54fff980 3839681f 17ffffc9 34000157 90009981 52800000
Jul  7 00:27:47 rockpro64 kernel: [  276.346226] 
Jul  7 00:27:47 rockpro64 kernel: [  276.346226] LR: 0xffffff800815629c:
Jul  7 00:27:47 rockpro64 kernel: [  276.356609] 629c  91022694 d538d080 38746801 34000041 3834681f 97ffff7a a94153f3 52800020
Jul  7 00:27:47 rockpro64 kernel: [  276.362696] 62bc  a9425bf5 a94363f7 a9446bf9 a94573fb a8c77bfd d65f03c0 91020282 f87a6860
Jul  7 00:27:47 rockpro64 kernel: [  276.368843] 62dc  f8626865 91022294 eb05001f 54000101 38746860 35fffb80 b9403082 90007200
Jul  7 00:27:47 rockpro64 kernel: [  276.375029] 62fc  910de000 340000c2 9400dfb4 f8226860 f87c7b60 3834681f 17ffffd3 9400e03e
Jul  7 00:27:47 rockpro64 kernel: [  276.381277] 631c  d4210000 f87c7b60 52800021 38346801 17ffffcd 97fe4698 f9444ea1 36080941
Jul  7 00:27:47 rockpro64 kernel: [  276.387587] 633c  b9401341 910003e2 340008e1 531f7821 d35efc00 8b21c2c1 cb000021 b6f80841
Jul  7 00:27:47 rockpro64 kernel: [  276.393943] 635c  6b160016 54000800 91022a99 d538d080 38606b21 34000101 91024294 f8746802
Jul  7 00:27:47 rockpro64 kernel: [  276.400375] 637c  d5384101 eb01005f 54fff980 3839681f 17ffffc9 34000157 90009981 52800000
Jul  7 00:27:47 rockpro64 kernel: [  276.406846] 
Jul  7 00:27:47 rockpro64 kernel: [  276.406846] SP: 0xffffffc0f7f43c80:
Jul  7 00:27:47 rockpro64 kernel: [  276.418255] 3c80  000000fd 00000000 00000000 00000000 f2b0fda0 ffffffc0 f7f46250 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.424700] 3ca0  0922b238 ffffff80 092678a0 ffffff80 00000000 00000000 f7f43d00 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.430999] 3cc0  0815631c ffffff80 f7f43d00 ffffffc0 0815631c ffffff80 600001c5 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.437238] 3ce0  09266898 ffffff80 0000003a 00000000 00000000 00000080 f2400028 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.443416] 3d00  f7f43d70 ffffffc0 08120dbc ffffff80 f7f46250 ffffffc0 f7f45c00 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.449523] 3d20  4639b2b9 00000040 f7f45c80 ffffffc0 09225000 ffffff80 00000003 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.455638] 3d40  093d9290 ffffff80 4639b2b9 00000040 0815618c ffffff80 4639a1a2 00000040
Jul  7 00:27:47 rockpro64 kernel: [  276.461755] 3d60  f7f46250 ffffffc0 09266898 ffffff80 f7f43de0 ffffffc0 081213a0 ffffff80
Jul  7 00:27:47 rockpro64 kernel: [  276.467984] 
Jul  7 00:27:47 rockpro64 kernel: [  276.467984] X12: 0xffffffc0f29daba0:
Jul  7 00:27:47 rockpro64 kernel: [  276.478731] aba0  30303030 3030c630 30303030 787878c6 7c787878 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.484976] abc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.491248] abe0  06600606 06c00606 c6c6c60c 7cfe7c78 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.497530] ac00  c6060006 c606c606 c6067c06 c606c606 7c0cc60c 7c0c7c0c c618c018 c618c618
Jul  7 00:27:47 rockpro64 kernel: [  276.503868] ac20  00000000 00000000 00000000 00000000 00000000 00000000 00000000 38380000
Jul  7 00:27:47 rockpro64 kernel: [  276.510270] ac40  d6d6d6d6 d6d6d6d6 d6d6d6d6 d6d6d6d6 c6c6c6c6 c6c6c6c6 c6c6c6c6 c6c6c6c6
Jul  7 00:27:47 rockpro64 kernel: [  276.516765] ac60  d6d6d6cc c6ccc6cc cc00c0c0 ccc6ccc6 00c6c6c6 7c767c76 7c7c7c76 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.523335] ac80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.529992] 
Jul  7 00:27:47 rockpro64 kernel: [  276.529992] X13: 0xffffff800ba00640:
Jul  7 00:27:47 rockpro64 kernel: [  276.541821] 0640  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.548650] 0660  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.555451] 0680  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.562248] 06a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.569016] 06c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.575741] 06e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.582395] 0700  00000000 00000000 00000000 00000000 00aaaaaa 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.589027] 0720  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.595526] 
Jul  7 00:27:47 rockpro64 kernel: [  276.595526] X14: 0xffffff800ba00640:
Jul  7 00:27:47 rockpro64 kernel: [  276.606832] 0640  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.613276] 0660  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.619592] 0680  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.625788] 06a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.631856] 06c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.637803] 06e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.643607] 0700  00000000 00000000 00000000 00000000 00aaaaaa 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.649395] 0720  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.655058] 
Jul  7 00:27:47 rockpro64 kernel: [  276.655058] X19: 0xffffff800922af80:
Jul  7 00:27:47 rockpro64 kernel: [  276.664674] af80  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.670280] afa0  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.675762] afc0  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.681108] afe0  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.686326] b000  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.691420] b020  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.696375] b040  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.701298] b060  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.706079] 
Jul  7 00:27:47 rockpro64 kernel: [  276.706079] X20: 0xffffff800922b230:
Jul  7 00:27:47 rockpro64 kernel: [  276.714064] b230  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.718868] b250  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.723572] b270  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.728173] b290  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.732673] b2b0  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.737087] b2d0  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.741361] b2f0  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.745578] b310  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.749673] 
Jul  7 00:27:47 rockpro64 kernel: [  276.749673] X21: 0xffffff8009265f80:
Jul  7 00:27:47 rockpro64 kernel: [  276.756322] 5f80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.760484] 5fa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.764520] 5fc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.768417] 5fe0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.772209] 6000  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.775895] 6020  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.779436] 6040  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.782944] 6060  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.786312] 
Jul  7 00:27:47 rockpro64 kernel: [  276.786312] X24: 0xffffffc0f2b0fd20:
Jul  7 00:27:47 rockpro64 kernel: [  276.791734] fd20  00000000 00000000 f2af4600 ffffffc0 000001c0 00000000 089708c0 ffffff80
Jul  7 00:27:47 rockpro64 kernel: [  276.795422] fd40  00000002 00000000 08092188 ffffff80 f2b0fd60 ffffffc0 0809227c ffffff80
Jul  7 00:27:47 rockpro64 kernel: [  276.799179] fd60  f2b0fda0 ffffffc0 089efbbc ffffff80 f1d32210 ffffffc0 f1d31e10 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.803012] fd80  f2b0fda0 ffffffc0 089efc0c ffffff80 f1d32000 ffffffc0 085d0a14 ffffff80
Jul  7 00:27:47 rockpro64 kernel: [  276.806863] fda0  00000000 00000000 00000005 00000000 eed1b000 00000040 00000020 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.810814] fdc0  ffffffff 00ffffff 5d6e6ac3 00000001 00013240 00000000 00000005 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.814696] fde0  b5593519 00000034 08081800 ffffff80 00001000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.818622] fe00  34d5d91d 00000000 eed1b000 00000040 f2af4600 ffffffc0 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.822484] 
Jul  7 00:27:47 rockpro64 kernel: [  276.822484] X25: 0xffffffc0f7f461d0:
Jul  7 00:27:47 rockpro64 kernel: [  276.828883] 61d0  ffffffff 00000000 ffffffff ffffffff 00000001 00000000 f7f461e8 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.833044] 61f0  f7f461e8 ffffffc0 f7f461f8 ffffffc0 f7f461f8 ffffffc0 0814a210 ffffff80
Jul  7 00:27:47 rockpro64 kernel: [  276.837304] 6210  f2aa7cf8 ffffffc0 f2aa7c50 ffffffc0 00020002 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.841659] 6230  00000001 00000000 00000045 00000000 00000044 00000000 f2b87000 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.846093] 6250  f7f46250 ffffffc0 00000000 00000000 00000000 00000000 46384600 00000040
Jul  7 00:27:47 rockpro64 kernel: [  276.850589] 6270  46384600 00000040 0815618c ffffff80 f7f45c80 ffffffc0 00000000 0000001f
Jul  7 00:27:47 rockpro64 kernel: [  276.855051] 6290  08156160 ffffff80 63746177 676f6468 0000352f 00000000 00000044 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.859579] 62b0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.864041] 
Jul  7 00:27:47 rockpro64 kernel: [  276.864041] X26: 0xffffff800922b1b8:
Jul  7 00:27:47 rockpro64 kernel: [  276.871664] b1b8  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.876458] b1d8  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.881255] b1f8  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.886048] b218  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.890843] b238  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.895635] b258  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.900283] b278  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.904883] b298  ******** ******** ******** ******** ******** ******** ******** ********
Jul  7 00:27:47 rockpro64 kernel: [  276.909336] 
Jul  7 00:27:47 rockpro64 kernel: [  276.909336] X27: 0xffffff8009267820:
Jul  7 00:27:47 rockpro64 kernel: [  276.916932] 7820  00000001 00000000 00000001 00000000 000000fa 00000000 f7f55480 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.921706] 7840  00000015 00000000 f7f55440 ffffffc0 00000001 00000006 00000060 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.926579] 7860  00015000 00001000 00000001 00000000 f7f554c0 ffffffc0 f7f578c0 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.931567] 7880  00000000 00000005 00000010 00000000 f7f55340 ffffffc0 f7ed7000 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.936570] 78a0  eecb2000 00000040 eecc7000 00000040 eecdc000 00000040 eecf1000 00000040
Jul  7 00:27:47 rockpro64 kernel: [  276.941666] 78c0  eed06000 00000040 eed1b000 00000040 00000000 00000000 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.946663] 78e0  00000002 00000000 0000341a 00000000 00010000 00000000 000f7fff 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.951672] 7900  0001e26f 00000000 00002000 00000000 00000000 0000fffa 00000000 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.956584] 
Jul  7 00:27:47 rockpro64 kernel: [  276.956584] X29: 0xffffffc0f7f43c80:
Jul  7 00:27:47 rockpro64 kernel: [  276.965091] 3c80  000000fd 00000000 00000000 00000000 f2b0fda0 ffffffc0 f7f46250 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.970327] 3ca0  0922b238 ffffff80 092678a0 ffffff80 00000000 00000000 f7f43d00 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.975652] 3cc0  0815631c ffffff80 f7f43d00 ffffffc0 0815631c ffffff80 600001c5 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.981076] 3ce0  09266898 ffffff80 0000003a 00000000 00000000 00000080 f2400028 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.986605] 3d00  f7f43d70 ffffffc0 08120dbc ffffff80 f7f46250 ffffffc0 f7f45c00 ffffffc0
Jul  7 00:27:47 rockpro64 kernel: [  276.992229] 3d20  4639b2b9 00000040 f7f45c80 ffffffc0 09225000 ffffff80 00000003 00000000
Jul  7 00:27:47 rockpro64 kernel: [  276.997814] 3d40  093d9290 ffffff80 4639b2b9 00000040 0815618c ffffff80 4639a1a2 00000040
Jul  7 00:27:47 rockpro64 kernel: [  277.003464] 3d60  f7f46250 ffffffc0 09266898 ffffff80 f7f43de0 ffffffc0 081213a0 ffffff80
Jul  7 00:27:47 rockpro64 kernel: [  277.009073] 
Jul  7 00:27:47 rockpro64 kernel: [  277.013844] ---[ end trace 8ad4400a27e2532b ]---
Jul  7 00:27:47 rockpro64 kernel: [  277.018940] Call trace:
Jul  7 00:27:47 rockpro64 kernel: [  277.023832] Exception stack(0xffffffc0f7f43b30 to 0xffffffc0f7f43c60)
Jul  7 00:27:47 rockpro64 kernel: [  277.029162] 3b20:                                   ffffff800922b000 0000008000000000
Jul  7 00:27:47 rockpro64 kernel: [  277.034695] 3b40: ffffffc0f7f43d00 ffffff800815631c 0000000000000026 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  277.040252] 3b60: ffffffc0f7f43c00 ffffff800810b958 ffffffc0f7f43c80 ffffff8008f96378
Jul  7 00:27:47 rockpro64 kernel: [  277.045809] 3b80: ffffff8009266000 00000000000000fd 0000000000000000 ffffffc0f2b0fda0
Jul  7 00:27:47 rockpro64 kernel: [  277.051316] 3ba0: ffffffc0f7f46250 ffffff800922b238 ffffff80092678a0 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  277.056895] 3bc0: ffffffc0f7f43be0 ffffff8009225000 0000000000000026 0000000000000005
Jul  7 00:27:47 rockpro64 kernel: [  277.062391] 3be0: 00000040eed1b000 00000040eed1b000 0000000000000000 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  277.067850] 3c00: 0000000000000001 0000000000000000 0000000000001000 0000000000aaaaaa
Jul  7 00:27:47 rockpro64 kernel: [  277.073271] 3c20: 0000000000000008 00000000ffffffff ffffffc0f29dac20 ffffff800ba006c0
Jul  7 00:27:47 rockpro64 kernel: [  277.078672] 3c40: ffffff800ba006c0 0000000000000000 0000000000000000 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  277.084108] [<ffffff800815631c>] watchdog_timer_fn+0x190/0x2e8
Jul  7 00:27:47 rockpro64 kernel: [  277.089350] [<ffffff8008120dbc>] __hrtimer_run_queues+0x1bc/0x284
Jul  7 00:27:47 rockpro64 kernel: [  277.094585] [<ffffff80081213a0>] hrtimer_interrupt+0xac/0x1bc
Jul  7 00:27:47 rockpro64 kernel: [  277.099778] [<ffffff800812cdfc>] tick_handle_oneshot_broadcast+0x144/0x1c0
Jul  7 00:27:47 rockpro64 kernel: [  277.105045] [<ffffff80089719a4>] rk_timer_interrupt+0x4c/0x5c
Jul  7 00:27:47 rockpro64 kernel: [  277.110241] [<ffffff800810d0c8>] handle_irq_event_percpu+0xf4/0x1f0
Jul  7 00:27:47 rockpro64 kernel: [  277.115419] [<ffffff800810d214>] handle_irq_event+0x50/0x80
Jul  7 00:27:47 rockpro64 kernel: [  277.120565] [<ffffff8008110a50>] handle_fasteoi_irq+0xcc/0x134
Jul  7 00:27:47 rockpro64 kernel: [  277.125794] [<ffffff800810c5c4>] generic_handle_irq+0x2c/0x44
Jul  7 00:27:47 rockpro64 kernel: [  277.131064] [<ffffff800810c940>] __handle_domain_irq+0xb4/0xb8
Jul  7 00:27:47 rockpro64 kernel: [  277.136404] [<ffffff8008080e70>] gic_handle_irq+0xc8/0x180
Jul  7 00:27:47 rockpro64 kernel: [  277.141769] Exception stack(0xffffffc0f2b0fda0 to 0xffffffc0f2b0fed0)
Jul  7 00:27:47 rockpro64 kernel: [  277.147335] fda0: 0000000000000000 0000000000000005 00000040eed1b000 0000000000000020
Jul  7 00:27:47 rockpro64 kernel: [  277.153195] fdc0: 00ffffffffffffff 000000015d6e6ac3 0000000000013240 0000000000000005
Jul  7 00:27:47 rockpro64 kernel: [  277.158984] fde0: 00000034b5593519 ffffff8008081800 0000000000001000 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  277.164811] fe00: 0000000034d5d91d 00000040eed1b000 ffffffc0f2af4600 0000000000000000
Jul  7 00:27:47 rockpro64 kernel: [  277.170576] fe20: 0000000000000000 0000000000000000 0000000030d00800 00000040463979c2
Jul  7 00:27:47 rockpro64 kernel: [  277.176342] fe40: 0000000000000002 ffffffc0efd9b400 0000000000000002 ffffff80093d97b8
Jul  7 00:27:47 rockpro64 kernel: [  277.182075] fe60: 0000000000000001 0000004045fd1bb7 ffffff80093720c0 0000000002c9d234
Jul  7 00:27:47 rockpro64 kernel: [  277.187875] fe80: 0000000000000000 ffffffc0f2b0fed0 ffffff8008948f34 ffffffc0f2b0fed0
Jul  7 00:27:47 rockpro64 kernel: [  277.193661] fea0: ffffff8008948f38 0000000060000145 ffffffc0f2b0fed0 ffffff8008948f34
Jul  7 00:27:47 rockpro64 kernel: [  277.199506] fec0: ffffffffffffffff 0000000000000002
Jul  7 00:27:47 rockpro64 kernel: [  277.205075] [<ffffff80080827b4>] el1_irq+0xb4/0x140
Jul  7 00:27:47 rockpro64 kernel: [  277.210676] [<ffffff8008948f38>] cpuidle_enter_state+0x1fc/0x254
Jul  7 00:27:47 rockpro64 kernel: [  277.216358] [<ffffff8008949004>] cpuidle_enter+0x34/0x44
Jul  7 00:27:47 rockpro64 kernel: [  277.221947] [<ffffff80080fe400>] call_cpuidle+0x70/0x78
Jul  7 00:27:47 rockpro64 kernel: [  277.227432] [<ffffff80080fe6a4>] cpu_startup_entry+0x29c/0x2ac
Jul  7 00:27:47 rockpro64 kernel: [  277.232944] [<ffffff800808d550>] secondary_start_kernel+0x190/0x1bc
Jul  7 00:27:47 rockpro64 kernel: [  277.238509] [<0000000002c9d178>] 0x2c9d178

[ayufan]

Are you sure that you don't have underpower issue? This clearly seems to be the one. Maybe too weak power adapter?

[stevefan1999-personal]

Are you sure that you don't have underpower issue? This clearly seems to be the one. Maybe too weak power adapter?

I don't think 3A 5V with my R64 4GB is underpowered

[GuyNoIRQ]

I'm running the 12v 5A adapter on the RockPro64.

Also doing some testing I can dd straight from one disk to the other without hitting this. I'm seeing some of the below errors in kern.log sporadically, but if I md5sum large portions of the disks mirrored via dd I don't see any discrepancies.

errors

root@RockProNAS:~# tail -f /var/log/kern.log 
Jul  9 22:48:08 RockProNAS kernel: [  109.146034] zram: Decompression failed! err=-22, page=34
Jul  9 22:48:08 RockProNAS kernel: [  109.146119] zram: Decompression failed! err=-22, page=34
Jul  9 22:48:08 RockProNAS kernel: [  109.146129] Buffer I/O error on dev zram0, logical block 34, async page read
Jul  9 22:48:11 RockProNAS kernel: [  112.218832] md127: detected capacity change from 8001428979712 to 0
Jul  9 22:48:11 RockProNAS kernel: [  112.218864] md: md127 stopped.
Jul  9 22:48:11 RockProNAS kernel: [  112.218935] md: unbind<sdb>
Jul  9 22:48:11 RockProNAS kernel: [  112.239895] md: export_rdev(sdb)
Jul  9 22:48:11 RockProNAS kernel: [  112.240083] md: unbind<sda>
Jul  9 22:48:11 RockProNAS kernel: [  112.252111] md: export_rdev(sda)
Jul  9 22:48:29 RockProNAS kernel: [  129.772559] cdn-dp fec00000.dp: [drm:cdn_dp_request_firmware] *ERROR* Timed out trying to load firmware
Jul  9 22:52:19 RockProNAS kernel: [  359.473686] zram: Decompression failed! err=-22, page=619
Jul  9 22:52:19 RockProNAS kernel: [  359.495814] zram: Decompression failed! err=-22, page=619
Jul  9 22:52:19 RockProNAS kernel: [  359.497577] zram: Decompression failed! err=-22, page=619
Jul  9 22:52:20 RockProNAS kernel: [  361.193502] zram: Decompression failed! err=-22, page=34
Jul  9 22:52:20 RockProNAS kernel: [  361.193543] zram: Decompression failed! err=-22, page=34
Jul  9 22:52:20 RockProNAS kernel: [  361.193633] zram: Decompression failed! err=-22, page=34
Jul  9 22:52:20 RockProNAS kernel: [  361.193643] Buffer I/O error on dev zram0, logical block 34, async page read
Jul  9 22:53:09 RockProNAS kernel: [  410.166900] zram: Decompression failed! err=-22, page=34
Jul  9 22:53:09 RockProNAS kernel: [  410.166943] zram: Decompression failed! err=-22, page=34
Jul  9 22:53:09 RockProNAS kernel: [  410.167034] zram: Decompression failed! err=-22, page=34
Jul  9 22:53:09 RockProNAS kernel: [  410.167044] Buffer I/O error on dev zram0, logical block 34, async page read
.....truncated.....

Jul  9 23:25:49 RockProNAS kernel: [ 2369.682516] zram: Decompression failed! err=-22, page=34
Jul  9 23:25:49 RockProNAS kernel: [ 2369.682529] Buffer I/O error on dev zram0, logical block 34, async page read
Jul  9 23:26:47 RockProNAS kernel: [ 2428.151934] zram: Decompression failed! err=-22, page=34
Jul  9 23:26:47 RockProNAS kernel: [ 2428.151953] Buffer I/O error on dev zram0, logical block 34, async page read
Jul  9 23:26:47 RockProNAS kernel: [ 2428.207522] zram: Decompression failed! err=-22, page=34
Jul  9 23:26:47 RockProNAS kernel: [ 2428.207536] Buffer I/O error on dev zram0, logical block 34, async page read
Jul  9 23:27:50 RockProNAS kernel: [ 2490.648482] zram: Decompression failed! err=-22, page=34
Jul  9 23:27:50 RockProNAS kernel: [ 2490.648527] zram: Decompression failed! err=-22, page=34
Jul  9 23:27:50 RockProNAS kernel: [ 2490.648619] zram: Decompression failed! err=-22, page=34
Jul  9 23:27:50 RockProNAS kernel: [ 2490.648630] Buffer I/O error on dev zram0, logical block 34, async page read
Jul  9 23:27:50 RockProNAS kernel: [ 2490.692936] zram: Decompression failed! err=-22, page=34
Jul  9 23:27:50 RockProNAS kernel: [ 2490.692952] Buffer I/O error on dev zram0, logical block 34, async page read

the dd command I used to copy from sda to sdb

root@RockProNAS:~# dd if=/dev/sda of=/dev/sdb bs=4M status=progress
391034961920 bytes (391 GB, 364 GiB) copied, 2059.02 s, 190 MB/s^C^C

A coupld md5sum checks of what should be identical blocks.

root@RockProNAS:~# sudo dd if=/dev/sdb bs=4M skip=1000 count=10000 | md5sum
10000+0 records in
10000+0 records out
41943040000 bytes (42 GB, 39 GiB) copied, 436.12 s, 96.2 MB/s
d8c26ac026147737e73aee59eb37766a  -
root@RockProNAS:~# sudo dd if=/dev/sda bs=4M skip=1000 count=10000 | md5sum
10000+0 records in
10000+0 records out
41943040000 bytes (42 GB, 39 GiB) copied, 429.064 s, 97.8 MB/s
d8c26ac026147737e73aee59eb37766a  -

EDIT: Maybe dd doesn't work. I let it run overnight and came back to the box powered down and not respondig. Unfortunately I wasn't able to gather any logs from this. I'm going to run it again to see if I can repro the issue and tail -f kern.log out to a separate file so I can see if I can glean any more info into what happened.

EDIT 2: dd isn't stable. I hit this when dding /dev/zero to one of the disks.

EDIT 3: I hit the spinlock error when I simply formatted just one of the disks with ext4.

[GuyNoIRQ]

I actually figured out that I had a bad PCI card. I picked up the one at the link below, and added the "pci=nomsi" boot parameter as outlined in the following forum post and was able to initialize raid1 on 2x white label HGST as well as 2x older intel SSDs I had laying around at the same time while also running "stress -c4 -m16" then further could run dd of /dev/urandom to both raid devices and all is stable.

https://www.amazon.com/gp/product/B07KNXZFRH/ref=ppx_yo_dt_b_asin_title_o04_s00?ie=UTF8&psc=1

https://forum.pine64.org/showthread.php?tid=6496

[jwindhager]

I have the same issue.

My configuration: ROCKPro64 4GB Single Board Computer ROCKPro64 PCI-e to Dual SATA-II Interface Card ROCKPro64 Metal Desktop/NAS Casing with fan ROCKPro64 12V 5A EU Power Supply 2x Seagate IronWolf NAS HDD, 4TB each

Installed 0.9.14 (tried both armhf and arm64) first, updated all packages, activated SMART monitoring, quick-wiped both disks, created RAID1 via the OMV web interface. Server crashes at random times, web interface not reachable, SSH doesn't work. No dmesg output (via SSH) before the crash. Sometimes still responds to iCMP pings, sometimes doesn't. After a power cycle, RAID is in "sync pending" state, nothing suspicious in kern.log.

Then tried a vanilla armbian (stretch), updated all packages, installed OMV via armbian-config, created RAID1 via the OMV web interface as before. This time received a kernel panic while connected via SSH:

Message from syslogd@localhost at Nov 18 10:01:47 ...
kernel:[ 75.976355] Internal error: : 96000210 [#1] SMP

Message from syslogd@localhost at Nov 18 10:01:47 ...
kernel:[ 83.247622] BUG: spinlock lockup suspected on CPU#2, scsi_eh_1/284

Message from syslogd@localhost at Nov 18 10:01:47 ...
kernel:[ 83.251260] lock: 0xffffff8009121870, .magic: dead4ead, .owner: scsi_eh_1/284, .owner_cpu: 2

Message from syslogd@localhost at Nov 18 10:03:59 ...
kernel:[ 240.428491] Kernel panic - not syncing: hung_task: blocked tasks

Finally tried booting the armbian-setup with pci=nomsi, resulting in the same behavior as with the 0.9.14 image (no error message, just crashed).

Any updates on this issue?

[ayufan]

Can you try to force downclock RAM to 400MHz? I would assume that this is due to this (this was usually thing in the past). Is not that RAM is bad, but maybe some setting is off somewhere.

On Mon, Nov 18, 2019 at 12:22 PM Jonas Windhager notifications@github.com wrote:

I have the same issue.

My configuration: ROCKPro64 4GB Single Board Computer ROCKPro64 PCI-e to Dual SATA-II Interface Card ROCKPro64 Metal Desktop/NAS Casing with fan ROCKPro64 12V 5A EU Power Supply 2x Seagate IronWolf NAS HDD, 4TB each

Installed 0.9.14 (tried both armhf and arm64) first, updated all packages, activated SMART monitoring, quick-wiped both disks, created RAID1 via the OMV web interface. Server crashes at random times, web interface not reachable, SSH doesn't work. No dmesg output (via SSH) before the crash. Sometimes still responds to iCMP pings, sometimes doesn't. After a power cycle, RAID is in "sync pending" state, nothing suspicious in kern.log.

Then tried a vanilla armbian (stretch), updated all packages, installed OMV via armbian-config, created RAID1 via the OMV web interface as before. This time received a kernel panic while connected via SSH:

Message from syslogd@localhost at Nov 18 10:01:47 ... kernel:[ 75.976355] Internal error: : 96000210 [#1] SMP

Message from syslogd@localhost at Nov 18 10:01:47 ... kernel:[ 83.247622] BUG: spinlock lockup suspected on CPU#2, scsi_eh_1/284

Message from syslogd@localhost at Nov 18 10:01:47 ... kernel:[ 83.251260] lock: 0xffffff8009121870, .magic: dead4ead, .owner: scsi_eh_1/284, .owner_cpu: 2

Message from syslogd@localhost at Nov 18 10:03:59 ... kernel:[ 240.428491] Kernel panic - not syncing: hung_task: blocked tasks

Finally tried booting the armbian-setup with pci=nomsi, resulting in the same behavior as with the 0.9.14 image (no error message, just crashed).

Any updates on this issue?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ayufan-rock64/linux-build/issues/299?email_source=notifications&email_token=AASOSQNZAVHYAOKOC2HOG63QUJ3IBA5CNFSM4GBUZLP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEKDR3I#issuecomment-554973421, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASOSQP57VJPCZJV63UYAY3QUJ3IBANCNFSM4GBUZLPQ .

[jwindhager]

Hi ayufan! Thanks for your reply. How can I force the RAM clock down to 400MHz?

[ayufan]

Look in /sys/class/devfreq/dmc

[jwindhager]

Unfortunately, setting /sys/class/devfreq/dmc/max_freq to 400000000 didn't help. This time, however, dmesg recorded the problem "on time":

[Nov18 23:22] Unhandled fault: synchronous external abort (0x96000210) at 0xffffff80093e6130
[  +0.008153] Internal error: : 96000210 [#1] SMP

Message from syslogd@localhost at Nov 18 23:22:09 ...
 kernel:[ 2234.999648] Internal error: : 96000210 [#1] SMP
[  +0.007787] Modules linked in: softdog af_packet lz4hc lz4hc_compress zlib snd_soc_rockchip_hdmi_dp rk_vcodec lzo zram ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid1 raid0 multipath linear md_mod phy_rockchip_pcie
[  +0.026297] CPU: 0 PID: 275 Comm: scsi_eh_0 Not tainted 4.4.192-rockchip64 #1
[  +0.008800] Hardware name: Pine64 RockPro64 (DT)
[  +0.008527] task: ffffffc0dfb2f000 task.stack: ffffffc0f13c8000
[  +0.008705] PC is at ahci_scr_read+0x70/0x98
[  +0.008534] LR is at sata_scr_read+0x88/0x94
[  +0.008413] pc : [<ffffff800875bf08>] lr : [<ffffff80087427cc>] pstate: 80000145
[  +0.008847] sp : ffffffc0f13cba50
[  +0.008375] x29: ffffffc0f13cba50 x28: 0000000000000000 
[  +0.008557] x27: ffffffc0f111de80 x26: 0000000000000001 
[  +0.008501] x25: ffffffc0f111e258 x24: ffffffc0f111c000 
[  +0.008404] x23: 0000000000000000 x22: ffffffc0f13cbb28 
[  +0.008362] x21: ffffffc0f111de80 x20: ffffffc0f13cbb28 
[  +0.008370] x19: 0000000000000001 x18: 0000000030d00800 
[  +0.008406] x17: 0000000000000000 x16: 0000000000000000 
[  +0.008354] x15: 0000000000000000 x14: 0000000000000000 
[  +0.008249] x13: 0000000000000000 x12: 0000000000000000 
[  +0.008137] x11: 0000000000000000 x10: 0000000000000000 
[  +0.007989] x9 : 0000000000000000 x8 : ffffffc0f111e258 
[  +0.007940] x7 : 0000000000000000 x6 : ffffffc0f111e2d0 
[  +0.007889] x5 : 0000000000000000 x4 : ffffff8008caa2fc 
[  +0.007881] x3 : 0000000000000130 x2 : ffffffc0f13cbb28 
[  +0.007871] x1 : 0000000000000030 x0 : ffffff80093e6130 
[  +0.007827] 
              PC: 0xffffff800875be88:
[  +0.014995] be88  52800013 91392821 97fe24e0 17ffffe2 a9bd7bfd 910003fd a90153f3 f90013f5
[  +0.008527] bea8  aa0003f5 2a0103f3 aa1e03e0 aa0203f4 d503201f f94002a3 7100127f f959dc60
[  +0.008603] bec8  b9402c63 f9401801 53196063 91040063 f9400820 54000268 54000061 b9401821
[  +0.008668] bee8  36e80201 f0002a64 910b8084 91007084 b8735881 34000161 8b21c063 8b030000
[  +0.008674] bf08  b9400000 d5033d9f b9000280 52800000 a94153f3 f94013f5 a8c37bfd d65f03c0
[  +0.008708] bf28  128002a0 17fffffb a9be7bfd 910003fd f9000bf3 aa0003f3 aa1e03e0 d503201f
[  +0.008725] bf48  f9400260 f9400000 f959dc00 f9401800 b9400000 36400120 52801fe0 b9038660
[  +0.008736] bf68  f0003c22 900038a1 91399842 912ff421 aa1303e0 97ff9d5d f9400bf3 a8c27bfd
[  +0.008855] 
              LR: 0xffffff800874274c:
[  +0.016111] 274c  a90153f3 a9025bf5 aa0003f3 2a0103f5 aa1e03e0 aa0203f6 d503201f f9400274
[  +0.008943] 276c  d283d000 8b000280 eb00027f 54000180 f959c280 eb00027f 54000120 aa1603e2
[  +0.008966] 278c  2a1503e1 aa1303e0 9400573d a94153f3 a9425bf5 a8c37bfd d65f03c0 aa1303e0
[  +0.008985] 27ac  97fffb19 34000100 f9400680 aa1603e2 2a1503e1 f9406c03 aa1303e0 d63f0060
[  +0.009029] 27cc  17fffff3 12800bc0 17fffff1 a9ba7bfd 910003fd a90153f3 a9025bf5 a90363f7
[  +0.009210] 27ec  a9046bf9 aa0103f5 aa0203f6 aa0003f7 aa1e03e0 d503201f f0004e93 b94012a0
[  +0.009226] 280c  f940e274 a94066b8 97e76ebe 8b000280 cb160001 910173a2 f100003f 52800001
[  +0.009244] 282c  9a80a2d6 aa1703e0 97ffffc4 2a0003f5 350003c0 b9405fb4 f940e27a 12000e94
[  +0.009336] 
              SP: 0xffffffc0f13cb9d0:
[  +0.016472] b9d0  f13cbb28 ffffffc0 00000000 00000000 f111c000 ffffffc0 f111e258 ffffffc0
[  +0.008879] b9f0  00000001 00000000 f111de80 ffffffc0 00000000 00000000 f13cba50 ffffffc0
[  +0.008658] ba10  087427cc ffffff80 f13cba50 ffffffc0 0875bf08 ffffff80 80000145 00000000
[  +0.008461] ba30  091178a0 ffffff80 f7ede200 ffffffc0 ffffffff ffffffff 080f0094 ffffff80
[  +0.008373] ba50  f13cba80 ffffffc0 087427cc ffffff80 f111de80 ffffffc0 f111c000 ffffffc0
[  +0.008306] ba70  00000001 00000000 f2aa4680 ffffffc0 f13cbab0 ffffffc0 0874f6ac ffffff80
[  +0.008282] ba90  f111e258 ffffffc0 f111de80 ffffffc0 f111f858 ffffffc0 00003008 00000000
[  +0.008266] bab0  f13cbb60 ffffffc0 0874fe6c ffffff80 f111c000 ffffffc0 f111de80 ffffffc0
[  +0.008353] 
              X0: 0xffffff80093e60b0:
[  +0.014831] 60b0 
[  +0.050245] Unhandled fault: synchronous external abort (0x96000210) at 0xffffff80093e60b0
[  +0.961736] rk3x-i2c ff3c0000.i2c: timeout, ipd: 0x10, state: 1

[romanczukiewiczs]

Hi,

Is there some person who has this setup with HDD disk and this works correctly?

ROCKPro64 4GB Single Board Computer ROCKPro64 PCI-e to Dual SATA-II Interface Card ROCKPro64 Metal Desktop/NAS Casing with fan ROCKPro64 12V 5A EU Power Supply

Thanks in advance.

[bullet64]

@romanczukiewiczs Please ask in https://forum.pine64.org/index.php

I have massiv problems with pine64 pcie-to-sata adapter. Now i use Marvell 88SE9230 without problems.

[romanczukiewiczs]

@bullet64 do You use this with pine64rockpro? You did on this setup something with mdadm?

[bullet64]

Yes. Please read about it in my forum. My forum profile https://forum.pine64.org/member.php?action=profile&uid=8601

[abdel-unxp]

Hi Any news on that ? I had the same issue on manjaro and debian buster. But the issue is not only on PCIe to sata card:

I can get, like you, the issue in 5 minutes with PCIe and SATA disk if I use syncthing and let it scan my files or if I use dd command. But I have the same issue if I remove the PCIe card and if I do hexdump /dev/eMMC disk or sdcard and wait 10 to 40 minutes. I would have never noticed that if I didn't decided to do more test on my eMMC and sd card driver.

When I use the sdcard I can see that when I have the issue, it is always after 10 seconds of DMA failure (DMA is started but we dont get the DMA done interruption). So I first though it was an issue on sdcard driver but no: If I remove the sdcard and use eMMC only, I will also get the same (a)synchronous external abort oops

Note: When adding debug messages in the kernel driver, I was taking more time to get the oops, but you will get it

It is unlikely that we have similar BUG in eMMC, sdcard and PCIe so I think it is a hardware issue. But why some people dont have this issue ? Maybe only some devices have this BUG ?

I am talking about this issue here: https://forum.pine64.org/showthread.php?tid=8372

[jwindhager]

I couldn't make it work with the Pine64 card. Now using a Marvell-based one (88SE9128) without a single dmesg error for 3 weeks already.

[chippey5]

Bump, I'm having the same issue and I have created a ticket to the Pine64 support staff, waiting for a reply. I feel like there should be more users out there having the same issue, unless other users either don't use/have this card (Asmedia ASM1061), or we're just unlucky to have received defect ones.

I'll keep you posted for their reply.

[abdel-unxp]

Do you have (a)synchronous external abort ? Which OS are you using ? I dont have the issue anymore. There is no hardware issue. My rockpro64 is working fine without issue and I even did intensive read/write for several hours. The problem was bootloader uboot which use a blob FW that use memory that Linux kernel is not aware of. The solution is to use blobless uboot version. Depending on the OS, I know people were working on it, but i dont know if they are done or not. But you can do it manually by following patstew's instruction: https://forum.pine64.org/showthread.php?tid=8372&page=2

[chippey5]

This is the image I use: stretch-openmediavault-rockpro64-0.9.14-1159-arm64.

The only odd thing is that when i inspect the chip itself it says ASM1061 but lspci returns SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01).

[abdel-unxp]

I also have that: SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) I suppose that previous hardware was ASM1061 and people didnt see the issue with this hardware.

But in any case, you should know that the issue is not related to PCIe at all. You can unplug it, and then do hexdum /dev//dev/mmcblk1p1 in a loop and wait 30 minutes. You will get the same kernel oops. You should get this BUG on every operation that use the same RAM address that rockchip blob FW

[chippey5]

I did as you suggested; unplugged the PCIe-adapter, booted the board and ran hexdump /dev/mmcblk0p1 in a loop alongside with stress --vm 2 --vm-bytes=800m to fill up the RAM for about two hours. I averaged on 78° and a load average of 4.83 4.70 4.61, yet it doesn't spinlock without the PCIe-adapter in. I'm not sure we have/had the same root cause of the spinlock.

I'm still a bit confused of the model mismatch on the PCIe-adapter and controller. The chip clearly says ASM1061 and controller ASM1062. The spinlock only happens after about 30 minutes after I initiate RAID0 on the HDDs.

[abdel-unxp]

On my card it is also written ASM1061. reference and datasheet in the wiki mention also this same number: https://wiki.pine64.org/index.php/ROCKPro64 But lspci show like you ASM1062

Maybe you couldn't trigger the issue because the bad memory was unused but reserved by someone else like in the kernel ? Or maybe ayufan (I think he is rockpro64's openmediavault guy) did the fix and your are facing something else ? I would be surprised because using my PCIe card I also had freeze in no time when using syncthing (or dd I think) You can try to reach him on IRC

Do you have serial console ? If you have similar oops as what jwindhager had, then we are likely talking about the same issue.

If I were you I would try to use newer uboot as mentioned before. 90% sure it will fix your problem.

[chippey5]

Update: I followed the steps mentioned in https://forum.pine64.org/showthread.php?tid=8372&page=2, but no ball. the board just crashes within 30 minutes again after initiating my RAID0 resync.

Here are the logs:

Jan 26 00:26:04 rockpro64 kernel: [ 1327.968455] md: md0 switched to read-write mode.
Jan 26 00:26:06 rockpro64 kernel: [ 1329.506015] md: resync of RAID array md0
Jan 26 00:26:06 rockpro64 kernel: [ 1329.506458] md: minimum _guaranteed_  speed: 10000 KB/sec/disk.
Jan 26 00:26:06 rockpro64 kernel: [ 1329.507070] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for resync.
Jan 26 00:26:06 rockpro64 kernel: [ 1329.508125] md: using 128k window, over a total of 3906887488k.

Message from syslogd@rockpro64 at Jan 26 00:46:10 ...
 kernel:[ 2493.888554] Internal error: : 96000210 [#1] SMP

Message from syslogd@rockpro64 at Jan 26 00:46:10 ...
 kernel:[ 2501.628391] BUG: spinlock lockup suspected on CPU#1, swapper/1/0

Message from syslogd@rockpro64 at Jan 26 00:46:10 ...
 kernel:[ 2501.634647]  lock: 0xffffff800926fdf0, .magic: dead4ead, .owner: swapper/1/0, .owner_cpu: 1
Jan 26 00:46:10 rockpro64 kernel: [ 2493.887680] Unhandled fault: synchronous external abort (0x96000210) at 0xffffff80096ce008
Jan 26 00:46:10 rockpro64 kernel: [ 2493.888554] Internal error: : 96000210 [#1] SMP
Jan 26 00:46:10 rockpro64 kernel: [ 2493.889023] Modules linked in: midgard_kbase rockchip_saradc ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid0 mu                                                                   ltipath linear raid1 md_mod
Jan 26 00:46:10 rockpro64 kernel: [ 2493.890954] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.4.190-1233-rockchip-ayufan-gd3f1be0ed310 #1
Jan 26 00:46:10 rockpro64 kernel: [ 2493.891875] Hardware name: Pine64 RockPro64 (DT)
Jan 26 00:46:10 rockpro64 kernel: [ 2493.892350] task: ffffffc078ee0e00 task.stack: ffffffc078eec000
Jan 26 00:46:10 rockpro64 kernel: [ 2493.892967] PC is at ahci_single_level_irq_intr+0x28/0x84
Jan 26 00:46:10 rockpro64 kernel: [ 2493.893524] LR is at handle_irq_event_percpu+0xf4/0x1f0
Jan 26 00:46:10 rockpro64 kernel: [ 2493.894060] pc : [<ffffff8008781824>] lr : [<ffffff800810ca20>] pstate: 600001c5
Jan 26 00:46:10 rockpro64 kernel: [ 2493.894814] sp : ffffffc07ff01c20
Jan 26 00:46:10 rockpro64 kernel: [ 2493.895156] x29: ffffffc07ff01c20 x28: ffffff8008cbd000
Jan 26 00:46:10 rockpro64 kernel: [ 2493.895712] x27: 0000000000000000 x26: ffffffc07ff02050
Jan 26 00:46:10 rockpro64 kernel: [ 2493.896269] x25: ffffff8009228000 x24: ffffff80093d9438
Jan 26 00:46:10 rockpro64 kernel: [ 2493.896825] x23: ffffffc07619d200 x22: 0000000000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2493.897381] x21: 00000000000000e9 x20: ffffff80096ce008
Jan 26 00:46:10 rockpro64 kernel: [ 2493.897938] x19: ffffffc0760fae28 x18: 0000000030d00800
Jan 26 00:46:10 rockpro64 kernel: [ 2493.898494] x17: 0000000000000000 x16: 0000000000000150
Jan 26 00:46:10 rockpro64 kernel: [ 2493.899050] x15: 000000000000000e x14: 0000000000000001
Jan 26 00:46:10 rockpro64 kernel: [ 2493.899605] x13: 0000000000000001 x12: 000000000000005e
Jan 26 00:46:10 rockpro64 kernel: [ 2493.900162] x11: 0000000000000191 x10: 0000000000000a30
Jan 26 00:46:10 rockpro64 kernel: [ 2493.900718] x9 : ffffffc078eefef0 x8 : ffffffc078858a90
Jan 26 00:46:10 rockpro64 kernel: [ 2493.901274] x7 : ffffffc078858948 x6 : ffffffc078858920
Jan 26 00:46:10 rockpro64 kernel: [ 2493.901831] x5 : 00000000fffffffa x4 : ffffffc07619d200
Jan 26 00:46:10 rockpro64 kernel: [ 2493.902387] x3 : 0000004076cd6000 x2 : ffffff80087817fc
Jan 26 00:46:10 rockpro64 kernel: [ 2493.902943] x1 : ffffffc0760fae28 x0 : ffffffc07614ce68
Jan 26 00:46:10 rockpro64 kernel: [ 2493.903501]
Jan 26 00:46:10 rockpro64 kernel: [ 2493.903501] PC: 0xffffff80087817a4:
Jan 26 00:46:10 rockpro64 kernel: [ 2493.904009] 17a4  f9401822 b9402c01 f9400842 53196021 91040021 8b010041 91004023 b9400062
Jan 26 00:46:10 rockpro64 kernel: [ 2493.904908] 17c4  d5033d9f d5033e9f b9000062 97fffe99 52800020 11000673 17ffffe3 97ff993c
Jan 26 00:46:10 rockpro64 kernel: [ 2493.905806] 17e4  34ffff80 f9400e80 2a1303e2 aa1503e1 97fe1f14 17fffff7 a9bd7bfd 910003fd
Jan 26 00:46:10 rockpro64 kernel: [ 2493.906704] 1804  a90153f3 a9025bf5 aa0103f3 aa1e03e0 d503201f f9401a60 f9400814 91002294
Jan 26 00:46:10 rockpro64 kernel: [ 2493.907604] 1824  b9400295 d5033d9f 34000275 b9402016 aa1303e0 94147436 0a1602b6 aa1303e0
Jan 26 00:46:10 rockpro64 kernel: [ 2493.908501] 1844  2a1603e1 97ffffba 2a0003f6 d5033e9f b9000295 aa1303e0 94147455 710002df
Jan 26 00:46:10 rockpro64 kernel: [ 2493.909398] 1864  1a9f07e0 a94153f3 a9425bf5 a8c37bfd d65f03c0 52800000 17fffffb a9bd7bfd
Jan 26 00:46:10 rockpro64 kernel: [ 2493.910294] 1884  910003fd a90153f3 a9025bf5 aa0103f3 aa1e03e0 d503201f f9401a60 f9400814
Jan 26 00:46:10 rockpro64 kernel: [ 2493.911193]
Jan 26 00:46:10 rockpro64 kernel: [ 2493.911193] LR: 0xffffff800810c9a0:
Jan 26 00:46:10 rockpro64 kernel: [ 2493.911702] c9a0  2a1603e0 a94153f3 a9425bf5 a94363f7 a9446bf9 a94573fb a8c67bfd d65f03c0
Jan 26 00:46:10 rockpro64 kernel: [ 2493.912600] c9c0  b9400b00 7100001f 5400026d aa1903e0 f9448782 d538d081 b8616801 52800800
Jan 26 00:46:10 rockpro64 kernel: [ 2493.913497] c9e0  1ac00c20 937d7c00 f8606840 9ac12400 36000120 f9401314 b40000f4 a9400283
Jan 26 00:46:10 rockpro64 kernel: [ 2493.914396] ca00  aa1303e2 2a1503e1 d63f0060 f8418e80 b5ffff60 a9400662 2a1503e0 d63f0040
Jan 26 00:46:10 rockpro64 kernel: [ 2493.915295] ca20  2a0003fa b0009661 91104021 b9400820 7100001f 5400028d aa1903e0 f9448783
Jan 26 00:46:10 rockpro64 kernel: [ 2493.916192] ca40  d538d082 b8626802 52800800 1ac00c40 937d7c00 f8606860 9ac22400 36000140
Jan 26 00:46:10 rockpro64 kernel: [ 2493.917086] ca60  f9401034 b4000114 a9400284 2a1a03e3 aa1303e2 2a1503e1 d63f0080 f8418e80
Jan 26 00:46:10 rockpro64 kernel: [ 2493.917987] ca80  b5ffff40 d53b4220 373801a0 90009674 39659e80 35000120 f9400262 2a1503e1
Jan 26 00:46:10 rockpro64 kernel: [ 2493.918887]
Jan 26 00:46:10 rockpro64 kernel: [ 2493.918887] SP: 0xffffffc07ff01ba0:
Jan 26 00:46:10 rockpro64 kernel: [ 2493.919395] 1ba0  00000000 00000000 7619d200 ffffffc0 093d9438 ffffff80 09228000 ffffff80
Jan 26 00:46:10 rockpro64 kernel: [ 2493.920293] 1bc0  7ff02050 ffffffc0 00000000 00000000 08cbd000 ffffff80 7ff01c20 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2493.921190] 1be0  0810ca20 ffffff80 7ff01c20 ffffffc0 08781824 ffffff80 600001c5 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2493.922090] 1c00  7ff01c30 ffffffc0 080bbfb8 ffffff80 00000000 00000080 73b40f00 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2493.922989] 1c20  7ff01c50 ffffffc0 0810ca20 ffffff80 76165600 ffffffc0 7619d220 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2493.923886] 1c40  000000e9 00000000 00000000 00000000 7ff01cb0 ffffffc0 0810cb6c ffffff80
Jan 26 00:46:10 rockpro64 kernel: [ 2493.924787] 1c60  7619d200 ffffffc0 7619d220 ffffffc0 7619d2a0 ffffffc0 7619d2a0 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2493.925685] 1c80  00000001 00000000 7ff01dc0 ffffffc0 092ff960 ffffff80 7ff02050 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2493.926586]
Jan 26 00:46:10 rockpro64 kernel: [ 2493.926586] X0: 0xffffffc07614cde8:
Jan 26 00:46:10 rockpro64 kernel: [ 2493.933681] cde8  00000000 00000000 00000000 00000000 785eb108 ffffffc0 7629f5c0 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2493.941149] ce08  7614cd40 ffffffc0 00000001 00000000 787a3400 ffffffc0 093295c0 ffffff80
Jan 26 00:46:10 rockpro64 kernel: [ 2493.948584] ce28  00000000 00000000 00000000 00000000 00000000 00000000 760fae00 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2493.955997] ce48  76165300 ffffffc0 08711a90 ffffff80 090045b8 ffffff80 00000098 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2493.963417] ce68  00000000 00000000 00000000 00000000 096ce000 ffffff80 eb32ffa1 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2493.970825] ce88  00000003 eb32ffa1 00000000 00000003 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2493.978241] cea8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2493.985642] cec8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2493.992982]
Jan 26 00:46:10 rockpro64 kernel: [ 2493.992982] X1: 0xffffffc0760fada8:
Jan 26 00:46:10 rockpro64 kernel: [ 2494.006364] ada8  080cd650 ffffff80 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.013705] adc8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.020997] ade8  00000000 00000000 00000000 00000000 00000000 00000000 760ac800 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.028305] ae08  7614ce40 ffffffc0 087678cc ffffff80 0900dec4 ffffff80 000000b0 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.035622] ae28  b9beb9be dead4ead ffffffff 00000000 ffffffff ffffffff 76041898 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.042946] ae48  00000000 00000000 00000002 0000001f 7614ce68 ffffffc0 09329db8 ffffff80
Jan 26 00:46:10 rockpro64 kernel: [ 2494.050292] ae68  00000002 00000000 00000001 00000000 00000000 dead4ead ffffffff 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.057561] ae88  ffffffff ffffffff 760fae90 ffffffc0 760fae90 ffffffc0 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.064766]
Jan 26 00:46:10 rockpro64 kernel: [ 2494.064766] X2: 0xffffff800878177c:
Jan 26 00:46:10 rockpro64 kernel: [ 2494.077993] 177c  f9401bf7 a8c47bfd d65f03c0 1ad322e1 6a16003f 54000240 8b334e80 f9404c00
Jan 26 00:46:10 rockpro64 kernel: [ 2494.085192] 179c  b4000220 f959dc01 f9401822 b9402c01 f9400842 53196021 91040021 8b010041
Jan 26 00:46:10 rockpro64 kernel: [ 2494.092459] 17bc  91004023 b9400062 d5033d9f d5033e9f b9000062 97fffe99 52800020 11000673
Jan 26 00:46:10 rockpro64 kernel: [ 2494.099780] 17dc  17ffffe3 97ff993c 34ffff80 f9400e80 2a1303e2 aa1503e1 97fe1f14 17fffff7
Jan 26 00:46:10 rockpro64 kernel: [ 2494.107169] 17fc  a9bd7bfd 910003fd a90153f3 a9025bf5 aa0103f3 aa1e03e0 d503201f f9401a60
Jan 26 00:46:10 rockpro64 kernel: [ 2494.114613] 181c  f9400814 91002294 b9400295 d5033d9f 34000275 b9402016 aa1303e0 94147436
Jan 26 00:46:10 rockpro64 kernel: [ 2494.122121] 183c  0a1602b6 aa1303e0 2a1603e1 97ffffba 2a0003f6 d5033e9f b9000295 aa1303e0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.129667] 185c  94147455 710002df 1a9f07e0 a94153f3 a9425bf5 a8c37bfd d65f03c0 52800000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.137194]
Jan 26 00:46:10 rockpro64 kernel: [ 2494.137194] X4: 0xffffffc07619d180:
Jan 26 00:46:10 rockpro64 kernel: [ 2494.151033] d180  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.158569] d1a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.166013] d1c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.173360] d1e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.180620] d200  00441000 00000000 00000000 00000000 76165400 ffffffc0 00000002 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.187687] d220  00000000 000000e9 00080000 00000000 7619d200 ffffffc0 093001c0 ffffff80
Jan 26 00:46:10 rockpro64 kernel: [ 2494.194567] d240  78d84f00 ffffffc0 7616a800 ffffffc0 00000000 00000000 0923be64 ffffff80
Jan 26 00:46:10 rockpro64 kernel: [ 2494.201404] d260  081102e4 ffffff80 76165600 ffffffc0 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.208220]
Jan 26 00:46:10 rockpro64 kernel: [ 2494.208220] X6: 0xffffffc0788588a0:
Jan 26 00:46:10 rockpro64 kernel: [ 2494.220592] 88a0  7878c000 ffffffc0 7878c200 ffffffc0 7878c400 ffffffc0 7878c600 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.227424] 88c0  7878c800 ffffffc0 7878ca00 ffffffc0 7878cc00 ffffffc0 7878ce00 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.234188] 88e0  7878d000 ffffffc0 7878d200 ffffffc0 7878d400 ffffffc0 7878d600 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.240941] 8900  00000000 00000000 00000000 00000000 00000000 00000000 78858920 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.247722] 8920  00003001 00000038 78858248 ffffffc0 00000000 00000000 78858938 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.254503] 8940  78858938 ffffffc0 7878d800 ffffffc0 7878da00 ffffffc0 7878dc00 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.261314] 8960  7878de00 ffffffc0 7878e000 ffffffc0 7878e400 ffffffc0 7878e600 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.268102] 8980  7878e800 ffffffc0 7878ea00 ffffffc0 7878ec00 ffffffc0 7878ee00 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.274860]
Jan 26 00:46:10 rockpro64 kernel: [ 2494.274860] X7: 0xffffffc0788588c8:
Jan 26 00:46:10 rockpro64 kernel: [ 2494.287222] 88c8  7878ca00 ffffffc0 7878cc00 ffffffc0 7878ce00 ffffffc0 7878d000 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.294135] 88e8  7878d200 ffffffc0 7878d400 ffffffc0 7878d600 ffffffc0 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.301062] 8908  00000000 00000000 00000000 00000000 78858920 ffffffc0 00003001 00000038
Jan 26 00:46:10 rockpro64 kernel: [ 2494.308001] 8928  78858248 ffffffc0 00000000 00000000 78858938 ffffffc0 78858938 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.314971] 8948  7878d800 ffffffc0 7878da00 ffffffc0 7878dc00 ffffffc0 7878de00 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.321956] 8968  7878e000 ffffffc0 7878e400 ffffffc0 7878e600 ffffffc0 7878e800 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.328922] 8988  7878ea00 ffffffc0 7878ec00 ffffffc0 7878ee00 ffffffc0 7878f000 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.335871] 89a8  7878f200 ffffffc0 7878f400 ffffffc0 7878f600 ffffffc0 7878f800 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.342783]
Jan 26 00:46:10 rockpro64 kernel: [ 2494.342783] X8: 0xffffffc078858a10:
Jan 26 00:46:10 rockpro64 kernel: [ 2494.355384] 8a10  78790c00 ffffffc0 78790e00 ffffffc0 78791000 ffffffc0 78791200 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.362317] 8a30  78791400 ffffffc0 78791600 ffffffc0 78791800 ffffffc0 78791a00 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.369179] 8a50  78791c00 ffffffc0 78791e00 ffffffc0 78792000 ffffffc0 78792200 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.375983] 8a70  76752400 ffffffc0 76752600 ffffffc0 7608a000 ffffffc0 7608a400 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.382735] 8a90  7619d200 ffffffc0 76834e00 ffffffc0 76835000 ffffffc0 762f0800 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.389431] 8ab0  762f0a00 ffffffc0 762f0c00 ffffffc0 762f0e00 ffffffc0 762f1000 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.396071] 8ad0  762f1200 ffffffc0 762f1400 ffffffc0 762f1600 ffffffc0 762f1800 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.402666] 8af0  76369600 ffffffc0 76369c00 ffffffc0 76369e00 ffffffc0 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.409212]
Jan 26 00:46:10 rockpro64 kernel: [ 2494.409212] X9: 0xffffffc078eefe70:
Jan 26 00:46:10 rockpro64 kernel: [ 2494.421076] fe70  093725e8 ffffff80 02ca5244 00000000 00000000 00000000 78eefed0 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.427636] fe90  0894d9b0 ffffff80 78eefed0 ffffffc0 0894d9e4 ffffff80 60000145 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.434184] feb0  78eefed0 ffffffc0 0894d9b0 ffffff80 ffffffff ffffffff 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.440699] fed0  78eeff20 ffffffc0 0894dab0 ffffff80 76368400 ffffffc0 093725d0 ffffff80
Jan 26 00:46:10 rockpro64 kernel: [ 2494.447190] fef0  00000000 00000000 09265000 ffffff80 09265738 ffffff80 08cbd000 ffffff80
Jan 26 00:46:10 rockpro64 kernel: [ 2494.453644] ff10  03603000 00000000 03606000 00000000 78eeff50 ffffffc0 080fd910 ffffff80
Jan 26 00:46:10 rockpro64 kernel: [ 2494.460065] ff30  00000000 00000000 76368400 ffffffc0 093725d0 ffffff80 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.466461] ff50  78eeff80 ffffffc0 080fdbb4 ffffff80 76368400 ffffffc0 093725d0 ffffff80
Jan 26 00:46:10 rockpro64 kernel: [ 2494.472832]
Jan 26 00:46:10 rockpro64 kernel: [ 2494.472832] X19: 0xffffffc0760fada8:
Jan 26 00:46:10 rockpro64 kernel: [ 2494.484355] ada8  080cd650 ffffff80 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.490760] adc8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.497123] ade8  00000000 00000000 00000000 00000000 00000000 00000000 760ac800 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.503446] ae08  7614ce40 ffffffc0 087678cc ffffff80 0900dec4 ffffff80 000000b0 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.509813] ae28  b9beb9be dead4ead ffffffff 00000000 ffffffff ffffffff 76041898 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2494.516199] ae48  00000000 00000000 00000002 0000001f 7614ce68 ffffffc0 09329db8 ffffff80
Jan 26 00:46:10 rockpro64 kernel: [ 2494.522591] ae68  00000002 00000000 00000001 00000000 00000000 dead4ead ffffffff 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.528978] ae88  ffffffff ffffffff 760fae90 ffffffc0 760fae90 ffffffc0 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2494.535354]
Jan 26 00:46:10 rockpro64 kernel: [ 2494.535354] X20: 0xffffff80096cdf88:
Jan 26 00:46:10 rockpro64 kernel: [ 2494.546887] df88  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2494.553315] dfa8  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2494.559691] dfc8  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2494.566000] dfe8  ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2494.616704] Unhandled fault: synchronous external abort (0x96000210) at 0xffffff80096ce000
Jan 26 00:46:10 rockpro64 kernel: [ 2501.628391] BUG: spinlock lockup suspected on CPU#1, swapper/1/0
Jan 26 00:46:10 rockpro64 kernel: [ 2501.634647]  lock: 0xffffff800926fdf0, .magic: dead4ead, .owner: swapper/1/0, .owner_cpu: 1
Jan 26 00:46:10 rockpro64 kernel: [ 2501.641124] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.4.190-1233-rockchip-ayufan-gd3f1be0ed310 #1
Jan 26 00:46:10 rockpro64 kernel: [ 2501.647676] Hardware name: Pine64 RockPro64 (DT)
Jan 26 00:46:10 rockpro64 kernel: [ 2501.653863] Call trace:
Jan 26 00:46:10 rockpro64 kernel: [ 2501.659792] [<ffffff80080882b8>] dump_backtrace+0x0/0x220
Jan 26 00:46:10 rockpro64 kernel: [ 2501.666045] [<ffffff80080884fc>] show_stack+0x24/0x30
Jan 26 00:46:10 rockpro64 kernel: [ 2501.672277] [<ffffff80085a512c>] dump_stack+0x98/0xc0
Jan 26 00:46:10 rockpro64 kernel: [ 2501.678458] [<ffffff8008103fa0>] spin_dump+0x84/0xa4
Jan 26 00:46:10 rockpro64 kernel: [ 2501.684581] [<ffffff800810413c>] do_raw_spin_lock+0xdc/0x164
Jan 26 00:46:10 rockpro64 kernel: [ 2501.690758] [<ffffff8008c9e8d0>] _raw_spin_lock_irqsave+0x28/0x38
Jan 26 00:46:10 rockpro64 kernel: [ 2501.696972] [<ffffff800808853c>] die+0x34/0x1b4
Jan 26 00:46:10 rockpro64 kernel: [ 2501.703008] [<ffffff80080887b0>] arm64_notify_die+0x6c/0x70
Jan 26 00:46:10 rockpro64 kernel: [ 2501.709095] [<ffffff8008080b50>] do_mem_abort+0x9c/0xac
Jan 26 00:46:10 rockpro64 kernel: [ 2501.715164] Exception stack(0xffffffc07ff01730 to 0xffffffc07ff01860)
Jan 26 00:46:10 rockpro64 kernel: [ 2501.721335] 1720:                                   ffffffc078ee0e00 0000008000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2501.727654] 1740: ffffffc07ff01900 ffffff80085a33c4 0000000000000007 0000000000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2501.733958] 1760: ffffff80096ce000 0000000000000000 ffffff8009420314 0000000000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2501.740233] 1780: ffffffc07ff017a0 ffffff800810b150 0000000000000009 0000000000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2501.746497] 17a0: ffffffc07ff01840 ffffff800810b2b0 ffffffc07ff018c0 ffffff8008f88c73
Jan 26 00:46:10 rockpro64 kernel: [ 2501.752732] 17c0: ffffff8008f88c6c ffffff8008feec88 ffffffc07ff0198c ffffff80096ce000
Jan 26 00:46:10 rockpro64 kernel: [ 2501.758906] 17e0: 0000000000000004 0000000000000010 0000000000000000 ffffffc07ff01990
Jan 26 00:46:10 rockpro64 kernel: [ 2501.765069] 1800: ffffffc07ff0198c 0000000000000000 0000000000002800 0000000000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2501.771194] 1820: 0000000000000048 00000000ffffffff ffffffc078dcbeb0 ffffff800ee0f140
Jan 26 00:46:10 rockpro64 kernel: [ 2501.777363] 1840: ffffff800ee0f140 0000000000000000 0000000000000150 0000000000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2501.783511] [<ffffff80080825cc>] el1_da+0x24/0x84
Jan 26 00:46:10 rockpro64 kernel: [ 2501.789409] [<ffffff8008084924>] show_data.constprop.3+0xa4/0xf8
Jan 26 00:46:10 rockpro64 kernel: [ 2501.795478] [<ffffff8008084cb0>] __show_regs+0x154/0x1a8
Jan 26 00:46:10 rockpro64 kernel: [ 2501.801529] [<ffffff80080885c0>] die+0xb8/0x1b4
Jan 26 00:46:10 rockpro64 kernel: [ 2501.807501] [<ffffff80080887b0>] arm64_notify_die+0x6c/0x70
Jan 26 00:46:10 rockpro64 kernel: [ 2501.813594] [<ffffff8008080b50>] do_mem_abort+0x9c/0xac
Jan 26 00:46:10 rockpro64 kernel: [ 2501.819648] Exception stack(0xffffffc07ff01a50 to 0xffffffc07ff01b80)
Jan 26 00:46:10 rockpro64 kernel: [ 2501.825842] 1a40:                                   ffffffc0760fae28 0000008000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2501.832145] 1a60: ffffffc07ff01c20 ffffff8008781824 0000000000000007 ffffffc000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2501.838499] 1a80: ffffff80096ce008 ffffff80081d54a0 ffffffc078c01a00 ffffffbdc1cdb000
Jan 26 00:46:10 rockpro64 kernel: [ 2501.844806] 1aa0: 0000000000000001 0000000000000140 ffffffc0736c0200 0000000000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2501.851164] 1ac0: ffffffc0736c0200 0000000000000001 ffffff8009265000 0000000000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2501.857542] 1ae0: ffffffc07fef31c0 0000000100100007 ffffffc07614ce68 ffffffc0760fae28
Jan 26 00:46:10 rockpro64 kernel: [ 2501.863908] 1b00: ffffff80087817fc 0000004076cd6000 ffffffc07619d200 00000000fffffffa
Jan 26 00:46:10 rockpro64 kernel: [ 2501.870292] 1b20: ffffffc078858920 ffffffc078858948 ffffffc078858a90 ffffffc078eefef0
Jan 26 00:46:10 rockpro64 kernel: [ 2501.876677] 1b40: 0000000000000a30 0000000000000191 000000000000005e 0000000000000001
Jan 26 00:46:10 rockpro64 kernel: [ 2501.883126] 1b60: 0000000000000001 000000000000000e 0000000000000150 0000000000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2501.889577] [<ffffff80080825cc>] el1_da+0x24/0x84
Jan 26 00:46:10 rockpro64 kernel: [ 2501.895707] [<ffffff800810ca20>] handle_irq_event_percpu+0xf4/0x1f0
Jan 26 00:46:10 rockpro64 kernel: [ 2501.902024] [<ffffff800810cb6c>] handle_irq_event+0x50/0x80
Jan 26 00:46:10 rockpro64 kernel: [ 2501.908211] [<ffffff80081103b0>] handle_fasteoi_irq+0xcc/0x134
Jan 26 00:46:10 rockpro64 kernel: [ 2501.914432] [<ffffff800810bf1c>] generic_handle_irq+0x2c/0x44
Jan 26 00:46:10 rockpro64 kernel: [ 2501.920622] [<ffffff800810c298>] __handle_domain_irq+0xb4/0xb8
Jan 26 00:46:10 rockpro64 kernel: [ 2501.926815] [<ffffff8008080e80>] gic_handle_irq+0xc8/0x180
Jan 26 00:46:10 rockpro64 kernel: [ 2501.932969] Exception stack(0xffffffc07ff01d80 to 0xffffffc07ff01eb0)
Jan 26 00:46:10 rockpro64 kernel: [ 2501.939282] 1d80: ffffffc07ff01dc0 0000008000000000 ffffffc07ff01ef0 ffffff8008080fe8
Jan 26 00:46:10 rockpro64 kernel: [ 2501.945794] 1da0: 0000000040000145 ffffffc078eefda0 00000000071ec000 0000000000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2501.952366] 1dc0: 0000000000000040 ffffff800941ce80 0000004076cd6000 ffffff800941ce80
Jan 26 00:46:10 rockpro64 kernel: [ 2501.958896] 1de0: 0000004076cd6000 0000004076cd6000 ffffffc078858920 ffffffc078858948
Jan 26 00:46:10 rockpro64 kernel: [ 2501.965460] 1e00: ffffffc078858a90 ffffffc078eefef0 0000000000000a30 0000000000000191
Jan 26 00:46:10 rockpro64 kernel: [ 2501.972010] 1e20: 000000000000005e 0000000000000001 0000000000000001 000000000000000e
Jan 26 00:46:10 rockpro64 kernel: [ 2501.978595] 1e40: 0000000000000150 0000000000000000 0000000030d00800 ffffff8009228000
Jan 26 00:46:10 rockpro64 kernel: [ 2501.985198] 1e60: ffffff800922f000 0000000000000010 ffffff800941c000 0000000000000001
Jan 26 00:46:10 rockpro64 kernel: [ 2501.991829] 1e80: ffffffc078eefda0 ffffff80092ff960 ffffffc07ff02050 0000000002ca5244
Jan 26 00:46:10 rockpro64 kernel: [ 2501.998488] 1ea0: ffffffc078ee0e00 ffffffc07ff01ef0
Jan 26 00:46:10 rockpro64 kernel: [ 2502.004881] [<ffffff80080827b4>] el1_irq+0xb4/0x140
Jan 26 00:46:10 rockpro64 kernel: [ 2502.011314] [<ffffff80080bbf60>] irq_exit+0x8c/0xd8
Jan 26 00:46:10 rockpro64 kernel: [ 2502.017733] [<ffffff800810c268>] __handle_domain_irq+0x84/0xb8
Jan 26 00:46:10 rockpro64 kernel: [ 2502.024205] [<ffffff8008080e80>] gic_handle_irq+0xc8/0x180
Jan 26 00:46:10 rockpro64 kernel: [ 2502.030661] Exception stack(0xffffffc078eefda0 to 0xffffffc078eefed0)
Jan 26 00:46:10 rockpro64 kernel: [ 2502.037193] fda0: ffffffc07ff081c0 0000004076cd6000 ffffffc07ff081c0 ffffffc078eefeb0
Jan 26 00:46:10 rockpro64 kernel: [ 2502.043879] fdc0: 00ffffffffffffff 0000000e01d44d0f 000000000009cc4c 0000000000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2502.050591] fde0: ffffffc078ee1890 ffffffc078eefef0 0000000000000a30 0000000000000191
Jan 26 00:46:10 rockpro64 kernel: [ 2502.057308] fe00: 000000000000005e 0000000000000001 0000000000000001 000000000000000e
Jan 26 00:46:10 rockpro64 kernel: [ 2502.064034] fe20: 0000000000000150 0000000000000000 0000000030d00800 000002449ef00426
Jan 26 00:46:10 rockpro64 kernel: [ 2502.070746] fe40: 0000000000000000 ffffffc076368400 0000000000000000 ffffff80093da1f8
Jan 26 00:46:10 rockpro64 kernel: [ 2502.077448] fe60: 0000000000000000 000002449eec0abc ffffff80093725e8 0000000002ca5244
Jan 26 00:46:10 rockpro64 kernel: [ 2502.084157] fe80: 0000000000000000 ffffffc078eefed0 ffffff800894d9b0 ffffffc078eefed0
Jan 26 00:46:10 rockpro64 kernel: [ 2502.090825] fea0: ffffff800894d9e4 0000000060000145 ffffffc078eefed0 ffffff800894d9b0
Jan 26 00:46:10 rockpro64 kernel: [ 2502.097525] fec0: ffffffffffffffff 0000000000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2502.103898] [<ffffff80080827b4>] el1_irq+0xb4/0x140
Jan 26 00:46:10 rockpro64 kernel: [ 2502.110311] [<ffffff800894d9e4>] cpuidle_enter_state+0x1fc/0x254
Jan 26 00:46:10 rockpro64 kernel: [ 2502.116850] [<ffffff800894dab0>] cpuidle_enter+0x34/0x44
Jan 26 00:46:10 rockpro64 kernel: [ 2502.123280] [<ffffff80080fd910>] call_cpuidle+0x70/0x78
Jan 26 00:46:10 rockpro64 kernel: [ 2502.129654] [<ffffff80080fdbb4>] cpu_startup_entry+0x29c/0x2ac
Jan 26 00:46:10 rockpro64 kernel: [ 2502.136073] [<ffffff800808c5a8>] secondary_start_kernel+0x190/0x1bc
Jan 26 00:46:10 rockpro64 kernel: [ 2502.142525] [<0000000002ca5188>] 0x2ca5188
Jan 26 00:46:10 rockpro64 kernel: [ 2532.146200] Watchdog detected hard LOCKUP on cpu 1
Jan 26 00:46:10 rockpro64 kernel: [ 2532.146706] ------------[ cut here ]------------
Jan 26 00:46:10 rockpro64 kernel: [ 2532.158835] WARNING: at kernel/watchdog.c:352
Jan 26 00:46:10 rockpro64 kernel: [ 2532.165007] Modules linked in: midgard_kbase rockchip_saradc ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid0 mu                                                                   ltipath linear raid1 md_mod
Jan 26 00:46:10 rockpro64 kernel: [ 2532.172867]
Jan 26 00:46:10 rockpro64 kernel: [ 2532.179181] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.4.190-1233-rockchip-ayufan-gd3f1be0ed310 #1
Jan 26 00:46:10 rockpro64 kernel: [ 2532.186330] Hardware name: Pine64 RockPro64 (DT)
Jan 26 00:46:10 rockpro64 kernel: [ 2532.193041] task: ffffff800926df90 task.stack: ffffff8009260000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.199859] PC is at watchdog_timer_fn+0x190/0x2e8
Jan 26 00:46:10 rockpro64 kernel: [ 2532.206519] LR is at watchdog_timer_fn+0x190/0x2e8
Jan 26 00:46:10 rockpro64 kernel: [ 2532.213076] pc : [<ffffff8008155de4>] lr : [<ffffff8008155de4>] pstate: 600001c5
Jan 26 00:46:10 rockpro64 kernel: [ 2532.219875] sp : ffffffc07feecd00
Jan 26 00:46:10 rockpro64 kernel: [ 2532.226293] x29: ffffffc07feecd00 x28: 0000000000000001
Jan 26 00:46:10 rockpro64 kernel: [ 2532.232902] x27: ffffff80092678a0 x26: ffffff800922e238
Jan 26 00:46:10 rockpro64 kernel: [ 2532.239510] x25: ffffffc07feef250 x24: ffffff8009263d70
Jan 26 00:46:10 rockpro64 kernel: [ 2532.246120] x23: 0000000000000000 x22: 0000000000000932
Jan 26 00:46:10 rockpro64 kernel: [ 2532.252736] x21: ffffff8009266000 x20: ffffff800922e2b0
Jan 26 00:46:10 rockpro64 kernel: [ 2532.259351] x19: ffffff800922e000 x18: 0000000000000005
Jan 26 00:46:10 rockpro64 kernel: [ 2532.265947] x17: 0000000000000000 x16: 0000000000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.272518] x15: 0000000000000000 x14: ffffff800ee0eea0
Jan 26 00:46:10 rockpro64 kernel: [ 2532.279070] x13: ffffff800ee0eea0 x12: ffffffc078dca7a0
Jan 26 00:46:10 rockpro64 kernel: [ 2532.285614] x11: 00000000ffffffff x10: 00000000000001a8
Jan 26 00:46:10 rockpro64 kernel: [ 2532.292132] x9 : 0000000000000000 x8 : 0000000000002800
Jan 26 00:46:10 rockpro64 kernel: [ 2532.298595] x7 : 0000000000000000 x6 : 0000000000000001
Jan 26 00:46:10 rockpro64 kernel: [ 2532.305007] x5 : 0000000000000000 x4 : 0000000000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.311375] x3 : 0000004076cc1000 x2 : 0000004076cc1000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.317717] x1 : 0000000000000000 x0 : 0000000000000026
Jan 26 00:46:10 rockpro64 kernel: [ 2532.324038]
Jan 26 00:46:10 rockpro64 kernel: [ 2532.324038] PC: 0xffffff8008155d64:
Jan 26 00:46:10 rockpro64 kernel: [ 2532.336062] 5d64  91022694 d538d080 38746801 34000041 3834681f 97ffff7a a94153f3 52800020
Jan 26 00:46:10 rockpro64 kernel: [ 2532.342748] 5d84  a9425bf5 a94363f7 a9446bf9 a94573fb a8c77bfd d65f03c0 91020282 f87a6860
Jan 26 00:46:10 rockpro64 kernel: [ 2532.349438] 5da4  f8626865 91022294 eb05001f 54000101 38746860 35fffb80 b9403082 b0007200
Jan 26 00:46:10 rockpro64 kernel: [ 2532.356137] 5dc4  913c9c00 340000c2 9400dfdd f8226860 f87c7b60 3834681f 17ffffd3 9400e067
Jan 26 00:46:10 rockpro64 kernel: [ 2532.362835] 5de4  d4210000 f87c7b60 52800021 38346801 17ffffcd 97fe44f3 f9444ea1 36080941
Jan 26 00:46:10 rockpro64 kernel: [ 2532.369569] 5e04  b9401341 910003e2 340008e1 531f7821 d35efc00 8b21c2c1 cb000021 b6f80841
Jan 26 00:46:10 rockpro64 kernel: [ 2532.376333] 5e24  6b160016 54000800 91022a99 d538d080 38606b21 34000101 91024294 f8746802
Jan 26 00:46:10 rockpro64 kernel: [ 2532.383139] 5e44  d5384101 eb01005f 54fff980 3839681f 17ffffc9 34000157 d0009981 52800000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.389981]
Jan 26 00:46:10 rockpro64 kernel: [ 2532.389981] LR: 0xffffff8008155d64:
Jan 26 00:46:10 rockpro64 kernel: [ 2532.402435] 5d64  91022694 d538d080 38746801 34000041 3834681f 97ffff7a a94153f3 52800020
Jan 26 00:46:10 rockpro64 kernel: [ 2532.409302] 5d84  a9425bf5 a94363f7 a9446bf9 a94573fb a8c77bfd d65f03c0 91020282 f87a6860
Jan 26 00:46:10 rockpro64 kernel: [ 2532.416173] 5da4  f8626865 91022294 eb05001f 54000101 38746860 35fffb80 b9403082 b0007200
Jan 26 00:46:10 rockpro64 kernel: [ 2532.423067] 5dc4  913c9c00 340000c2 9400dfdd f8226860 f87c7b60 3834681f 17ffffd3 9400e067
Jan 26 00:46:10 rockpro64 kernel: [ 2532.429980] 5de4  d4210000 f87c7b60 52800021 38346801 17ffffcd 97fe44f3 f9444ea1 36080941
Jan 26 00:46:10 rockpro64 kernel: [ 2532.436939] 5e04  b9401341 910003e2 340008e1 531f7821 d35efc00 8b21c2c1 cb000021 b6f80841
Jan 26 00:46:10 rockpro64 kernel: [ 2532.443927] 5e24  6b160016 54000800 91022a99 d538d080 38606b21 34000101 91024294 f8746802
Jan 26 00:46:10 rockpro64 kernel: [ 2532.450941] 5e44  d5384101 eb01005f 54fff980 3839681f 17ffffc9 34000157 d0009981 52800000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.457974]
Jan 26 00:46:10 rockpro64 kernel: [ 2532.457974] SP: 0xffffffc07feecc80:
Jan 26 00:46:10 rockpro64 kernel: [ 2532.470766] cc80  00000932 00000000 00000000 00000000 09263d70 ffffff80 7feef250 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2532.477813] cca0  0922e238 ffffff80 092678a0 ffffff80 00000001 00000000 7feecd00 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2532.484872] ccc0  08155de4 ffffff80 7feecd00 ffffffc0 08155de4 ffffff80 600001c5 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.491960] cce0  09266898 ffffff80 0000026f 00000000 00000000 00000080 78800028 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2532.499078] cd00  7feecd70 ffffffc0 08120750 ffffff80 7feef250 ffffffc0 7feeec00 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2532.506230] cd20  8a4d27ed 0000024d 7feeec80 ffffffc0 09228000 ffffff80 00000003 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.513404] cd40  093d9cd0 ffffff80 8a4d27ed 0000024d 08155c54 ffffff80 8a4d2117 0000024d
Jan 26 00:46:10 rockpro64 kernel: [ 2532.520605] cd60  7feef250 ffffffc0 09266898 ffffff80 7feecde0 ffffffc0 08120d34 ffffff80
Jan 26 00:46:10 rockpro64 kernel: [ 2532.527809]
Jan 26 00:46:10 rockpro64 kernel: [ 2532.527809] X12: 0xffffffc078dca720:
Jan 26 00:46:10 rockpro64 kernel: [ 2532.540907] a720  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.548115] a740  00000000 00000000 00000000 00000000 00000000 c0c67c00 7cc6c0c0 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.555282] a760  c67c0000 067ec6c6 780c0606 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.562415] a780  7c7c3838 60c6c67c 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.569526] a7a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.576628] a7c0  00000000 00000000 00000000 00000000 6c380000 d6d6c6c6 386cc6c6 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.583702] a7e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.590761] a800  06600030 06006006 c630c606 cc000c18 00c6c6c6 06c00030 0600c006 c630c606
Jan 26 00:46:10 rockpro64 kernel: [ 2532.597819]
Jan 26 00:46:10 rockpro64 kernel: [ 2532.597819] X13: 0xffffff800ee0ee20:
Jan 26 00:46:10 rockpro64 kernel: [ 2532.610728] ee20  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.617869] ee40  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.624968] ee60  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.632002] ee80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.638960] eea0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.645853] eec0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.652699] eee0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.659496] ef00  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.666271]
Jan 26 00:46:10 rockpro64 kernel: [ 2532.666271] X14: 0xffffff800ee0ee20:
Jan 26 00:46:10 rockpro64 kernel: [ 2532.678584] ee20  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.685406] ee40  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.692167] ee60  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.698873] ee80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.705531] eea0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.712150] eec0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.718747] eee0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.725313] ef00  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.731847]
Jan 26 00:46:10 rockpro64 kernel: [ 2532.731847] X19: 0xffffff800922df80:
Jan 26 00:46:10 rockpro64 kernel: [ 2532.743674] df80  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2532.750263] dfa0  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2532.756805] dfc0  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2532.763305] dfe0  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2532.769588] e000  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2532.775658] e020  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2532.781634] e040  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2532.787508] e060  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2532.793318]
Jan 26 00:46:10 rockpro64 kernel: [ 2532.793318] X20: 0xffffff800922e230:
Jan 26 00:46:10 rockpro64 kernel: [ 2532.803737] e230  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2532.809606] e250  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2532.815403] e270  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2532.821172] e290  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2532.826908] e2b0  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2532.832625] e2d0  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2532.838304] e2f0  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2532.843949] e310  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2532.849575]
Jan 26 00:46:10 rockpro64 kernel: [ 2532.849575] X21: 0xffffff8009265f80:
Jan 26 00:46:10 rockpro64 kernel: [ 2532.859678] 5f80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.865430] 5fa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.871143] 5fc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.876837] 5fe0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.882520] 6000  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.888205] 6020  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.893869] 6040  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.899512] 6060  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.905109]
Jan 26 00:46:10 rockpro64 kernel: [ 2532.905109] X24: 0xffffff8009263cf0:
Jan 26 00:46:10 rockpro64 kernel: [ 2532.915087] 3cf0  00000000 00000000 0926df90 ffffff80 000001c0 00000000 089754b8 ffffff80
Jan 26 00:46:10 rockpro64 kernel: [ 2532.920689] 3d10  00000002 00000000 080911f8 ffffff80 09263d30 ffffff80 080912ec ffffff80
Jan 26 00:46:10 rockpro64 kernel: [ 2532.926252] 3d30  09263d70 ffffff80 089f49e4 ffffff80 78122a10 ffffffc0 78122610 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2532.931793] 3d50  78122800 ffffffc0 00000002 00000000 09273298 ffffff80 085d3180 ffffff80
Jan 26 00:46:10 rockpro64 kernel: [ 2532.937310] 3d70  00000000 00000000 00000000 00000000 76cc1000 00000040 00000001 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.942809] 3d90  ffffffff 00ffffff 38a11c0f 0000000e 000a1294 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.948277] 3db0  b5593519 00000032 08081800 ffffff80 00001000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.953713] 3dd0  34d5d91d 00000000 76cc1000 00000040 0926df90 ffffff80 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.959112]
Jan 26 00:46:10 rockpro64 kernel: [ 2532.959112] X25: 0xffffffc07feef1d0:
Jan 26 00:46:10 rockpro64 kernel: [ 2532.968693] f1d0  ffffffff 00000000 ffffffff ffffffff 00000001 00000000 7feef1e8 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2532.974093] f1f0  7feef1e8 ffffffc0 7feef1f8 ffffffc0 7feef1f8 ffffffc0 08149c48 ffffff80
Jan 26 00:46:10 rockpro64 kernel: [ 2532.979478] f210  78e97cf8 ffffffc0 78e97c50 ffffffc0 00020002 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2532.984841] f230  00000001 00000000 00000279 00000000 00000278 00000000 78ee0000 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2532.990187] f250  7feef250 ffffffc0 00000000 00000000 00000000 00000000 8a4c6600 0000024d
Jan 26 00:46:10 rockpro64 kernel: [ 2532.995511] f270  8a4c6600 0000024d 08155c54 ffffff80 7feeec80 ffffffc0 00000000 0000000a
Jan 26 00:46:10 rockpro64 kernel: [ 2533.000824] f290  08155c28 ffffff80 63746177 676f6468 0000302f 00000000 00000276 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2533.006133] f2b0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2533.011409]
Jan 26 00:46:10 rockpro64 kernel: [ 2533.011409] X26: 0xffffff800922e1b8:
Jan 26 00:46:10 rockpro64 kernel: [ 2533.020743] e1b8  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2533.026047] e1d8  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2533.031294] e1f8  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2533.036476] e218  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2533.041601] e238  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2533.046662] e258  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2533.051655] e278  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2533.056589] e298  ******** ******** ******** ******** ******** ******** ******** ********
Jan 26 00:46:10 rockpro64 kernel: [ 2533.061437]
Jan 26 00:46:10 rockpro64 kernel: [ 2533.061437] X27: 0xffffff8009267820:
Jan 26 00:46:10 rockpro64 kernel: [ 2533.069918] 7820  00000001 00000000 00000001 00000000 000000fa 00000000 7ff67b80 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2533.074836] 7840  00000015 00000000 7ff67b40 ffffffc0 00000001 00000006 00000060 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2533.079766] 7860  00015000 00001000 00000001 00000000 7ff67bc0 ffffffc0 7ff67c00 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2533.084731] 7880  00000000 00000005 00000010 00000000 7ff67a40 ffffffc0 7fee9000 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2533.089696] 78a0  76cc1000 00000040 76cd6000 00000040 76ceb000 00000040 76d00000 00000040
Jan 26 00:46:10 rockpro64 kernel: [ 2533.094686] 78c0  76d15000 00000040 76d2a000 00000040 00000000 00000000 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2533.099671] 78e0  00000002 00000000 0000341b 00000000 00010000 00000000 0007ffff 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2533.104630] 7900  0000eecf 00000000 00002000 00000000 00000000 0000fffa 00000000 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2533.109530]
Jan 26 00:46:10 rockpro64 kernel: [ 2533.109530] X29: 0xffffffc07feecc80:
Jan 26 00:46:10 rockpro64 kernel: [ 2533.118107] cc80  00000932 00000000 00000000 00000000 09263d70 ffffff80 7feef250 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2533.123088] cca0  0922e238 ffffff80 092678a0 ffffff80 00000001 00000000 7feecd00 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2533.128128] ccc0  08155de4 ffffff80 7feecd00 ffffffc0 08155de4 ffffff80 600001c5 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2533.133224] cce0  09266898 ffffff80 0000026f 00000000 00000000 00000080 78800028 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2533.138387] cd00  7feecd70 ffffffc0 08120750 ffffff80 7feef250 ffffffc0 7feeec00 ffffffc0
Jan 26 00:46:10 rockpro64 kernel: [ 2533.143604] cd20  8a4d27ed 0000024d 7feeec80 ffffffc0 09228000 ffffff80 00000003 00000000
Jan 26 00:46:10 rockpro64 kernel: [ 2533.148882] cd40  093d9cd0 ffffff80 8a4d27ed 0000024d 08155c54 ffffff80 8a4d2117 0000024d
Jan 26 00:46:10 rockpro64 kernel: [ 2533.154221] cd60  7feef250 ffffffc0 09266898 ffffff80 7feecde0 ffffffc0 08120d34 ffffff80
Jan 26 00:46:10 rockpro64 kernel: [ 2533.159539]
Jan 26 00:46:10 rockpro64 kernel: [ 2533.164183] ---[ end trace e1cc157d40f32fa7 ]---
Jan 26 00:46:10 rockpro64 kernel: [ 2533.169098] Call trace:
Jan 26 00:46:10 rockpro64 kernel: [ 2533.173835] Exception stack(0xffffffc07feecb20 to 0xffffffc07feecc50)
Jan 26 00:46:10 rockpro64 kernel: [ 2533.179040] cb20: ffffff800922e000 0000008000000000 ffffffc07feecd00 ffffff8008155de4
Jan 26 00:46:10 rockpro64 kernel: [ 2533.184454] cb40: 00000000600001c5 ffffff800810b150 0000000000000026 0000000000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2533.189940] cb60: ffffffc07feecc00 ffffff800810b2b0 ffffffc07feecc80 ffffff8008f96f27
Jan 26 00:46:10 rockpro64 kernel: [ 2533.195486] cb80: ffffff8009266000 0000000000000932 0000000000000000 ffffff8009263d70
Jan 26 00:46:10 rockpro64 kernel: [ 2533.201086] cba0: ffffffc07feef250 ffffff800922e238 ffffff80092678a0 0000000000000001
Jan 26 00:46:10 rockpro64 kernel: [ 2533.206748] cbc0: 0000000000000000 ffffff8009228000 0000000000000026 0000000000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2533.212367] cbe0: 0000004076cc1000 0000004076cc1000 0000000000000000 0000000000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2533.217992] cc00: 0000000000000001 0000000000000000 0000000000002800 0000000000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2533.223553] cc20: 00000000000001a8 00000000ffffffff ffffffc078dca7a0 ffffff800ee0eea0
Jan 26 00:46:10 rockpro64 kernel: [ 2533.229161] cc40: ffffff800ee0eea0 0000000000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2533.234541] [<ffffff8008155de4>] watchdog_timer_fn+0x190/0x2e8
Jan 26 00:46:10 rockpro64 kernel: [ 2533.240078] [<ffffff8008120750>] __hrtimer_run_queues+0x1bc/0x284
Jan 26 00:46:10 rockpro64 kernel: [ 2533.245679] [<ffffff8008120d34>] hrtimer_interrupt+0xac/0x1bc
Jan 26 00:46:10 rockpro64 kernel: [ 2533.251288] [<ffffff800812c7ec>] tick_handle_oneshot_broadcast+0x144/0x1c0
Jan 26 00:46:10 rockpro64 kernel: [ 2533.257067] [<ffffff800897659c>] rk_timer_interrupt+0x4c/0x5c
Jan 26 00:46:10 rockpro64 kernel: [ 2533.262785] [<ffffff800810ca20>] handle_irq_event_percpu+0xf4/0x1f0
Jan 26 00:46:10 rockpro64 kernel: [ 2533.268515] [<ffffff800810cb6c>] handle_irq_event+0x50/0x80
Jan 26 00:46:10 rockpro64 kernel: [ 2533.274178] [<ffffff80081103b0>] handle_fasteoi_irq+0xcc/0x134
Jan 26 00:46:10 rockpro64 kernel: [ 2533.279815] [<ffffff800810bf1c>] generic_handle_irq+0x2c/0x44
Jan 26 00:46:10 rockpro64 kernel: [ 2533.285467] [<ffffff800810c298>] __handle_domain_irq+0xb4/0xb8
Jan 26 00:46:10 rockpro64 kernel: [ 2533.291169] [<ffffff8008080e80>] gic_handle_irq+0xc8/0x180
Jan 26 00:46:10 rockpro64 kernel: [ 2533.296871] Exception stack(0xffffff8009263d70 to 0xffffff8009263ea0)
Jan 26 00:46:10 rockpro64 kernel: [ 2533.302738] 3d60:                                   0000000000000000 0000000000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2533.308831] 3d80: 0000004076cc1000 0000000000000001 00ffffffffffffff 0000000e38a11c0f
Jan 26 00:46:10 rockpro64 kernel: [ 2533.314991] 3da0: 00000000000a1294 0000000000000000 00000032b5593519 ffffff8008081800
Jan 26 00:46:10 rockpro64 kernel: [ 2533.321196] 3dc0: 0000000000001000 0000000000000000 0000000034d5d91d 0000004076cc1000
Jan 26 00:46:10 rockpro64 kernel: [ 2533.327351] 3de0: ffffff800926df90 0000000000000000 0000000000000000 0000000000000000
Jan 26 00:46:10 rockpro64 kernel: [ 2533.333521] 3e00: 0000000030d00800 0000024d8a4d148f 0000000000000002 ffffffc076368200
Jan 26 00:46:10 rockpro64 kernel: [ 2533.339655] 3e20: 0000000000000002 ffffff80093da1f8 0000000000000001 0000024d8a104604
Jan 26 00:46:10 rockpro64 kernel: [ 2533.345839] 3e40: ffffff80093726a8 0000000002ca523c 000000000313001c ffffff8009263ea0
Jan 26 00:46:10 rockpro64 kernel: [ 2533.352086] 3e60: ffffff800894d9e0 ffffff8009263ea0 ffffff800894d9e4 0000000060000145
Jan 26 00:46:10 rockpro64 kernel: [ 2533.358399] 3e80: ffffff8009263ea0 ffffff800894d9e0 ffffffffffffffff 0000000000000002
Jan 26 00:46:10 rockpro64 kernel: [ 2533.364743] [<ffffff80080827b4>] el1_irq+0xb4/0x140
Jan 26 00:46:10 rockpro64 kernel: [ 2533.370876] [<ffffff800894d9e4>] cpuidle_enter_state+0x1fc/0x254
Jan 26 00:46:10 rockpro64 kernel: [ 2533.377161] [<ffffff800894dab0>] cpuidle_enter+0x34/0x44
Jan 26 00:46:10 rockpro64 kernel: [ 2533.383403] [<ffffff80080fd910>] call_cpuidle+0x70/0x78
Jan 26 00:46:10 rockpro64 kernel: [ 2533.389581] [<ffffff80080fdbb4>] cpu_startup_entry+0x29c/0x2ac
Jan 26 00:46:10 rockpro64 kernel: [ 2533.395835] [<ffffff8008c9a444>] rest_init+0x74/0x7c
Jan 26 00:46:10 rockpro64 kernel: [ 2533.401953] [<ffffff8009130bc4>] start_kernel+0x3d0/0x3f4
Jan 26 00:46:10 rockpro64 kernel: [ 2533.408113] [<ffffff80091301c4>] __primary_switched+0x30/0x6c

Message from syslogd@rockpro64 at Jan 26 00:47:57 ...
 kernel:[ 2640.468684] Kernel panic - not syncing: hung_task: blocked tasks
Jan 26 00:47:57 rockpro64 kernel: [ 2640.386638] INFO: task md0_resync:7560 blocked for more than 120 seconds.
Jan 26 00:47:57 rockpro64 kernel: [ 2640.393096]       Tainted: G        W       4.4.190-1233-rockchip-ayufan-gd3f1be0ed310 #1
Jan 26 00:47:57 rockpro64 kernel: [ 2640.399647] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 26 00:47:57 rockpro64 kernel: [ 2640.406172] md0_resync      D ffffff8008085004     0  7560      2 0x00000010
Jan 26 00:47:57 rockpro64 kernel: [ 2640.412684] Call trace:
Jan 26 00:47:57 rockpro64 kernel: [ 2640.418730] [<ffffff8008085004>] __switch_to+0x9c/0xc8
Jan 26 00:47:57 rockpro64 kernel: [ 2640.425003] [<ffffff8008c9b79c>] __schedule+0x5a8/0x64c
Jan 26 00:47:57 rockpro64 kernel: [ 2640.431285] [<ffffff8008c9b8a8>] schedule+0x68/0x7c
Jan 26 00:47:57 rockpro64 kernel: [ 2640.437487] [<ffffff8000ce09fc>] sync_request+0x2e0/0x8ac [raid1]
Jan 26 00:47:57 rockpro64 kernel: [ 2640.443826] [<ffffff8000cb7678>] md_do_sync+0x678/0xb98 [md_mod]
Jan 26 00:47:57 rockpro64 kernel: [ 2640.450143] [<ffffff8000cb4ae8>] md_thread+0x138/0x13c [md_mod]
Jan 26 00:47:57 rockpro64 kernel: [ 2640.456396] [<ffffff80080d4850>] kthread+0xe0/0xf0
Jan 26 00:47:57 rockpro64 kernel: [ 2640.462528] [<ffffff8008082ef0>] ret_from_fork+0x10/0x20
Jan 26 00:47:57 rockpro64 kernel: [ 2640.468684] Kernel panic - not syncing: hung_task: blocked tasks

I haven't really experienced any kernel panics this consistent before, but considering this only happens when I use the adapter and because this line is mentioned in the logs: Jan 26 00:46:10 rockpro64 kernel: [ 2493.892967] PC is at ahci_single_level_irq_intr+0x28/0x84, I'm almost certain it's caused by the PCIe-adapter. In my stress test without the adapter plugged in, the board was on for about 2 hours without any issues.

[abdel-unxp]

One more thing: This uboot fix doesnt work on MrFixit debian OS. I think this is because it uses extlinux. I dont know about openmediavault. If you have time to check one last thing, you still can try to flash a manjaro ARM 19.12 (on a SD card if you want to keep your eMMC module) and apply the uboot fix. If you do so, be sure to load the linux kernel from uboot's SD card and not from the one in eMMC. (you can unplug your eMMC module to be sure) This way we can say you are using the same FW as me on same hardware.

[chippey5]

Alright, I now have a minimal manjaro image up and running on the board. As it only contains 1 partition, (correct me if this is wrong) I replaced the blobs u-boot.img and idbloader.img (not u-boot.itb by mistake).

I initiated raid1 and let mdadm do its thing for more than two hours and no oops as of yet, which is fantastic! Now I didn't actually try to lure out a kernel oops before replacing u-boot and idbloader, but I'll check and see if I can somehow use this on OMV on my eMMC. I'll keep you posted. Thanks @abdel-unxp

[ayufan]

Interesting. So, using different boot loader does help?

On Sun, 26 Jan 2020 at 18:52, Simon notifications@github.com wrote:

Alright, I now have a minimal manjaro image up and running on the board. As it only contains 1 partition, (correct me if this is wrong) I replaced the blobs u-boot.img and idbloader.img (not u-boot.itb by mistake).

I initiated raid1 and let mdadm do its thing for more than two hours and no oops as of yet, which is fantastic! Now I didn't actually try to lure out a kernel oops before replacing u-boot and idbloader, but I'll check and see if I can somehow use this on OMV on my eMMC. I'll keep you posted. Thanks @abdel-unxp https://github.com/abdel-unxp

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ayufan-rock64/linux-build/issues/299?email_source=notifications&email_token=AASOSQPW7ELE47GN7FAZA5LQ7XEW3A5CNFSM4GBUZLP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJ5ZXQY#issuecomment-578526147, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASOSQJ4SFHRVKV7BWVEJUDQ7XEW3ANCNFSM4GBUZLPQ .

[chippey5]

Could be. I'm going to flash the stock u-boot.itb blob from the manjaro image onto OMV and see if that helps, following the instruction previously mentioned by @abdel-unxp. I'll report back after testing.

[abdel-unxp]

As it only contains 1 partition, (correct me if this is wrong)

correct

[chippey5]

Unfortunately it's a bust, the kernel keeps panicking even when using the same U-boot as Manjaro, same oops as before:

kernel:[ 2240.945071] Internal error: : 96000210 [#1] SMP
Jan 26 21:05:13 rockpro64 kernel: [ 2240.943829] Unhandled fault: synchronous external abort (0x96000210) at 0xffffff80096ce1b0
Jan 26 21:05:13 rockpro64 kernel: [ 2240.945071] Internal error: : 96000210 [#1] SMP
Jan 26 21:05:13 rockpro64 kernel: [ 2240.945732] Modules linked in: midgard_kbase rockchip_saradc ip_tables x_tables autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx raid0 multipath linear                                                                                                raid1 md_mod
Jan 26 21:05:13 rockpro64 kernel: [ 2240.948631] CPU: 2 PID: 144 Comm: scsi_eh_1 Not tainted 4.4.190-1233-rockchip-ayufan-gd3f1be0ed310 #1

I'm open for suggestions, can't really get my head around this one.

[chippey5]

I found the setup working stable for 3 days and counting on the DietPi testing image (buster) for anyone that is encountering the same issue.

[mbainter]

FYI - I have the same configuration and same problem as @jwindhager reported. I have the u-boot-flash-spi-rockpro64.img in place, and am booting from USB instead of an SD card, but the hardware setup is similar (just different hard drives), and I'm running stretch-openmediavault-rockpro64-0.9.14-1159-armhf. I haven't yet tried swapping out for the idloader, but I'll try to do that early next week.

[hemertje]

+1, same hardware OMV5 runs at Armbian

Crashes with Creating a File System:

 ____            _    ____               __   _  _   
|  _ \ ___   ___| | _|  _ \ _ __ ___    / /_ | || |  
| |_) / _ \ / __| |/ / |_) | '__/ _ \  | '_ \| || |_ 
|  _ < (_) | (__|   <|  __/| | | (_) | | (_) |__   _|
|_| \_\___/ \___|_|\_\_|   |_|  \___/   \___/   |_|  

Welcome to Armbian buster with Linux 4.4.210-rockchip64

System load:   2.13 0.66 0.23   Up time:       0 min        
Memory usage:  10 % of 3874MB   IP:            192.168.2.3
CPU temp:      51°C             
Usage of /:    17% of 15G       

Last login: Sun Feb  2 19:08:55 2020 from 192.168.2.106

root@NAS-rockpro64:~# mkfs.ext4 /dev/sda
mke2fs 1.44.5 (15-Dec-2018)
Creating filesystem with 732566646 4k blocks and 183148544 inodes
Filesystem UUID: 21668c66-fbee-4fd2-8e1f-2127a8c48970
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
    102400000, 214990848, 512000000, 550731776, 644972544

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (262144 blocks): 
Message from syslogd@NAS-rockpro64 at Feb  3 18:04:10 ...
 kernel:[  121.843550] Internal error: : 96000210 [#1] SMP

Message from syslogd@NAS-rockpro64 at Feb  3 18:04:10 ...
 kernel:[  129.131405] BUG: spinlock lockup suspected on CPU#0, scsi_eh_0/274

Message from syslogd@NAS-rockpro64 at Feb  3 18:04:10 ...
 kernel:[  129.135057]  lock: 0xffffff8009321870, .magic: dead4ead, .owner: scsi_eh_0/274, .owner_cpu: 0

[chippey5]

@hemertje Can you check if you get the same results on DietPi?

[hemertje]

OK, Stable or Testing image?

[chippey5]

Testing, as linked.

[hemertje]

OK, Wil try

DietPi_RockPro64-ARMv8-Buster.7z | 2020-01-22 01:31 | 98M

Can it run OMV? Do you have a link to a Howto?

[chippey5]

OK, Wil try

DietPi_RockPro64-ARMv8-Buster.7z | 2020-01-22 01:31 | 98M

Can it run OMV? Do you have a link to a Howto?

Here's how to get started: https://forum.openmediavault.org/index.php/Thread/25062-Install-OMV5-on-Debian-10-Buster/

Try formatting the disk(s) first just to test and please share the results.

[hemertje]

At the OMV forum https://forum.openmediavault.org/index.php/Thread/30666-Create-filesystem-error-communication-failure/

ryecoaaron told me:

ayufan is the expert. He suggests too small of a power supply at beginning. Probably an issue with the kernel. I'm running Bionic on mine and have no issues. I would try one of the armbian images (those come from ayufan's work too) - dl.armbian.com/rockpro64/archive/. I have had better luck with the 4.x kernels over the 5.x kernels.

[chippey5]

Well, I've tried Armbian, Ayufan's OMV and Manjaro and all of those just kernel panic for me. The only one that has worked so far is DietPi. Others might get other results, but that seems doubtable as long as the identical hardware is used.

root@DietPi:~# uptime
 21:20:11 up 4 days,  3:29,  1 user,  load average: 0.07, 0.03, 0.01
root@DietPi:~# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb[1] sda[0]
      3906886464 blocks super 1.2 [2/2] [UU]
      bitmap: 0/30 pages [0KB], 65536KB chunk

unused devices: <none>

[hemertje]

https://dietpi.com/phpbb/viewtopic.php?p=5#p5 Open Media Vault - NOT available anymore due to incompatibilities!

Installed Dietpi as minimal with OMV as stated here https://forum.openmediavault.org/index.php/Thread/25062-Install-OMV5-on-Debian-10-Buster/

Errors were encountered while processing: openssh-server openmediavault E: Sub-process /usr/bin/dpkg returned an error code (1)

root@DietPi:~# cat /etc/issue Armbian 20.02.0-rc0 Buster \l

[hemertje]

on the WebIf

Welcome to nginx!

If you see this page, the nginx web server is successfully installed and working. Further configuration is required.

For online documentation and support please refer to nginx.org. Commercial support is available at nginx.com.

Thank you for using nginx.

[hemertje]

Feb 03 21:12:00 DietPi systemd[1]: Starting OpenBSD Secure Shell server... Feb 03 21:12:00 DietPi sshd[20297]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use. Feb 03 21:12:00 DietPi sshd[20297]: error: Bind to port 22 on :: failed: Address already in use. Feb 03 21:12:00 DietPi sshd[20297]: fatal: Cannot bind any address. Feb 03 21:12:00 DietPi systemd[1]: ssh.service: Main process exited, code=exited, status=255/EXCEPTION Feb 03 21:12:00 DietPi systemd[1]: ssh.service: Failed with result 'exit-code'. Feb 03 21:12:00 DietPi systemd[1]: Failed to start OpenBSD Secure Shell server. dpkg: error processing package openssh-server (--configure): installed openssh-server package post-installation script subprocess returned error exit status 1 Setting up php7.3-fpm (7.3.11-1~deb10u1) ...

Creating config file /etc/php/7.3/fpm/php.ini with new version Created symlink /etc/systemd/system/multi-user.target.wants/php7.3-fpm.service → /lib/systemd/system/php7.3-fpm.service. Setting up php-pam (1.0.3-1+deb10u1) ... Setting up collectd (5.8.1-1.3) ... dpkg: dependency problems prevent configuration of openmediavault: openmediavault depends on openssh-server; however: Package openssh-server is not configured yet.

[hemertje]

dpkg: error processing package openmediavault (--configure): dependency problems - leaving unconfigured Setting up rrdcached (1.7.1-2) ... Setting up libnginx-mod-http-image-filter (1.14.2-2+deb10u1) ... Setting up libjavascript-minifier-xs-perl (0.11-1+b5) ... Setting up libbind9-161:arm64 (1:9.11.5.P4+dfsg-5.1) ... Setting up samba-libs:arm64 (2:4.9.5+dfsg-5+deb10u1) ... Setting up php-yaml (2.0.2+1.3.1-4) ... Setting up nginx-full (1.14.2-2+deb10u1) ... Setting up salt-minion (2019.2.2+ds-1) ... Created symlink /etc/systemd/system/multi-user.target.wants/salt-minion.service → /lib/systemd/system/salt-minion.service. Setting up liblocale-po-perl (0.27-2) ... Setting up php-fpm (2:7.3+69) ... Setting up rrdtool (1.7.1-2) ... Setting up bind9-host (1:9.11.5.P4+dfsg-5.1) ... Setting up php-cgi (2:7.3+69) ... Setting up python-samba (2:4.9.5+dfsg-5+deb10u1) ... Setting up nginx (1.14.2-2+deb10u1) ... Setting up samba-vfs-modules:arm64 (2:4.9.5+dfsg-5+deb10u1) ... Setting up samba-common-bin (2:4.9.5+dfsg-5+deb10u1) ... Checking smb.conf with testparm Load smb config files from /etc/samba/smb.conf Loaded services file OK. Server role: ROLE_STANDALONE

Done Setting up avahi-daemon (0.7-4+b1) ... Created symlink /etc/systemd/system/dbus-org.freedesktop.Avahi.service → /lib/systemd/system/avahi-daemon.service. Created symlink /etc/systemd/system/multi-user.target.wants/avahi-daemon.service → /lib/systemd/system/avahi-daemon.service. Created symlink /etc/systemd/system/sockets.target.wants/avahi-daemon.socket → /lib/systemd/system/avahi-daemon.socket. Setting up libnss-mdns:arm64 (0.14.1-1) ... First installation detected... Checking NSS setup... Setting up samba (2:4.9.5+dfsg-5+deb10u1) ... Adding group `sambashare' (GID 120) ... Done. Samba is not being run as an AD Domain Controller: Masking samba-ad-dc.service Please ignore the following error about deb-systemd-helper not finding those services. (samba-ad-dc.service masked) Created symlink /etc/systemd/system/multi-user.target.wants/nmbd.service → /lib/systemd/system/nmbd.service. Failed to preset unit: Unit file /etc/systemd/system/samba-ad-dc.service is masked. /usr/bin/deb-systemd-helper: error: systemctl preset failed on samba-ad-dc.service: No such file or directory Created symlink /etc/systemd/system/multi-user.target.wants/smbd.service → /lib/systemd/system/smbd.service. Setting up wsdd (0.5-1) ... wsdd.service is a disabled or a static unit, not starting it. Setting up liblvm2cmd2.03:arm64 (2.03.02-3) ... Setting up dmeventd (2:1.02.155-3) ... Created symlink /etc/systemd/system/sockets.target.wants/dm-event.socket → /lib/systemd/system/dm-event.socket. dm-event.service is a disabled or a static unit, not starting it. Setting up lvm2 (2.03.02-3) ... update-initramfs: deferring update (trigger activated) Created symlink /etc/systemd/system/sysinit.target.wants/blk-availability.service → /lib/systemd/system/blk-availability.service. Created symlink /etc/systemd/system/sysinit.target.wants/lvm2-monitor.service → /lib/systemd/system/lvm2-monitor.service. Created symlink /etc/systemd/system/sysinit.target.wants/lvm2-lvmpolld.socket → /lib/systemd/system/lvm2-lvmpolld.socket. Processing triggers for mime-support (3.62) ... Processing triggers for initramfs-tools (0.133+deb10u1) ... update-initramfs: Generating /boot/initrd.img-5.4.13-rockchip64 update-initramfs: Converting to u-boot format Processing triggers for libc-bin (2.28-10) ... Processing triggers for systemd (241-7~deb10u2) ... Processing triggers for rsyslog (8.1901.0-1) ... Processing triggers for dbus (1.12.16-1) ... Errors were encountered while processing: openssh-server openmediavault E: Sub-process /usr/bin/dpkg returned an error code (1) root@DietPi:~# omv-confdbadm populate [WARNING ] /usr/lib/python3/dist-packages/salt/utils/path.py:265: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working if not isinstance(exes, collections.Iterable):

Traceback (most recent call last): File "/usr/sbin/omv-confdbadm", line 74, in sys.exit(main()) File "/usr/sbin/omv-confdbadm", line 70, in main return cmd_inst.execute(sys.argv) File "/usr/share/openmediavault/confdbadm/commands.d/update.py", line 59, in execute db.set(obj) File "/usr/lib/python3/dist-packages/openmediavault/config/database.py", line 294, in set query = openmediavault.config.DatabaseSetQuery(obj) File "/usr/lib/python3/dist-packages/openmediavault/config/database.py", line 757, in init super().init(obj.model.id) File "/usr/lib/python3/dist-packages/openmediavault/config/database.py", line 305, in init self._database_file = openmediavault.getenv("OMV_CONFIG_FILE") File "/usr/lib/python3/dist-packages/openmediavault/init.py", line 56, in getenv value = openmediavault.settings.Environment.get_str(key, default) File "/usr/lib/python3/dist-packages/openmediavault/settings.py", line 77, in get_str value = Environment.get(key, default) File "/usr/lib/python3/dist-packages/openmediavault/settings.py", line 71, in get % (key, DEFAULT_FILE) KeyError: "The environment variable 'OMV_CONFIG_FILE' does not exist in '/etc/default/openmediavault'" Traceback (most recent call last): File "/usr/sbin/omv-confdbadm", line 74, in sys.exit(main()) File "/usr/sbin/omv-confdbadm", line 70, in main return cmd_inst.execute(sys.argv) File "/usr/share/openmediavault/confdbadm/commands.d/populate.py", line 48, in execute openmediavault.subprocess.check_call([script_path]) File "/usr/lib/python3/dist-packages/openmediavault/subprocess.py", line 58, in check_call return subprocess.check_call(*popenargs, **kwargs) File "/usr/lib/python3.7/subprocess.py", line 347, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/usr/share/openmediavault/confdb/populate.d/20hostname.sh']' returned non-zero exit status 1. root@DietPi:~# cat /etc/issue Armbian 20.02.0-rc0 Buster \l

[chippey5]

Please don't clutter this thread with issues that aren't related directly to the issue. Try formatting the disk(s) instead and see if that works without the kernel panicking. Regarding your current issue, try the stable version of DietPi and then install OMV. Any issues should be reported on their forums.

[hemertje]

OK, Wil try DietPi_RockPro64-ARMv8-Buster.7z | 2020-01-22 01:31 | 98M Can it run OMV? Do you have a link to a Howto?

Here's how to get started: https://forum.openmediavault.org/index.php/Thread/25062-Install-OMV5-on-Debian-10-Buster/

Try formatting the disk(s) first just to test and please share the results.

Above instruction won't work anymore on Stable DietPi...

Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation:

The following packages have unmet dependencies: openmediavault : Depends: php-pam but it is not going to be installed Depends: monit but it is not going to be installed Depends: salt-minion but it is not going to be installed E: Unable to correct problems, you have held broken packages. root@DietPi:~#

Sorry can't test for you...

[hemertje]

​Put the 3TB WD Red HDD in an external case with his own power supply Connected via USB3 between RockPro64 and the external case

Now it formats without problems, see screenshot Never came this far...

[chippey5]

Which rom are you using? DietPi or something else? The issue in this thread was also specifically reported for the pcie-adapter from pine64 and hence should be discussed around that but thanks for reporting back of the success.

[hemertje]

Armbian_19.11.7_Rockpro64_buster_legacy_4.4.208.7z from

https://dl.armbian.com/rockpro64/archive/

My Pine64 PCI card

root@rockpro64:~# lspci 00:00.0 PCI bridge: Fuzhou Rockchip Electronics Co., Ltd RK3399 PCI Express Root Port 01:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01) root@rockpro64:~#

[hemertje]

I found at my computing stuff a WD 1TB Green Caviar HDD and tried it without the external power supply, similar like before with the 3TB WD Red HDD

Got the same errors as before , but it took a little longer before the system crashed

Creating the file system, please wait ...
===== Wipe signatures from device =====
===== Create partition =====
Creating new GPT entries.
Disk /dev/sda: 1953525168 sectors, 931.5 GiB
Model: WDC WD10EADS-00L
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 1E8AE397-E393-4A56-8835-5F9A81FA53CB
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 1953525134
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048      1953525134   931.5 GiB   8300  
The operation has completed successfully.
===== Create file system =====
mke2fs 1.44.5 (15-Dec-2018)
Creating filesystem with 244190385 4k blocks and 61054976 inodes
Filesystem UUID: 88b18685-3bde-4609-99d8-539883efb79f
Superblock backups stored on blocks: 
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
    102400000, 214990848

Allocating group tables:    0/7453         done                            
Writing inode tables:    0/7453  45/7453 241/7453 317/7453 366/7453 414/7453 463/7453 511/7453 557/7453 600/7453 641/7453 680/7453 720/7453 759/7453 798/7453 838/7453 863/7453 871/7453 874/7453

 ____            _    ____               __   _  _   
|  _ \ ___   ___| | _|  _ \ _ __ ___    / /_ | || |  
| |_) / _ \ / __| |/ / |_) | '__/ _ \  | '_ \| || |_ 
|  _ < (_) | (__|   <|  __/| | | (_) | | (_) |__   _|
|_| \_\___/ \___|_|\_\_|   |_|  \___/   \___/   |_|  

Welcome to Armbian buster with Linux 4.4.208-rockchip64

System load:   0.11 0.07 0.02   Up time:       2:41 hours       
Memory usage:  5 % of 3874MB    Zram usage:    2 % of 1937Mb    IP:            192.168.2.3
CPU temp:      36°C             
Usage of /:    7% of 29G        

[ General system configuration (beta): armbian-config ]

Last login: Wed Feb  5 17:21:39 2020 from 192.168.2.106

root@rockpro64:~# 
Message from syslogd@localhost at Feb  5 19:57:03 ...
 kernel:[ 9735.004750] Internal error: : 96000210 [#1] SMP

Message from syslogd@localhost at Feb  5 19:57:10 ...
 kernel:[ 9742.274127] BUG: spinlock lockup suspected on CPU#0, scsi_eh_2/264

Message from syslogd@localhost at Feb  5 19:57:10 ...
 kernel:[ 9742.277966]  lock: 0xffffff8009321870, .magic: dead4ead, .owner: scsi_eh_2/264, .owner_cpu: 0

Message from syslogd@localhost at Feb  5 19:58:48 ...
 kernel:[ 9840.525926] Kernel panic - not syncing: hung_task: blocked tasks

[hemertje]

How to bring it down to the specific cause?

Power supply or PCI-SATA card or ?

Faulty hardware or caused by software?