rcu_sched detected stalls on CPUs/tasks

[rudrasagar]

Hi,

We are using QorIQ P1021 processor, SDK - QorIQ-SDK-V1.7 , Linux version 3.12.19. Facing a kernel crash once I pushed factory reset button. please find the below two logs And please help on this issue.

logs No.1:

The system is going down for reboot NOW!

INIT: Switching to run INIT: Sending processes the TERM signal

GOING TO REBOOT THE SYSTEM...............!!!

DUE TO FACTORY RESET.................!!!

INITStopping Dropbear SSH server: stopped /usr/sbin/dropbear (pid 2551)

dropbear.

Stopping syslogd/klogd: stopped syslogd (pid 2559)

stopped klogd (pid 2562)

done

Deconfiguring network interfaces... done.

Sending all processes the TERM signal...

Sending all processes the KILL signal...

Unmounting remote filesystems...

Deactivating swap...

Unmounting local filesystems...

Rebooting... reboot: Restarting system

INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=5252 jiffies, g=825, c=824, q=1)

Task dump for CPU 0:

reboot R running 0 2642 2641 0x00000000

Call Trace:

[c7363c70] [c0631338] schedule+0x30/0xbc (unreliable)

[c7363d30] [c007d6cc] call_console_drivers.constprop.24+0xe8/0x108

[c7363d50] [c007db14] console_unlock+0x1fc/0x458

[c7363d80] [c007e038] vprintk_emit+0x2c8/0x4cc

[c7363dd0] [c0634b58] printk+0x68/0x78

[c7363e10] [c0065178] kernel_restart+0x74/0x8c

[c7363e20] [c00653f4] SyS_reboot+0xd0/0x1d4

[c7363f40] [c000ef98] ret_from_syscall+0x0/0x3c

--- Exception: c01 at 0xff1641c

LR = 0x10000c20

INFO: rcu_sched detected stalls on CPUs/tasks: { 0} (detected by 1, t=21007 jiffies, g=825, c=824, q=1)

Task dump for CPU 0:

reboot R running 0 2642 2641 0x00000000

Call Trace:

[c7363c70] [c0631338] schedule+0x30/0xbc (unreliable)

[c7363d30] [c007d6cc] call_console_drivers.constprop.24+0xe8/0x108

[c7363d50] [c007db14] console_unlock+0x1fc/0x458

[c7363d80] [c007e038] vprintk_emit+0x2c8/0x4cc

[c7363dd0] [c0634b58] printk+0x68/0x78

[c7363e10] [c0065178] kernel_restart+0x74/0x8c

[c7363e20] [c00653f4] SyS_reboot+0xd0/0x1d4

[c7363f40] [c000ef98] ret_from_syscall+0x0/0x3c

--- Exception: c01 at 0xff1641c

LR = 0x10000c20

logs No.2:

The system is going down for reboot NOW!

GOING TO REBOOT THE SYSTEM...............!!!

DUE TO FACTORY RESET.................!!!

INITStopping Dropbear SSH server: stopped /usr/sbin/dropbear (pid 2551)

dropbear.

Stopping syslogd/klogd: stopped syslogd (pid 2559)

stopped klogd (pid 2562)

done

Deconfiguring network interfaces... done.

INIT: Sending processes the TERM signal

INIT: Remote log enabled: False

Remote log server:

Starting syslog - local logging only

Deconfiguring network interfaces... done.

Sending all processes the TERM signal...

tcpsvd: got signal 15, exit

Sending all processes the KILL signal...

Unmounting remote filesystems...

Deactivating swap...

Unmounting local filesystems...

UBIFS: background thread "ubifs_bgt0_0" stops

INFO: rcu_sched self-detected stall on CPU { 1} (t=5250 jiffies g=3251 c=3250 q=736)

CPU: 1 PID: 3477 Comm: umount Tainted: P O 3.12.19-rt30isdn-svn943 #1

Call Trace:

[c73c1ab0] [c0006fd8] show_stack+0x44/0x15c (unreliable)

[c73c1af0] [c063df18] dump_stack+0x78/0xa0

[c73c1b00] [c00a60ec] rcu_check_callbacks+0x30c/0x6d0

[c73c1b60] [c004ae1c] update_process_times+0x40/0x68

[c73c1b80] [c0091ee4] tick_sched_timer+0x68/0xb0

[c73c1bb0] [c0061594] __run_hrtimer.isra.28+0x54/0xd8

[c73c1bd0] [c00621d8] hrtimer_interrupt+0x15c/0x418

[c73c1c50] [c0009bd4] timer_interrupt+0x1a0/0x1d0

[c73c1c80] [c000f69c] ret_from_except+0x0/0x18

--- Exception: 901 at _raw_spin_lock+0x30/0x60

LR = shmem_undo_range+0x45c/0x644

[c73c1d40] [c00bd00c] shmem_undo_range+0x264/0x644 (unreliable)

[c73c1e10] [c00bd408] shmem_truncate_range+0x1c/0x50

[c73c1e40] [c00bd534] shmem_evict_inode+0xf8/0x16c

[c73c1e60] [c01078f4] evict+0xc0/0x1b0

[c73c1e80] [c010434c] shrink_dcache_for_umount_subtree+0x15c/0x1ec

[c73c1ea0] [c0105d30] shrink_dcache_for_umount+0x40/0x78

[c73c1eb0] [c00eff74] generic_shutdown_super+0x28/0x100

[c73c1ed0] [c00f02a4] kill_litter_super+0x30/0x48

[c73c1ee0] [c00f0690] deactivate_locked_super+0x60/0x88

[c73c1ef0] [c010e064] SyS_umount+0x8c/0x388

[c73c1f40] [c000efe0] ret_from_syscall+0x0/0x3c

--- Exception: c01 at 0xff1e21c

LR = 0x10099cec

[behlendorf]

Closing. Nothing ZFS specific appears in the stacks, there's nothing to indicate this is a ZFS issue. Plus there's nothing actionable in the log to investigate.