search

open-power / skiboot

OPAL boot and runtime firmware for POWER
Apache License 2.0
98 stars 134 forks source link

putscom kernel oops then checkstop #194

Closed abali999 closed 6 years ago

abali999 commented 6 years ago

putscom -c 0 0x1001180b 0x0000000000000000 works putscom -c 0 0x1001180b 0xf000000000000000 crashes Any suggestions what I am doing wrong? The scom register is supposed to turn off some L3 ways, and can be changed dynamically according to docs. This is on a P9 LC922 box with powerNV kernel and using putscom from the opal-utils package.

[ 218.533737] Unable to handle kernel paging request for data at address 0x00000000 [ 218.533821] Faulting instruction address: 0xc000000000353484 [ 218.533879] Oops: Kernel access of bad area, sig: 11 [#1] [ 218.533917] LE SMP NR_CPUS=2048 NUMA PowerNV [ 218.533961] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 218.534303] CPU: 24 PID: 3409 Comm: sshd Not tainted 4.15.0-33-generic #36-Ubuntu [ 218.534369] NIP: c000000000353484 LR: c00000000033720c CTR: c00000000032d280 [ 218.534436] REGS: c000003fdce376f0 TRAP: 0300 Not tainted (4.15.0-33-generic) [ 218.534491] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 24002828 XER: 20040000 [ 218.534552] CFAR: c000000000337208 DAR: 0000000000000000 DSISR: 40000000 SOFTE: 1 [ 218.534552] GPR00: c00000000033720c c000003fdce37970 c0000000016eb000 c000003fbafa9990 [ 218.534552] GPR04: c00000000441e168 00007fc03c5d0000 00007655f99e0000 00007fc03c5d0000 [ 218.534552] GPR08: ffffffff00000001 0000000000000000 0000000000000000 c000003fd57dfa98 [ 218.534552] GPR12: c00000000032d280 c00000000fa90800 0000000000000d15 0000000000000148 [ 218.534552] GPR16: 00000e199e4b86e0 000000000000000b 0000000000008005 0000000000040100 [ 218.534552] GPR20: 0000000000000000 00000000418004fc 00000000003c0000 0000000008430000 [ 218.534552] GPR24: c00000000188e9f0 0000000000000000 c000003fbafa9990 c000003fdce37a50 [ 218.534552] GPR28: c000003fbafa9a08 0000000000000001 c000003fbafabfc0 0000000000000000 [ 218.535042] NIP [c000000000353484] unlink_anon_vmas+0x44/0x310 [ 218.535089] LR [c00000000033720c] free_pgtables+0x16c/0x240 [ 218.535124] Call Trace: [ 218.535145] [c000003fdce37970] [c000000000344e08] remove_shared_vm_struct+0x68/0x80 (unreliable) [ 218.535208] [c000003fdce379d0] [c00000000033720c] free_pgtables+0x16c/0x240 [ 218.535255] [c000003fdce37a30] [c0000000003481a4] exit_mmap+0xc4/0x210 [ 218.535301] [c000003fdce37af0] [c000000000109ec8] mmput+0xa8/0x1d0 [ 218.535347] [c000003fdce37b20] [c000000000114540] do_exit+0x360/0xc80 [ 218.535394] [c000003fdce37be0] [c000000000114f30] do_group_exit+0x60/0x100 [ 218.535440] [c000003fdce37c20] [c0000000001264ec] get_signal+0x3cc/0x7a0 [ 218.535487] [c000003fdce37d10] [c00000000001d98c] do_signal+0x7c/0x2c0 [ 218.535534] [c000003fdce37e00] [c00000000001dd70] do_notify_resume+0xd0/0x100 [ 218.535589] [c000003fdce37e30] [c00000000000b8c4] ret_from_except_lite+0x70/0x74 [ 218.535642] Instruction dump: [ 218.535670] fb41ffd0 fb61ffd8 fb81ffe0 fba1ffe8 fbc1fff0 fbe1fff8 f8010010 f821ffa1 [ 218.535727] 7c7c1b78 7c7a1b78 ebfc0079 7fbcf800 419e02a8 fb010020 fb210028 [ 218.535786] ---[ end trace 4456a1917aa7e1d1 ]--- [ 218.679869] Watchdog CPU:0 Hard LOCKUP [ 218.679869] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 218.679884] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 218.679885] NIP: c00000000017414c LR: c000000000174570 CTR: 0000000000000008 [ 218.679886] REGS: c00000003ffffd80 TRAP: 0900 Tainted: G D (4.15.0-33-generic) [ 218.679886] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 24000424 XER: 20040000 [ 218.679890] CFAR: c00000000017456c SOFTE: 0 [ 218.679890] GPR00: c000000000174570 c0000000016e7e30 c0000000016eb000 c00000000161caa8 [ 218.679890] GPR04: c000003feeb96fd8 0000000000000000 0000000000000000 0000000000000000 [ 218.679890] GPR08: c0000000016e4080 c0000000016e4000 0000000000000008 c000003feeb9727c [ 218.679890] GPR12: 0000000000000000 c00000000fa80000 0000000000000000 0000000000000000 [ 218.679890] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 218.679890] GPR20: 0000000000000000 0000000000000001 0000000010004d9c 00000000100053ed [ 218.679890] GPR24: 0000000010005411 c000000000000000 c00000000161caa8 0000000000000008 [ 218.679890] GPR28: c000003feeb96fd8 c000000001722364 0000000000000001 c000000001721ee0 [ 218.679903] NIP [c00000000017414c] call_cpuidle+0x3c/0x90 [ 218.679905] LR [c000000000174570] do_idle+0x2b0/0x330 [ 218.679905] Call Trace: [ 218.679907] [c0000000016e7e30] [c000000000ac4898] cpuidle_select+0x38/0x50 (unreliable) [ 218.679909] [c0000000016e7e50] [c000000000174570] do_idle+0x2b0/0x330 [ 218.679911] [c0000000016e7ea0] [c00000000017482c] cpu_startup_entry+0x3c/0x50 [ 218.679912] [c0000000016e7ed0] [c00000000000d4a0] rest_init+0xe0/0x100 [ 218.679914] [c0000000016e7f00] [c0000000010b4160] start_kernel+0x5a0/0x5c0 [ 218.679916] [c0000000016e7f90] [c00000000000ac7c] start_here_common+0x1c/0x520 [ 218.679916] Instruction dump: [ 218.679918] 60000000 7c0802a6 f8010010 f821ffe1 39400008 78290464 39090080 7cc040a8 [ 218.679920] 7cc65078 7cc041ad 40c2fff4 7c0004ac 71290004 4082001c 48950761 [ 218.681381] Unable to handle kernel paging request for data at address 0xaa530476c81c08f5 [ 218.681451] Faulting instruction address: 0xc000000000ac0ce8 [ 218.753807] Unable to handle kernel paging request for data at address 0xaa530476c81cd4f5 [ 218.753855] Faulting instruction address: 0xc000000000ac0ce8 [ 219.539965] [ 219.539991] Fixing recursive fault but reboot is needed! [ 219.539993] Oops: Kernel access of bad area, sig: 11 [#2] [ 219.539994] LE SMP NR_CPUS=2048 NUMA PowerNV [ 219.539996] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 219.540011] CPU: 41 PID: 256 Comm: migration/41 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 219.540012] NIP: c000000000ac0ce8 LR: c00000000016314c CTR: c000000000ac0cd0 [ 219.540013] REGS: c000003fe4903730 TRAP: 0380 Tainted: G D (4.15.0-33-generic) [ 219.540013] MSR: 900000000280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 44000224 XER: 20040000 [ 219.540017] CFAR: c000000000163148 SOFTE: 0 [ 219.540017] GPR00: c00000000016314c c000003fe49039b0 c0000000016eb000 c000003feed96fb8 [ 219.540017] GPR04: 00000032eebe243c 0000000000000000 0000000000000000 aa530476c81cd0c5 [ 219.540017] GPR08: c00000000171dd78 c0000000055fa740 0000003fedbc0000 c000003fd8ceba30 [ 219.540017] GPR12: c000000000ac0cd0 c00000000fa9c300 c00000000013d748 c000003fed0e6a80 [ 219.540017] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 219.540017] GPR20: 0000000000000000 0000000000000000 0000000000000000 c000003fef5d8580 [ 219.540017] GPR24: c00000000171dd78 c000000000ac0cd0 c000003feed96fb8 0000000000000000 [ 219.540017] GPR28: 0000000000000001 c000003feed98580 c000003fe1eee400 c000003feed98600 [ 219.540032] NIP [c000000000ac0ce8] dbs_update_util_handler+0x18/0xf0 [ 219.540033] LR [c00000000016314c] enqueue_entity+0x37c/0xd90 [ 219.540034] Call Trace: [ 219.540035] [c000003fe49039b0] [c0000000001630d0] enqueue_entity+0x300/0xd90 (unreliable) [ 219.540037] [c000003fe4903a60] [c000000000163c10] enqueue_task_fair+0xb0/0x7c0 [ 219.540039] [c000003fe4903b30] [c00000000014d8f8] activate_task+0x88/0x130 [ 219.540041] [c000003fe4903bb0] [c000000000157278] attach_task+0x68/0xa0 [ 219.540042] [c000003fe4903be0] [c00000000015e2d4] active_load_balance_cpu_stop+0x124/0x370 [ 219.540044] [c000003fe4903cb0] [c000000000206c1c] cpu_stopper_thread+0xfc/0x1f0 [ 219.540046] [c000003fe4903d60] [c000000000143ca0] smpboot_thread_fn+0x250/0x290 [ 219.540047] [c000003fe4903dc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 219.540049] [c000003fe4903e30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 219.540051] Instruction dump: [ 219.540052] ebe1fff8 7d908120 4e800020 3860ffea 4bffffe0 60420000 3c4c00c3 3842a330 [ 219.540054] 7c0802a6 60000000 e9230008 e8e90000 <89470430> 2f8a0000 409e0020 a14d0008 [ 219.540057] ---[ end trace 4456a1917aa7e1d2 ]--- [ 220.035966] Unable to handle kernel paging request for data at address 0xaa530476c81cd4f5 [ 220.035968] Unable to handle kernel paging request for data at address 0xaa530476c81cd4f5 [ 220.035969] Unable to handle kernel paging request for data at address 0xaa530476c81cd4f5 [ 220.035973] Faulting instruction address: 0xc000000000ac0ce8 [ 220.035974] Faulting instruction address: 0xc000000000ac0ce8 [ 220.036250] Faulting instruction address: 0xc000000000ac0ce8 [ 220.080696] Unable to handle kernel paging request for instruction fetch [ 220.080757] Faulting instruction address: 0x00000000 [ 220.096694] Unable to handle kernel paging request for data at address 0xaa530476c81c13b5 [ 220.096695] Unable to handle kernel paging request for data at address 0xaa530476c81c13b5 [ 220.096698] Faulting instruction address: 0xc000000000ac0ce8 [ 220.096915] Faulting instruction address: 0xc000000000ac0ce8 [ 220.100036] Unable to handle kernel paging request for data at address 0xaa530476c81c13b5 [ 220.100110] Faulting instruction address: 0xc000000000ac0ce8 [ 220.104040] Unable to handle kernel paging request for data at address 0xaa530476c81c13b5 [ 220.104123] Faulting instruction address: 0xc000000000ac0ce8 [ 220.219983] Unable to handle kernel paging request for data at address 0xaa530476c81c08f5 [ 220.220072] Faulting instruction address: 0xc000000000ac0ce8 [ 220.227981] Unable to handle kernel paging request for data at address 0xaa530476c81c08f5 [ 220.228061] Faulting instruction address: 0xc000000000ac0ce8 [ 220.231983] Unable to handle kernel paging request for data at address 0xaa530476c81c08f5 [ 220.232071] Faulting instruction address: 0xc000000000ac0ce8 [ 220.343828] [ 220.343857] Oops: Kernel access of bad area, sig: 11 [#3] [ 220.343865] note: migration/41[256] exited with preempt_count 1 [ 220.343945] LE SMP NR_CPUS=2048 NUMA PowerNV [ 220.343988] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 220.344291] CPU: 9 PID: 63 Comm: watchdog/9 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 220.344360] NIP: c000000000ac0ce8 LR: c00000000016b260 CTR: c000000000ac0cd0 [ 220.344419] REGS: c000003fe50478a0 TRAP: 0380 Tainted: G D (4.15.0-33-generic) [ 220.344486] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 44000884 XER: 00000000 [ 220.344554] CFAR: c00000000016b25c SOFTE: 0 [ 220.344554] GPR00: c00000000016b260 c000003fe5047b20 c0000000016eb000 c000003feedd6fb8 [ 220.344554] GPR04: 000000333b2a589c 0000000000000001 0000002d8edd465d aa530476c81cd0c5 [ 220.344554] GPR08: 0000000000000000 c0000000055fa740 0000003fedc00000 0000000000000005 [ 220.344554] GPR12: c000000000ac0cd0 c00000000fa86300 c00000000013d748 c000003fed0e5a80 [ 220.344554] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 220.344554] GPR20: 0000000000000000 0000000000000000 c000003fe51c4a80 0000003fedc00000 [ 220.344554] GPR24: c000003feedd8580 0000000000000000 c000000000ac0cd0 c000003feedd6fb8 [ 220.344554] GPR28: c00000000171dd78 00000000000006d8 c000003fe51c4380 c000003feedd8580 [ 220.345084] NIP [c000000000ac0ce8] dbs_update_util_handler+0x18/0xf0 [ 220.345135] LR [c00000000016b260] update_curr_rt+0xd0/0x3f0 [ 220.345175] Call Trace: [ 220.345198] [c000003fe5047bb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 220.345250] [c000003fe5047bf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 220.345311] [c000003fe5047c70] [c000000000cfb9dc] schedule+0x3bc/0xaf0 [ 220.345363] [c000003fe5047d40] [c000000000cfc150] schedule+0x40/0xc0 [ 220.345415] [c000003fe5047d60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 220.345475] [c000003fe5047dc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 220.345528] [c000003fe5047e30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 220.345586] Instruction dump: [ 220.345618] ebe1fff8 7d908120 4e800020 3860ffea 4bffffe0 60420000 3c4c00c3 3842a330 [ 220.345683] 7c0802a6 60000000 e9230008 e8e90000 <89470430> 2f8a0000 409e0020 a14d0008 [ 220.345751] ---[ end trace 4456a1917aa7e1d3 ]--- [ 220.476649] Unable to handle kernel paging request for data at address 0xaa530476c81ca3b5 [ 220.476715] Faulting instruction address: 0xc000000000ac0ce8 [ 220.484106] Unable to handle kernel paging request for data at address 0xaa530476c81ca3b5 [ 220.484184] Faulting instruction address: 0xc000000000ac0ce8 [ 220.492106] Unable to handle kernel paging request for data at address 0xaa530476c81ca3b5 [ 220.492185] Faulting instruction address: 0xc000000000ac0ce8 [ 220.504108] Unable to handle kernel paging request for data at address 0xaa530476c81ca3b5 [ 220.504196] Faulting instruction address: 0xc000000000ac0ce8 [ 221.153481] [ 221.153520] Oops: Kernel access of bad area, sig: 11 [#4] [ 221.153580] LE SMP NR_CPUS=2048 NUMA PowerNV [ 221.153644] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 221.154105] CPU: 94 PID: 574 Comm: watchdog/94 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 221.154207] NIP: c000000000ac0ce8 LR: c00000000016b260 CTR: c000000000ac0cd0 [ 221.154295] REGS: c00020397bf838a0 TRAP: 0380 Tainted: G D (4.15.0-33-generic) [ 221.154410] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 48000884 XER: 00000000 [ 221.154513] CFAR: c00000000016b25c SOFTE: 0 [ 221.154513] GPR00: c00000000016b260 c00020397bf83b20 c0000000016eb000 c000203993be6fb8 [ 221.154513] GPR04: 00000033469c44ea 0000000000000001 0000002d94b96509 aa530476c81c04c5 [ 221.154513] GPR08: 0000000000000000 c0000000055fce40 0000203992a10000 0000000000000005 [ 221.154513] GPR12: c000000000ac0cd0 c00000000fac0a00 c00000000013d748 c000003fed0e8500 [ 221.154513] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 221.154513] GPR20: 0000000000000000 0000000000000000 c00020397bc9c900 0000203992a10000 [ 221.154513] GPR24: c000203993be8580 0000000000000000 c000000000ac0cd0 c000203993be6fb8 [ 221.154513] GPR28: c00000000171dd78 00000000000009a4 c00020397bc9c200 c000203993be8580 [ 221.155286] NIP [c000000000ac0ce8] dbs_update_util_handler+0x18/0xf0 [ 221.155362] LR [c00000000016b260] update_curr_rt+0xd0/0x3f0 [ 221.155421] Call Trace: [ 221.155456] [c00020397bf83bb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 221.155533] [c00020397bf83bf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 221.155625] [c00020397bf83c70] [c000000000cfb9dc] schedule+0x3bc/0xaf0 [ 221.155702] [c00020397bf83d40] [c000000000cfc150] schedule+0x40/0xc0 [ 221.155795] [c00020397bf83d60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 221.155884] [c00020397bf83dc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 221.155963] [c00020397bf83e30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 221.156049] Instruction dump: [ 221.156096] ebe1fff8 7d908120 4e800020 3860ffea 4bffffe0 60420000 3c4c00c3 3842a330 [ 221.156195] 7c0802a6 60000000 e9230008 e8e90000 <89470430> 2f8a0000 409e0020 a14d0008 [ 221.156297] ---[ end trace 4456a1917aa7e1d4 ]--- [ 221.971085] [ 221.971130] Oops: Kernel access of bad area, sig: 11 [#5] [ 221.971188] LE SMP NR_CPUS=2048 NUMA PowerNV [ 221.971249] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 221.971691] CPU: 93 PID: 568 Comm: watchdog/93 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 221.971791] NIP: c000000000ac0ce8 LR: c00000000016b260 CTR: c000000000ac0cd0 [ 221.971893] REGS: c00020397bfa38a0 TRAP: 0380 Tainted: G D (4.15.0-33-generic) [ 221.971991] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 48000884 XER: 00000000 [ 221.972091] CFAR: c00000000016b25c SOFTE: 0 [ 221.972091] GPR00: c00000000016b260 c00020397bfa3b20 c0000000016eb000 c000203993ba6fb8 [ 221.972091] GPR04: 000000334622324c 0000000000000001 0000002d947ae473 aa530476c81c04c5 [ 221.972091] GPR08: 0000000000000000 c0000000055fce40 00002039929d0000 0000000000000005 [ 221.972091] GPR12: c000000000ac0cd0 c00000000fabff00 c00000000013d748 c000003fed0e8480 [ 221.972091] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 221.972091] GPR20: 0000000000000000 0000000000000000 c00020397bcb5e00 00002039929d0000 [ 221.972091] GPR24: c000203993ba8580 0000000000000000 c000000000ac0cd0 c000203993ba6fb8 [ 221.972091] GPR28: c00000000171dd78 0000000000000be6 c00020397bcb5700 c000203993ba8580 [ 221.972848] NIP [c000000000ac0ce8] dbs_update_util_handler+0x18/0xf0 [ 221.972922] LR [c00000000016b260] update_curr_rt+0xd0/0x3f0 [ 221.972979] Call Trace: [ 221.973014] [c00020397bfa3bb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 221.973089] [c00020397bfa3bf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 221.973179] [c00020397bfa3c70] [c000000000cfb9dc] schedule+0x3bc/0xaf0 [ 221.973271] [c00020397bfa3d40] [c000000000cfc150] schedule+0x40/0xc0 [ 221.973348] [c00020397bfa3d60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 221.973436] [c00020397bfa3dc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 221.973514] [c00020397bfa3e30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 221.973598] Instruction dump: [ 221.973643] ebe1fff8 7d908120 4e800020 3860ffea 4bffffe0 60420000 3c4c00c3 3842a330 [ 221.973741] 7c0802a6 60000000 e9230008 e8e90000 <89470430> 2f8a0000 409e0020 a14d0008 [ 221.973840] ---[ end trace 4456a1917aa7e1d5 ]--- [ 222.788742] [ 222.788784] Oops: Kernel access of bad area, sig: 11 [#6] [ 222.788841] LE SMP NR_CPUS=2048 NUMA PowerNV [ 222.788902] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 222.789351] CPU: 95 PID: 580 Comm: watchdog/95 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 222.789448] NIP: c000000000ac0ce8 LR: c00000000016b260 CTR: c000000000ac0cd0 [ 222.789532] REGS: c00020397bfa78a0 TRAP: 0380 Tainted: G D (4.15.0-33-generic) [ 222.789629] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 48000884 XER: 00000000 [ 222.789727] CFAR: c00000000016b25c SOFTE: 0 [ 222.789727] GPR00: c00000000016b260 c00020397bfa7b20 c0000000016eb000 c000203993c26fb8 [ 222.789727] GPR04: 0000003346d95494 0000000000000001 0000002d94d8a858 aa530476c81c04c5 [ 222.789727] GPR08: 0000000000000000 c0000000055fce40 0000203992a50000 0000000000000005 [ 222.789727] GPR12: c000000000ac0cd0 c00000000fac1500 c00000000013d748 c000003fed0e8580 [ 222.789727] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 222.789727] GPR20: 0000000000000000 0000000000000000 c00020397bc84a80 0000203992a50000 [ 222.789727] GPR24: c000203993c28580 0000000000000000 c000000000ac0cd0 c000203993c26fb8 [ 222.789727] GPR28: c00000000171dd78 0000000000000ac8 c00020397bc84380 c000203993c28580 [ 222.790468] NIP [c000000000ac0ce8] dbs_update_util_handler+0x18/0xf0 [ 222.790541] LR [c00000000016b260] update_curr_rt+0xd0/0x3f0 [ 222.790597] Call Trace: [ 222.790631] [c00020397bfa7bb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 222.790720] [c00020397bfa7bf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 222.790808] [c00020397bfa7c70] [c000000000cfb9dc] schedule+0x3bc/0xaf0 [ 222.790883] [c00020397bfa7d40] [c000000000cfc150] schedule+0x40/0xc0 [ 222.790958] [c00020397bfa7d60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 222.791044] [c00020397bfa7dc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 222.791119] [c00020397bfa7e30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 222.791202] Instruction dump: [ 222.791247] ebe1fff8 7d908120 4e800020 3860ffea 4bffffe0 60420000 3c4c00c3 3842a330 [ 222.791343] 7c0802a6 60000000 e9230008 e8e90000 <89470430> 2f8a0000 409e0020 a14d0008 [ 222.791440] ---[ end trace 4456a1917aa7e1d6 ]--- [ 223.284156] Unable to handle kernel paging request for data at address 0xe0a3ebad6cd23f6c [ 223.284222] Faulting instruction address: 0xc00000000038c300 [ 223.605875] [ 223.605916] Oops: Kernel access of bad area, sig: 11 [#7] [ 223.605950] LE SMP NR_CPUS=2048 NUMA PowerNV [ 223.605984] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 223.606221] CPU: 80 PID: 1617 Comm: systemd-journal Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 223.606277] NIP: c000000000ac0ce8 LR: c000000000159748 CTR: c000000000ac0cd0 [ 223.606342] REGS: c000003fdc18ba00 TRAP: 0380 Tainted: G D (4.15.0-33-generic) [ 223.606397] MSR: 900000000280b033 <SF,HV,VEC,VSX,EE,FP,ME,IR,DR,RI,LE> CR: 48022284 XER: 20040000 [ 223.606458] CFAR: c000000000159744 SOFTE: 0 [ 223.606458] GPR00: c000000000159748 c000003fdc18bc80 c0000000016eb000 c000203993b66fb8 [ 223.606458] GPR04: 00000032ea6cf128 0000000000000000 0000000000000000 aa530476c81c04c5 [ 223.606458] GPR08: c00000000171dd78 c0000000055fce40 0000203992990000 00000000000004fd [ 223.606458] GPR12: c000000000ac0cd0 c00000000fab7000 00000f765a3f0728 00007af734010000 [ 223.606458] GPR16: 00000f765a3f05d0 00007af734c74410 00007af734c74420 00007af73481f030 [ 223.606458] GPR20: c000000000ac0cd0 c000203993b66fb8 0000000000000001 000000000000ba7e [ 223.606458] GPR24: 000000000000ba7e afb504000afb5041 0000000000000000 0000000000000001 [ 223.606458] GPR28: 0000000000000000 c000203993b68580 c0002039699f8c00 c000203993b68600 [ 223.606866] NIP [c000000000ac0ce8] dbs_update_util_handler+0x18/0xf0 [ 223.606907] LR [c000000000159748] propagate_entity_cfs_rq.isra.11+0x4a8/0x5f0 [ 223.606954] Call Trace: [ 223.606972] [c000003fdc18bc80] [c000000000159400] propagate_entity_cfs_rq.isra.11+0x160/0x5f0 (unreliable) [ 223.607030] [c000003fdc18bd40] [c000000000150aec] wake_up_new_task+0x13c/0x310 [ 223.607079] [c000003fdc18bda0] [c00000000010c124] _do_fork+0x164/0x510 [ 223.607120] [c000003fdc18be30] [c00000000000b580] ppc_clone+0x8/0xc [ 223.607160] Instruction dump: [ 223.607185] ebe1fff8 7d908120 4e800020 3860ffea 4bffffe0 60420000 3c4c00c3 3842a330 [ 223.607236] 7c0802a6 60000000 e9230008 e8e90000 <89470430> 2f8a0000 409e0020 a14d0008 [ 223.607287] ---[ end trace 4456a1917aa7e1d7 ]--- [ 224.411343] [ 224.411372] Oops: Kernel access of bad area, sig: 11 [#8] [ 224.411403] LE SMP NR_CPUS=2048 NUMA PowerNV [ 224.411434] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 224.411649] CPU: 46 PID: 3323 Comm: iprinit Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 224.411698] NIP: c00000000038c300 LR: c00000000038c2ec CTR: 00000000000001f6 [ 224.411741] REGS: c000003fbe667890 TRAP: 0380 Tainted: G D (4.15.0-33-generic) [ 224.411790] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 24000814 XER: 20040000 [ 224.411836] CFAR: c000000000016f1c SOFTE: 1 [ 224.411836] GPR00: c00000000038c1d4 c000003fbe667b10 c0000000016eb000 0000000000000001 [ 224.411836] GPR04: c000003fd5254600 0000000000001f28 0000000000000001 0000000000001f27 [ 224.411836] GPR08: e0a3ebad6cd2636c 0000000000000000 0000000000000000 0000000000000000 [ 224.411836] GPR12: 0000000000000000 c00000000fa9fa00 00000bc929199740 00000bc929199768 [ 224.411836] GPR16: 0000000000000000 00000bc929199770 00000bc929199780 00007ffff51cb498 [ 224.411836] GPR20: 0000000000000005 00000bc9291b07e0 00000bc929199730 0000000000000005 [ 224.411836] GPR24: 20a3eb92b9f7256c c000003fd5251a00 e0a3ebad6cd23f6c c0000000003d6d98 [ 224.411836] GPR28: c000003fed01a600 00000000014080c0 c000003fd5251a00 c000003fed01a600 [ 224.412209] NIP [c00000000038c300] kmem_cache_alloc+0x2e0/0x340 [ 224.412249] LR [c00000000038c2ec] kmem_cache_alloc+0x2cc/0x340 [ 224.412284] Call Trace: [ 224.412300] [c000003fbe667b10] [c00000000038c1d4] kmem_cache_alloc+0x1b4/0x340 (unreliable) [ 224.412344] [c000003fbe667b70] [c0000000003d6d98] get_empty_filp+0x78/0x230 [ 224.412380] [c000003fbe667bf0] [c0000000003ec840] path_openat+0x50/0x3e0 [ 224.412417] [c000003fbe667c70] [c0000000003ee150] do_filp_open+0x80/0x120 [ 224.412454] [c000003fbe667da0] [c0000000003d16a8] do_sys_open+0x248/0x3f0 [ 224.412491] [c000003fbe667e30] [c00000000000b284] system_call+0x58/0x6c [ 224.412526] Instruction dump: [ 224.412548] 7c98c278 fb410030 38a50001 7f3ac278 fb490000 f8aa0000 4bc8ac21 60000000 [ 224.412593] 7fb9c040 419e0028 e93f0022 e91f0140 <7d5a482a> 7d3a4a14 7d4a4278 7fa95040 [ 224.412639] ---[ end trace 4456a1917aa7e1d8 ]--- [ 225.216849] [ 225.216872] Oops: Kernel access of bad area, sig: 11 [#9] [ 225.216913] LE SMP NR_CPUS=2048 NUMA PowerNV [ 225.216955] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 225.217266] CPU: 11 PID: 75 Comm: watchdog/11 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 225.217334] NIP: c000000000ac0ce8 LR: c00000000016b260 CTR: c000000000ac0cd0 [ 225.217392] REGS: c000003fe50438a0 TRAP: 0380 Tainted: G D (4.15.0-33-generic) [ 225.217459] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 44000884 XER: 00000000 [ 225.217525] CFAR: c00000000016b25c SOFTE: 0 [ 225.217525] GPR00: c00000000016b260 c000003fe5043b20 c0000000016eb000 c000003feee56fb8 [ 225.217525] GPR04: 000000333b2a58a4 0000000000000001 0000002d8edd465d aa530476c81cd0c5 [ 225.217525] GPR08: 0000000000000000 c0000000055fa740 0000003fedc80000 0000000000000005 [ 225.217525] GPR12: c000000000ac0cd0 c00000000fa87900 c00000000013d748 c000003fed0e5b80 [ 225.217525] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 225.217525] GPR20: 0000000000000000 0000000000000000 c000003fe51e7d00 0000003fedc80000 [ 225.217525] GPR24: c000003feee58580 0000000000000000 c000000000ac0cd0 c000003feee56fb8 [ 225.217525] GPR28: c00000000171dd78 0000000000000772 c000003fe51e7600 c000003feee58580 [ 225.218034] NIP [c000000000ac0ce8] dbs_update_util_handler+0x18/0xf0 [ 225.218085] LR [c00000000016b260] update_curr_rt+0xd0/0x3f0 [ 225.218124] Call Trace: [ 225.218147] [c000003fe5043bb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 225.218197] [c000003fe5043bf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 225.218258] [c000003fe5043c70] [c000000000cfb9dc] __schedule+0x3bc/0xaf0 [ 225.218309] [c000003fe5043d40] [c000000000cfc150] schedule+0x40/0xc0 [ 225.218360] [c000003fe5043d60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 225.218420] [c000003fe5043dc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 225.218486] [c000003fe5043e30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 225.218544] Instruction dump: [ 225.218574] ebe1fff8 7d908120 4e800020 3860ffea 4bffffe0 60420000 3c4c00c3 3842a330 [ 225.218639] 7c0802a6 60000000 e9230008 e8e90000 <89470430> 2f8a0000 409e0020 a14d0008 [ 225.218705] ---[ end trace 4456a1917aa7e1d9 ]--- [ 226.028337] [ 226.028363] Oops: Kernel access of bad area, sig: 11 [#10] [ 226.028393] LE SMP NR_CPUS=2048 NUMA PowerNV [ 226.028423] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 226.028648] CPU: 48 PID: 297 Comm: watchdog/48 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 226.028697] NIP: 0000000000000000 LR: c00000000016b260 CTR: 0000000000000000 [ 226.028739] REGS: c000003fe4d2f8a0 TRAP: 0400 Tainted: G D (4.15.0-33-generic) [ 226.028788] MSR: 9000000040009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 44000884 XER: 00000000 [ 226.028833] CFAR: c00000000016b25c SOFTE: 0 [ 226.028833] GPR00: c00000000016b76c c000003fe4d2fb20 c0000000016eb000 c000003fef796fb8 [ 226.028833] GPR04: 000000333dd4e13c 0000000000000001 0000002d903abb2c 0000003fee5c0000 [ 226.028833] GPR08: 0000000000000000 0000000000000004 0000003fee5c0000 0000000000000005 [ 226.028833] GPR12: 0000000000000000 c00000000faa1000 c00000000013d748 c000003fed0e6e00 [ 226.028833] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 226.028833] GPR20: 0000000000000000 0000000000000000 c000003fe4e84a80 0000003fee5c0000 [ 226.028833] GPR24: c000003fef798580 0000000000000000 0000000000000000 c000003fef796fb8 [ 226.028833] GPR28: c00000000171dd78 0000000000000ade c000003fe4e84380 c000003fef798580 [ 226.029192] NIP [0000000000000000] (null) [ 226.029222] LR [c00000000016b260] update_curr_rt+0xd0/0x3f0 [ 226.029250] Call Trace: [ 226.029266] [c000003fe4d2fbb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 226.029303] [c000003fe4d2fbf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 226.029346] [c000003fe4d2fc70] [c000000000cfb9dc] schedule+0x3bc/0xaf0 [ 226.029383] [c000003fe4d2fd40] [c000000000cfc150] schedule+0x40/0xc0 [ 226.029420] [c000003fe4d2fd60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 226.029463] [c000003fe4d2fdc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 226.029499] [c000003fe4d2fe30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 226.029541] Instruction dump: [ 226.029564] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX [ 226.029608] XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX XXXXXXXX [ 226.029653] ---[ end trace 4456a1917aa7e1da ]--- [ 226.833815] [ 226.833843] Oops: Kernel access of bad area, sig: 11 [#11] [ 226.833904] LE SMP NR_CPUS=2048 NUMA PowerNV [ 226.833984] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 226.834445] CPU: 133 PID: 808 Comm: watchdog/133 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 226.834547] NIP: c000000000ac0ce8 LR: c00000000016b260 CTR: c000000000ac0cd0 [ 226.834635] REGS: c000203978bcb8a0 TRAP: 0380 Tainted: G D (4.15.0-33-generic) [ 226.834735] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 48000884 XER: 00000000 [ 226.834838] CFAR: c00000000016b25c SOFTE: 0 [ 226.834838] GPR00: c00000000016b260 c000203978bcbb20 c0000000016eb000 c0002039945a6fb8 [ 226.834838] GPR04: 0000003355e06e20 0000000000000001 0000002d9c8a5f58 aa530476c81c9f85 [ 226.834838] GPR08: 0000000000000000 c0000000055fb700 00002039933d0000 0000000000000005 [ 226.834838] GPR12: c000000000ac0cd0 c00000000fadb700 c00000000013d748 c000003fed0e9880 [ 226.834838] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 226.834838] GPR20: 0000000000000000 0000000000000000 c0002039788a6680 00002039933d0000 [ 226.834838] GPR24: c0002039945a8580 0000000000000000 c000000000ac0cd0 c0002039945a6fb8 [ 226.834838] GPR28: c00000000171dd78 0000000000000a52 c0002039788a5f80 c0002039945a8580 [ 226.835627] NIP [c000000000ac0ce8] dbs_update_util_handler+0x18/0xf0 [ 226.835704] LR [c00000000016b260] update_curr_rt+0xd0/0x3f0 [ 226.835762] Call Trace: [ 226.835797] [c000203978bcbbb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 226.835875] [c000203978bcbbf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 226.835967] [c000203978bcbc70] [c000000000cfb9dc] schedule+0x3bc/0xaf0 [ 226.836045] [c000203978bcbd40] [c000000000cfc150] schedule+0x40/0xc0 [ 226.836123] [c000203978bcbd60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 226.836212] [c000203978bcbdc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 226.836292] [c000203978bcbe30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 226.836379] Instruction dump: [ 226.836426] ebe1fff8 7d908120 4e800020 3860ffea 4bffffe0 60420000 3c4c00c3 3842a330 [ 226.836526] 7c0802a6 60000000 e9230008 e8e90000 <89470430> 2f8a0000 409e0020 a14d0008 [ 226.836628] ---[ end trace 4456a1917aa7e1db ]--- [ 227.653063] [ 227.653112] Oops: Kernel access of bad area, sig: 11 [#12] [ 227.653170] LE SMP NR_CPUS=2048 NUMA PowerNV [ 227.653233] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 227.653680] CPU: 135 PID: 820 Comm: watchdog/135 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 227.653782] NIP: c000000000ac0ce8 LR: c00000000016b260 CTR: c000000000ac0cd0 [ 227.653869] REGS: c000203978bef8a0 TRAP: 0380 Tainted: G D (4.15.0-33-generic) [ 227.653968] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 48000884 XER: 00000000 [ 227.654070] CFAR: c00000000016b25c SOFTE: 0 [ 227.654070] GPR00: c00000000016b260 c000203978befb20 c0000000016eb000 c000203994626fb8 [ 227.654070] GPR04: 000000335711a0b6 0000000000000001 0000002d9d26a24b aa530476c81c9f85 [ 227.654070] GPR08: 0000000000000000 c0000000055fb700 0000203993450000 0000000000000005 [ 227.654070] GPR12: c000000000ac0cd0 c00000000fadcd00 c00000000013d748 c000003fed0e9980 [ 227.654070] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 227.654070] GPR20: 0000000000000000 0000000000000000 c000203978e77480 0000203993450000 [ 227.654070] GPR24: c000203994628580 0000000000000000 c000000000ac0cd0 c000203994626fb8 [ 227.654070] GPR28: c00000000171dd78 0000000000000b60 c000203978e76d80 c000203994628580 [ 227.654838] NIP [c000000000ac0ce8] dbs_update_util_handler+0x18/0xf0 [ 227.654913] LR [c00000000016b260] update_curr_rt+0xd0/0x3f0 [ 227.654971] Call Trace: [ 227.655006] [c000203978befbb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 227.655082] [c000203978befbf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 227.655173] [c000203978befc70] [c000000000cfb9dc] schedule+0x3bc/0xaf0 [ 227.655250] [c000203978befd40] [c000000000cfc150] schedule+0x40/0xc0 [ 227.655327] [c000203978befd60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 227.655416] [c000203978befdc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 227.655494] [c000203978befe30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 227.655596] Instruction dump: [ 227.655642] ebe1fff8 7d908120 4e800020 3860ffea 4bffffe0 60420000 3c4c00c3 3842a330 [ 227.655739] 7c0802a6 60000000 e9230008 e8e90000 <89470430> 2f8a0000 409e0020 a14d0008 [ 227.655840] ---[ end trace 4456a1917aa7e1dc ]--- [ 228.032476] Watchdog CPU:2 Hard LOCKUP [ 228.032477] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 228.032502] CPU: 2 PID: 21 Comm: watchdog/2 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 228.032504] NIP: c000000000180518 LR: c00000000016b2d0 CTR: c000000000ac0cd0 [ 228.032505] REGS: c00000003ffe7d80 TRAP: 0900 Tainted: G D (4.15.0-33-generic) [ 228.032506] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 24c7f282 XER: 00000000 [ 228.032511] CFAR: c000000000180504 SOFTE: 0 [ 228.032511] GPR00: c00000000016b2d0 c000003fe6c97b20 c0000000016eb000 c000003fe6beb980 [ 228.032511] GPR04: 000000000000065e c000003feec14360 c0000000055f2d38 0000000000000001 [ 228.032511] GPR08: 0000000000000003 c00000000159c100 c000003fe6c08400 0000000000000005 [ 228.032511] GPR12: c000000000ac0cd0 c00000000fa81600 c00000000013d748 c000003fed0e5700 [ 228.032511] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 228.032511] GPR20: 0000000000000000 0000000000000000 c000003fe6bec080 0000003feda40000 [ 228.032511] GPR24: c000003feec18580 0000000000000000 c000000000cfc150 c000003feec16fb8 [ 228.032511] GPR28: c00000000171dd78 000000000000065e c000003fe6beb980 c000003feec18580 [ 228.032534] NIP [c000000000180518] cpuacct_charge+0x28/0x80 [ 228.032536] LR [c00000000016b2d0] update_curr_rt+0x140/0x3f0 [ 228.032537] Call Trace: [ 228.032539] [c000003fe6c97b20] [c00000000016b260] update_curr_rt+0xd0/0x3f0 (unreliable) [ 228.032542] [c000003fe6c97bb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 228.032545] [c000003fe6c97bf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 228.032548] [c000003fe6c97c70] [c000000000cfb9dc] schedule+0x3bc/0xaf0 [ 228.032551] [c000003fe6c97d40] [c000000000cfc150] schedule+0x40/0xc0 [ 228.032554] [c000003fe6c97d60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 228.032557] [c000003fe6c97dc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 228.032560] [c000003fe6c97e30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 228.032561] Instruction dump: [ 228.032562] 60000000 60420000 7c0802a6 60000000 38e00001 e9230b40 2fa90000 419e0010 [ 228.032567] e8e90108 68e74000 78e797e2 e9230950 2faa0000 419e0044 e92a00b8 [ 228.475380] [ 228.475434] Oops: Kernel access of bad area, sig: 11 [#13] [ 228.475492] LE SMP NR_CPUS=2048 NUMA PowerNV [ 228.475555] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 228.475997] CPU: 132 PID: 802 Comm: watchdog/132 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 228.476097] NIP: c000000000ac0ce8 LR: c00000000016b260 CTR: c000000000ac0cd0 [ 228.476183] REGS: c000203978bdb8a0 TRAP: 0380 Tainted: G D (4.15.0-33-generic) [ 228.476281] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 48000884 XER: 00000000 [ 228.476380] CFAR: c00000000016b25c SOFTE: 0 [ 228.476380] GPR00: c00000000016b260 c000203978bdbb20 c0000000016eb000 c000203994566fb8 [ 228.476380] GPR04: 00000033556ea036 0000000000000001 0000002d9c501b0c aa530476c81c9f85 [ 228.476380] GPR08: 0000000000000000 c0000000055fb700 0000203993390000 0000000000000005 [ 228.476380] GPR12: c000000000ac0cd0 c00000000fadac00 c00000000013d748 c000003fed0e9800 [ 228.476380] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 228.476380] GPR20: 0000000000000000 0000000000000000 c000203978895880 0000203993390000 [ 228.476380] GPR24: c000203994568580 0000000000000000 c000000000ac0cd0 c000203994566fb8 [ 228.476380] GPR28: c00000000171dd78 00000000000008c8 c000203978895180 c000203994568580 [ 228.477152] NIP [c000000000ac0ce8] dbs_update_util_handler+0x18/0xf0 [ 228.477226] LR [c00000000016b260] update_curr_rt+0xd0/0x3f0 [ 228.477283] Call Trace: [ 228.477318] [c000203978bdbbb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 228.477394] [c000203978bdbbf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 228.477484] [c000203978bdbc70] [c000000000cfb9dc] schedule+0x3bc/0xaf0 [ 228.477560] [c000203978bdbd40] [c000000000cfc150] schedule+0x40/0xc0 [ 228.477637] [c000203978bdbd60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 228.477724] [c000203978bdbdc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 228.477802] [c000203978bdbe30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 228.477887] Instruction dump: [ 228.477948] ebe1fff8 7d908120 4e800020 3860ffea 4bffffe0 60420000 3c4c00c3 3842a330 [ 228.478045] 7c0802a6 60000000 e9230008 e8e90000 <89470430> 2f8a0000 409e0020 a14d0008 [ 228.478144] ---[ end trace 4456a1917aa7e1dd ]--- [ 229.297786] [ 229.297831] Oops: Kernel access of bad area, sig: 11 [#14] [ 229.297888] LE SMP NR_CPUS=2048 NUMA PowerNV [ 229.297950] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 229.298385] CPU: 134 PID: 814 Comm: watchdog/134 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 229.298484] NIP: c000000000ac0ce8 LR: c00000000016b260 CTR: c000000000ac0cd0 [ 229.298585] REGS: c000203978b938a0 TRAP: 0380 Tainted: G D (4.15.0-33-generic) [ 229.298682] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 48000884 XER: 00000000 [ 229.298780] CFAR: c00000000016b25c SOFTE: 0 [ 229.298780] GPR00: c00000000016b260 c000203978b93b20 c0000000016eb000 c0002039945e6fb8 [ 229.298780] GPR04: 00000033565a7caa 0000000000000001 0000002d9cc8ddb6 aa530476c81c9f85 [ 229.298780] GPR08: 0000000000000000 c0000000055fb700 0000203993410000 0000000000000005 [ 229.298780] GPR12: c000000000ac0cd0 c00000000fadc200 c00000000013d748 c000003fed0e9900 [ 229.298780] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 229.298780] GPR20: 0000000000000000 0000000000000000 c000203978e78b00 0000203993410000 [ 229.298780] GPR24: c0002039945e8580 0000000000000000 c000000000ac0cd0 c0002039945e6fb8 [ 229.298780] GPR28: c00000000171dd78 00000000000009ee c000203978e78400 c0002039945e8580 [ 229.299526] NIP [c000000000ac0ce8] dbs_update_util_handler+0x18/0xf0 [ 229.299599] LR [c00000000016b260] update_curr_rt+0xd0/0x3f0 [ 229.299655] Call Trace: [ 229.299689] [c000203978b93bb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 229.299763] [c000203978b93bf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 229.299852] [c000203978b93c70] [c000000000cfb9dc] schedule+0x3bc/0xaf0 [ 229.299944] [c000203978b93d40] [c000000000cfc150] schedule+0x40/0xc0 [ 229.300019] [c000203978b93d60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 229.300106] [c000203978b93dc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 229.300181] [c000203978b93e30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 229.300266] Instruction dump: [ 229.300310] ebe1fff8 7d908120 4e800020 3860ffea 4bffffe0 60420000 3c4c00c3 3842a330 [ 229.300406] 7c0802a6 60000000 e9230008 e8e90000 <89470430> 2f8a0000 409e0020 a14d0008 [ 229.300503] ---[ end trace 4456a1917aa7e1de ]--- [ 230.119591] [ 230.119633] Oops: Kernel access of bad area, sig: 11 [#15] [ 230.119690] LE SMP NR_CPUS=2048 NUMA PowerNV [ 230.119750] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 230.120196] CPU: 59 PID: 363 Comm: watchdog/59 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 230.120292] NIP: c000000000ac0ce8 LR: c00000000016b260 CTR: c000000000ac0cd0 [ 230.120374] REGS: c000003fe35b78a0 TRAP: 0380 Tainted: G D (4.15.0-33-generic) [ 230.120467] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 44000884 XER: 00000000 [ 230.120564] CFAR: c00000000016b25c SOFTE: 0 [ 230.120564] GPR00: c00000000016b260 c000003fe35b7b20 c0000000016eb000 c000003fefa56fb8 [ 230.120564] GPR04: 000000333f391468 0000000000000001 0000002d90f11adc aa530476c81c0f85 [ 230.120564] GPR08: 0000000000000000 c0000000055f4c80 0000003fee880000 0000000000000005 [ 230.120564] GPR12: c000000000ac0cd0 c00000000faa8900 c00000000013d748 c000003fed0e7380 [ 230.120564] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 230.120564] GPR20: 0000000000000000 0000000000000000 c000003fe3499c00 0000003fee880000 [ 230.120564] GPR24: c000003fefa58580 0000000000000000 c000000000ac0cd0 c000003fefa56fb8 [ 230.120564] GPR28: c00000000171dd78 0000000000000b8c c000003fe3499500 c000003fefa58580 [ 230.121305] NIP [c000000000ac0ce8] dbs_update_util_handler+0x18/0xf0 [ 230.121377] LR [c00000000016b260] update_curr_rt+0xd0/0x3f0 [ 230.121431] Call Trace: [ 230.121465] [c000003fe35b7bb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 230.121538] [c000003fe35b7bf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 230.121624] [c000003fe35b7c70] [c000000000cfb9dc] schedule+0x3bc/0xaf0 [ 230.121697] [c000003fe35b7d40] [c000000000cfc150] schedule+0x40/0xc0 [ 230.121771] [c000003fe35b7d60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 230.121855] [c000003fe35b7dc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 230.121929] [c000003fe35b7e30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 230.122010] Instruction dump: [ 230.122055] ebe1fff8 7d908120 4e800020 3860ffea 4bffffe0 60420000 3c4c00c3 3842a330 [ 230.122148] 7c0802a6 60000000 e9230008 e8e90000 <89470430> 2f8a0000 409e0020 a14d0008 [ 230.122244] ---[ end trace 4456a1917aa7e1df ]--- [ 230.942666] [ 230.942711] Oops: Kernel access of bad area, sig: 11 [#16] [ 230.942768] LE SMP NR_CPUS=2048 NUMA PowerNV [ 230.942828] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 230.943250] CPU: 57 PID: 351 Comm: watchdog/57 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 230.943345] NIP: c000000000ac0ce8 LR: c00000000016b260 CTR: c000000000ac0cd0 [ 230.943427] REGS: c000003fe35878a0 TRAP: 0380 Tainted: G D (4.15.0-33-generic) [ 230.943520] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 44000884 XER: 00000000 [ 230.943616] CFAR: c00000000016b25c SOFTE: 0 [ 230.943616] GPR00: c00000000016b260 c000003fe3587b20 c0000000016eb000 c000003fef9d6fb8 [ 230.943616] GPR04: 000000333ec8f80c 0000000000000001 0000002d90b7b479 aa530476c81c0f85 [ 230.943616] GPR08: 0000000000000000 c0000000055f4c80 0000003fee800000 0000000000000005 [ 230.943616] GPR12: c000000000ac0cd0 c00000000faa7300 c00000000013d748 c000003fed0e7280 [ 230.943616] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 230.943616] GPR20: 0000000000000000 0000000000000000 c000003fe3481d80 0000003fee800000 [ 230.943616] GPR24: c000003fef9d8580 0000000000000000 c000000000ac0cd0 c000003fef9d6fb8 [ 230.943616] GPR28: c00000000171dd78 00000000000007c0 c000003fe3481680 c000003fef9d8580 [ 230.944350] NIP [c000000000ac0ce8] dbs_update_util_handler+0x18/0xf0 [ 230.944421] LR [c00000000016b260] update_curr_rt+0xd0/0x3f0 [ 230.944475] Call Trace: [ 230.944508] [c000003fe3587bb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 230.944580] [c000003fe3587bf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 230.944666] [c000003fe3587c70] [c000000000cfb9dc] schedule+0x3bc/0xaf0 [ 230.944739] [c000003fe3587d40] [c000000000cfc150] schedule+0x40/0xc0 [ 230.944812] [c000003fe3587d60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 230.944895] [c000003fe3587dc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 230.944969] [c000003fe3587e30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 230.945049] Instruction dump: [ 230.945093] ebe1fff8 7d908120 4e800020 3860ffea 4bffffe0 60420000 3c4c00c3 3842a330 [ 230.945186] 7c0802a6 60000000 e9230008 e8e90000 <89470430> 2f8a0000 409e0020 a14d0008 [ 230.945281] ---[ end trace 4456a1917aa7e1e0 ]--- [ 231.765947] [ 231.765993] Oops: Kernel access of bad area, sig: 11 [#17] [ 231.766047] LE SMP NR_CPUS=2048 NUMA PowerNV [ 231.766106] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 231.766519] CPU: 58 PID: 357 Comm: watchdog/58 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 231.766612] NIP: c000000000ac0ce8 LR: c00000000016b260 CTR: c000000000ac0cd0 [ 231.766693] REGS: c000003fe359b8a0 TRAP: 0380 Tainted: G D (4.15.0-33-generic) [ 231.766784] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 44000884 XER: 00000000 [ 231.766878] CFAR: c00000000016b25c SOFTE: 0 [ 231.766878] GPR00: c00000000016b260 c000003fe359bb20 c0000000016eb000 c000003fefa16fb8 [ 231.766878] GPR04: 000000333efbfeb4 0000000000000001 0000002d90d1d484 aa530476c81c0f85 [ 231.766878] GPR08: 0000000000000000 c0000000055f4c80 0000003fee840000 0000000000000005 [ 231.766878] GPR12: c000000000ac0cd0 c00000000faa7e00 c00000000013d748 c000003fed0e7300 [ 231.766878] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 231.766878] GPR20: 0000000000000000 0000000000000000 c000003fe34b1a80 0000003fee840000 [ 231.766878] GPR24: c000003fefa18580 0000000000000000 c000000000ac0cd0 c000003fefa16fb8 [ 231.766878] GPR28: c00000000171dd78 0000000000000926 c000003fe34b1380 c000003fefa18580 [ 231.767599] NIP [c000000000ac0ce8] dbs_update_util_handler+0x18/0xf0 [ 231.767669] LR [c00000000016b260] update_curr_rt+0xd0/0x3f0 [ 231.767722] Call Trace: [ 231.767754] [c000003fe359bbb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 231.767824] [c000003fe359bbf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 231.767908] [c000003fe359bc70] [c000000000cfb9dc] schedule+0x3bc/0xaf0 [ 231.767980] [c000003fe359bd40] [c000000000cfc150] schedule+0x40/0xc0 [ 231.768051] [c000003fe359bd60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 231.768133] [c000003fe359bdc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 231.768205] [c000003fe359be30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 231.768284] Instruction dump: [ 231.768326] ebe1fff8 7d908120 4e800020 3860ffea 4bffffe0 60420000 3c4c00c3 3842a330 [ 231.768418] 7c0802a6 60000000 e9230008 e8e90000 <89470430> 2f8a0000 409e0020 a14d0008 [ 231.768526] ---[ end trace 4456a1917aa7e1e1 ]--- [ 232.588765] [ 232.588808] Oops: Kernel access of bad area, sig: 11 [#18] [ 232.588862] LE SMP NR_CPUS=2048 NUMA PowerNV [ 232.588920] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 232.589330] CPU: 56 PID: 345 Comm: watchdog/56 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 232.589422] NIP: c000000000ac0ce8 LR: c00000000016b260 CTR: c000000000ac0cd0 [ 232.589503] REGS: c000003fe35fb8a0 TRAP: 0380 Tainted: G D (4.15.0-33-generic) [ 232.589594] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 44000884 XER: 00000000 [ 232.589702] CFAR: c00000000016b25c SOFTE: 0 [ 232.589702] GPR00: c00000000016b260 c000003fe35fbb20 c0000000016eb000 c000003fef996fb8 [ 232.589702] GPR04: 000000333ec8f7de 0000000000000001 0000002d90b7b445 aa530476c81c0f85 [ 232.589702] GPR08: 0000000000000000 c0000000055f4c80 0000003fee7c0000 0000000000000005 [ 232.589702] GPR12: c000000000ac0cd0 c00000000faa6800 c00000000013d748 c000003fed0e7200 [ 232.589702] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 232.589702] GPR20: 0000000000000000 0000000000000000 c000003fe34a7d00 0000003fee7c0000 [ 232.589702] GPR24: c000003fef998580 0000000000000000 c000000000ac0cd0 c000003fef996fb8 [ 232.589702] GPR28: c00000000171dd78 0000000000000a48 c000003fe34a7600 c000003fef998580 [ 232.590402] NIP [c000000000ac0ce8] dbs_update_util_handler+0x18/0xf0 [ 232.590472] LR [c00000000016b260] update_curr_rt+0xd0/0x3f0 [ 232.590525] Call Trace: [ 232.590557] [c000003fe35fbbb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 232.590627] [c000003fe35fbbf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 232.590710] [c000003fe35fbc70] [c000000000cfb9dc] schedule+0x3bc/0xaf0 [ 232.590781] [c000003fe35fbd40] [c000000000cfc150] schedule+0x40/0xc0 [ 232.590852] [c000003fe35fbd60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 232.590934] [c000003fe35fbdc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 232.591006] [c000003fe35fbe30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 232.591100] Instruction dump: [ 232.591142] ebe1fff8 7d908120 4e800020 3860ffea 4bffffe0 60420000 3c4c00c3 3842a330 [ 232.591233] 7c0802a6 60000000 e9230008 e8e90000 <89470430> 2f8a0000 409e0020 a14d0008 [ 232.591325] ---[ end trace 4456a1917aa7e1e2 ]--- [ 233.411669] [ 233.411709] Oops: Kernel access of bad area, sig: 11 [#19] [ 233.411750] LE SMP NR_CPUS=2048 NUMA PowerNV [ 233.411792] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 233.412083] CPU: 10 PID: 69 Comm: watchdog/10 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 233.412150] NIP: c000000000ac0ce8 LR: c00000000016b260 CTR: c000000000ac0cd0 [ 233.412208] REGS: c000003fe504f8a0 TRAP: 0380 Tainted: G D (4.15.0-33-generic) [ 233.412273] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 44000884 XER: 00000000 [ 233.412352] CFAR: c00000000016b25c SOFTE: 0 [ 233.412352] GPR00: c00000000016b260 c000003fe504fb20 c0000000016eb000 c000003feee16fb8 [ 233.412352] GPR04: 000000333b2a58c6 0000000000000001 0000002d8edd4662 aa530476c81cd0c5 [ 233.412352] GPR08: 0000000000000000 c0000000055fa740 0000003fedc40000 0000000000000005 [ 233.412352] GPR12: c000000000ac0cd0 c00000000fa86e00 c00000000013d748 c000003fed0e5b00 [ 233.412352] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 233.412352] GPR20: 0000000000000000 0000000000000000 c000003fe51e9380 0000003fedc40000 [ 233.412352] GPR24: c000003feee18580 0000000000000000 c000000000ac0cd0 c000003feee16fb8 [ 233.412352] GPR28: c00000000171dd78 00000000000007b4 c000003fe51e8c80 c000003feee18580 [ 233.412893] NIP [c000000000ac0ce8] dbs_update_util_handler+0x18/0xf0 [ 233.412942] LR [c00000000016b260] update_curr_rt+0xd0/0x3f0 [ 233.412981] Call Trace: [ 233.413003] [c000003fe504fbb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 233.413053] [c000003fe504fbf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 233.413113] [c000003fe504fc70] [c000000000cfb9dc] schedule+0x3bc/0xaf0 [ 233.413164] [c000003fe504fd40] [c000000000cfc150] schedule+0x40/0xc0 [ 233.413215] [c000003fe504fd60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 233.413273] [c000003fe504fdc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 233.413324] [c000003fe504fe30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 233.413381] Instruction dump: [ 233.413411] ebe1fff8 7d908120 4e800020 3860ffea 4bffffe0 60420000 3c4c00c3 3842a330 [ 233.413475] 7c0802a6 60000000 e9230008 e8e90000 <89470430> 2f8a0000 409e0020 a14d0008 [ 233.413540] ---[ end trace 4456a1917aa7e1e3 ]--- [ 233.956841] Watchdog CPU:80 detected Hard LOCKUP other CPUS:1,3-8,92 [ 233.957015] Watchdog CPU:1 Hard LOCKUP [ 233.957016] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 233.957033] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 233.957034] NIP: c000000000d02480 LR: c00000000016a760 CTR: c000000000024330 [ 233.957035] REGS: c00000003ffefd80 TRAP: 0100 Tainted: G D (4.15.0-33-generic) [ 233.957036] MSR: 9000000000089033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28004284 XER: 20040000 [ 233.957041] CFAR: c000000000d0249c SOFTE: 0 [ 233.957041] GPR00: c00000000016a760 c000003fed1c3840 c0000000016eb000 c000003feedd8580 [ 233.957041] GPR04: 0000000000000009 0000000000000009 0000000000000000 000000336cc05a99 [ 233.957041] GPR08: 0000000000000009 0000000080000009 0000000080000001 0000000000000000 [ 233.957041] GPR12: c000000000024330 c00000000fa80b00 7265677368657265 0000000000000000 [ 233.957041] GPR16: c000003fed1c3d20 c000003feedd8880 c000003feedd8f20 0000003fedc00000 [ 233.957041] GPR20: 0000000000000000 c0000000018b5338 0000000000000001 0000000000000001 [ 233.957041] GPR24: 0000000000000000 c0000000018b5350 c000000001721ee0 c000000001722494 [ 233.957041] GPR28: 0000000000000000 0000000000000009 c0000000011d8580 c000003feedd8580 [ 233.957061] NIP [c000000000d02480] _raw_spin_lock+0x60/0xe0 [ 233.957063] LR [c00000000016a760] sched_rt_period_timer+0x180/0x4d0 [ 233.957063] Call Trace: [ 233.957066] [c000003fed1c3840] [c000003fed1c3870] 0xc000003fed1c3870 (unreliable) [ 233.957068] [c000003fed1c3870] [c00000000016a6dc] sched_rt_period_timer+0xfc/0x4d0 [ 233.957071] [c000003fed1c3910] [c0000000001b9414] hrtimer_run_queues+0x144/0x370 [ 233.957074] [c000003fed1c3990] [c0000000001ba36c] hrtimer_interrupt+0xfc/0x350 [ 233.957077] [c000003fed1c3a60] [c0000000000247b0] __timer_interrupt+0x90/0x260 [ 233.957079] [c000003fed1c3ab0] [c000000000024bc8] timer_interrupt+0x98/0xe0 [ 233.957082] [c000003fed1c3ae0] [c000000000009104] decrementer_common+0x114/0x120 [ 233.957086] --- interrupt: 901 at replay_interrupt_return+0x0/0x4 [ 233.957086] LR = arch_local_irq_restore+0x74/0x90 [ 233.957087] [c000003fed1c3dd0] [0000000000000001] 0x1 (unreliable) [ 233.957090] [c000003fed1c3df0] [c000000000ac4500] cpuidle_enter_state+0xf0/0x450 [ 233.957093] [c000003fed1c3e50] [c00000000017415c] call_cpuidle+0x4c/0x90 [ 233.957095] [c000003fed1c3e70] [c000000000174570] do_idle+0x2b0/0x330 [ 233.957097] [c000003fed1c3ec0] [c00000000017482c] cpu_startup_entry+0x3c/0x50 [ 233.957100] [c000003fed1c3ef0] [c00000000004a540] start_secondary+0x4f0/0x510 [ 233.957102] [c000003fed1c3f90] [c00000000000ab6c] start_secondary_prolog+0x10/0x14 [ 233.957103] Instruction dump: [ 233.957104] 40c20010 7d40192d 40c2fff0 7c2004ac 2fa90000 409e001c 38210030 ebe1fff8 [ 233.957108] 4e800020 60000000 60000000 60420000 <7c210b78> e92d0000 89290009 71290002 [ 233.967126] Watchdog CPU:3 Hard LOCKUP [ 233.967126] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 233.967142] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 233.967143] NIP: c0000000000a5514 LR: c0000000000a5514 CTR: c000000000008000 [ 233.967144] REGS: c000003fed1cbc40 TRAP: 0100 Tainted: G D (4.15.0-33-generic) [ 233.967145] MSR: 9000000000001033 <SF,HV,ME,IR,DR,RI,LE> CR: 24004482 XER: 00000000 [ 233.967149] CFAR: c000003fed1cbda0 SOFTE: 3 [ 233.967149] GPR00: c0000000000a5514 c000003fed1cbda0 c0000000016eb000 c000003fed1cbc40 [ 233.967149] GPR04: b000000000001033 c0000000000a4f00 0000000024004484 0000000000000126 [ 233.967149] GPR08: c00020397b33c780 c000000000d10f08 00000000000000ff c000003feeb7cb38 [ 233.967149] GPR12: 9000000000121033 c00000000fa82100 c000003fed1cbf90 0000000000000000 [ 233.967149] GPR16: 0000000000000000 c000000000047fe0 c000000000047fb0 c0000000011b5380 [ 233.967149] GPR20: 0000000000000800 c000000001722494 0000000000000003 0000000000000000 [ 233.967149] GPR24: 000000000000000c 0000000000000008 0000000000000000 0000000000000008 [ 233.967149] GPR28: c00000000161caa8 c000003feec56fd8 c00000000161cdc0 0000000000000008 [ 233.967166] NIP [c0000000000a5514] power9_idle_type+0x24/0x40 [ 233.967168] LR [c0000000000a5514] power9_idle_type+0x24/0x40 [ 233.967168] Call Trace: [ 233.967170] [c000003fed1cbda0] [c0000000000a5514] power9_idle_type+0x24/0x40 (unreliable) [ 233.967172] [c000003fed1cbdc0] [c000000000ac7d60] stop_loop+0x40/0x5c [ 233.967174] [c000003fed1cbdf0] [c000000000ac44b4] cpuidle_enter_state+0xa4/0x450 [ 233.967177] [c000003fed1cbe50] [c00000000017415c] call_cpuidle+0x4c/0x90 [ 233.967179] [c000003fed1cbe70] [c000000000174570] do_idle+0x2b0/0x330 [ 233.967181] [c000003fed1cbec0] [c00000000017482c] cpu_startup_entry+0x3c/0x50 [ 233.967183] [c000003fed1cbef0] [c00000000004a540] start_secondary+0x4f0/0x510 [ 233.967185] [c000003fed1cbf90] [c00000000000ab6c] start_secondary_prolog+0x10/0x14 [ 233.967186] Instruction dump: [ 233.967187] ebe1fff8 7c0803a6 4e800020 3c4c0164 38425b10 7c0802a6 60000000 7c0802a6 [ 233.967191] f8010010 f821ffe1 4bfff97d 4bf71b69 <60000000> 38210020 e8010010 7c0803a6 [ 234.225947] ^C [ 238.503312] Watchdog CPU:8 Hard LOCKUP [ 238.503313] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 238.503341] CPU: 8 PID: 0 Comm: swapper/8 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 238.503343] NIP: c000000000d02480 LR: c000000000151318 CTR: c000000000024330 [ 238.503345] REGS: c00000003ff9bd80 TRAP: 0100 Tainted: G D (4.15.0-33-generic) [ 238.503346] MSR: 9000000000089033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28004284 XER: 00000000 [ 238.503354] CFAR: c000000000d0249c SOFTE: 0 [ 238.503354] GPR00: c000000000151318 c000003fed1df7e0 c0000000016eb000 c000003feed98580 [ 238.503354] GPR04: 0000000000000200 ffffffffffffffff 00000000ffffb0a9 0000000000000000 [ 238.503354] GPR08: c000000001721ee0 0000000080000029 0000000080000008 00000000000001b9 [ 238.503354] GPR12: 0000000000004000 c00000000fa85800 c000003fed1dff90 0000000000000000 [ 238.503354] GPR16: 0000000000000000 c000000000047fe0 0000000000000000 0000003fedbc0000 [ 238.503354] GPR20: c000003feed835c0 0000000000000000 0000000000000000 c0000000001d0650 [ 238.503354] GPR24: 0000000000000000 c000003fe69fb100 0000003fedbc0000 c00000000171dd78 [ 238.503354] GPR28: c0000000011d8580 0000000000000040 c0000000011d8580 c000003feed98580 [ 238.503386] NIP [c000000000d02480] _raw_spin_lock+0x60/0xe0 [ 238.503390] LR [c000000000151318] scheduler_tick+0x68/0x170 [ 238.503390] Call Trace: [ 238.503392] [c000003fed1df810] [0000000000000000] (null) [ 238.503396] [c000003fed1df870] [c0000000001b8888] update_process_times+0x68/0x90 [ 238.503400] [c000003fed1df8a0] [c0000000001d0628] tick_sched_handle.isra.5+0xa8/0xd0 [ 238.503403] [c000003fed1df8d0] [c0000000001d06b0] tick_sched_timer+0x60/0xe0 [ 238.503407] [c000003fed1df910] [c0000000001b9414] hrtimer_run_queues+0x144/0x370 [ 238.503411] [c000003fed1df990] [c0000000001ba36c] hrtimer_interrupt+0xfc/0x350 [ 238.503416] [c000003fed1dfa60] [c0000000000247b0] timer_interrupt+0x90/0x260 [ 238.503420] [c000003fed1dfab0] [c000000000024bc8] timer_interrupt+0x98/0xe0 [ 238.503424] [c000003fed1dfae0] [c000000000009104] decrementer_common+0x114/0x120 [ 238.503431] --- interrupt: 901 at replay_interrupt_return+0x0/0x4 [ 238.503431] LR = arch_local_irq_restore+0x74/0x90 [ 238.503433] [c000003fed1dfdd0] [0000000000000008] 0x8 (unreliable) [ 238.503438] [c000003fed1dfdf0] [c000000000ac4500] cpuidle_enter_state+0xf0/0x450 [ 238.503442] [c000003fed1dfe50] [c00000000017415c] call_cpuidle+0x4c/0x90 [ 238.503446] [c000003fed1dfe70] [c000000000174570] do_idle+0x2b0/0x330 [ 238.503450] [c000003fed1dfec0] [c00000000017482c] cpu_startup_entry+0x3c/0x50 [ 238.503454] [c000003fed1dfef0] [c00000000004a540] start_secondary+0x4f0/0x510 [ 238.503458] [c000003fed1dff90] [c00000000000ab6c] start_secondary_prolog+0x10/0x14 [ 238.503459] Instruction dump: [ 238.503461] 40c20010 7d40192d 40c2fff0 7c2004ac 2fa90000 409e001c 38210030 ebe1fff8 [ 238.503468] 4e800020 60000000 60000000 60420000 <7c210b78> e92d0000 89290009 71290002 [ 238.513631] Watchdog CPU:92 Hard LOCKUP [ 238.513633] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 238.513677] CPU: 92 PID: 0 Comm: swapper/92 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 238.513680] NIP: c000000000d02480 LR: c000000000151318 CTR: c000000000024330 [ 238.513683] REGS: c00000003fbabd80 TRAP: 0100 Tainted: G D (4.15.0-33-generic) [ 238.513685] MSR: 9000000000089033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28004284 XER: 00000000 [ 238.513697] CFAR: c000000000d0249c SOFTE: 0 [ 238.513697] GPR00: c000000000151318 c00020397b3cb7e0 c0000000016eb000 c000203993b68580 [ 238.513697] GPR04: 0000000000000200 ffffffffffffffff 00000000ffffb0a9 0000000000000008 [ 238.513697] GPR08: c000000001721ee0 0000000080000050 000000008000005c 00000000000001b9 [ 238.513697] GPR12: 0000000000004000 c00000000fabf400 c00020397b3cbf90 0000000000000000 [ 238.513697] GPR16: 0000000000000000 c000000000047fe0 0000000000000000 0000203992990000 [ 238.513697] GPR20: c000203993b535c0 0000000000000000 0000000000000000 c0000000001d0650 [ 238.513697] GPR24: 0000000000000000 c00020397b33c780 0000203992990000 c00000000171dd78 [ 238.513697] GPR28: c0000000011d8580 00000000000002e0 c0000000011d8580 c000203993b68580 [ 238.513745] NIP [c000000000d02480] _raw_spin_lock+0x60/0xe0 [ 238.513750] LR [c000000000151318] scheduler_tick+0x68/0x170 [ 238.513751] Call Trace: [ 238.513753] [c00020397b3cb810] [0000000000000000] (null) [ 238.513761] [c00020397b3cb870] [c0000000001b8888] update_process_times+0x68/0x90 [ 238.513767] [c00020397b3cb8a0] [c0000000001d0628] tick_sched_handle.isra.5+0xa8/0xd0 [ 238.513771] [c00020397b3cb8d0] [c0000000001d06b0] tick_sched_timer+0x60/0xe0 [ 238.513777] [c00020397b3cb910] [c0000000001b9414] __hrtimer_run_queues+0x144/0x370 [ 238.513783] [c00020397b3cb990] [c0000000001ba36c] hrtimer_interrupt+0xfc/0x350 [ 238.513790] [c00020397b3cba60] [c0000000000247b0] timer_interrupt+0x90/0x260 [ 238.513797] [c00020397b3cbab0] [c000000000024bc8] timer_interrupt+0x98/0xe0 [ 238.513803] [c00020397b3cbae0] [c000000000009104] decrementer_common+0x114/0x120 [ 238.513813] --- interrupt: 901 at replay_interrupt_return+0x0/0x4 [ 238.513813] LR = arch_local_irq_restore+0x74/0x90 [ 238.513817] [c00020397b3cbdd0] [000000000000005c] 0x5c (unreliable) [ 238.513825] [c00020397b3cbdf0] [c000000000ac4500] cpuidle_enter_state+0xf0/0x450 [ 238.513831] [c00020397b3cbe50] [c00000000017415c] call_cpuidle+0x4c/0x90 [ 238.513837] [c00020397b3cbe70] [c000000000174570] do_idle+0x2b0/0x330 [ 238.513844] [c00020397b3cbec0] [c00000000017482c] cpu_startup_entry+0x3c/0x50 [ 238.513850] [c00020397b3cbef0] [c00000000004a540] start_secondary+0x4f0/0x510 [ 238.513856] [c00020397b3cbf90] [c00000000000ab6c] start_secondary_prolog+0x10/0x14 [ 238.513858] Instruction dump: [ 238.513861] 40c20010 7d40192d 40c2fff0 7c2004ac 2fa90000 409e001c 38210030 ebe1fff8 [ 238.513871] 4e800020 60000000 60000000 60420000 <7c210b78> e92d0000 89290009 71290002 [ 238.524324] login[3334]: unhandled signal 11 at 5b0304a9803dcb65 nip 00007d785286314c lr 00007d7852a6a1c4 code 3 [ 239.769152] INFO: rcu_sched detected stalls on CPUs/tasks: [ 239.769193] 0-...0: (1 GPs behind) idle=aae/1/0 softirq=1031/1031 fqs=2176 [ 239.769235] (detected by 44, t=5252 jiffies, g=441, c=440, q=3775) [ 239.769274] Task dump for CPU 0: [ 239.769296] swapper/0 R running task 0 0 0 0x00000004 [ 239.769339] Call Trace: [ 239.769357] [c0000000016e7b30] [c0000000016eb000] .TOC.+0x0/0x1000 (unreliable) [ 248.425544] Watchdog CPU:12 Hard LOCKUP [ 248.425545] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 248.425557] CPU: 12 PID: 0 Comm: swapper/12 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 248.425559] NIP: c000000000d02480 LR: c00000000014f32c CTR: c00000000015f8f0 [ 248.425559] REGS: c00000003ff6fd80 TRAP: 0900 Tainted: G D (4.15.0-33-generic) [ 248.425560] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 28004884 XER: 00000000 [ 248.425563] CFAR: c000000000d0249c SOFTE: 0 [ 248.425563] GPR00: c00000000014f32c c000003fed1ef750 c0000000016eb000 c000003feed98580 [ 248.425563] GPR04: c000003feee98600 c000003feed98600 0000000000000000 0000000000000000 [ 248.425563] GPR08: 0000003fedcc0000 0000000080000029 000000008000000c c000003feee7ca38 [ 248.425563] GPR12: c00000000015f8f0 c00000000fa88400 c000003fed1eff90 0000000000000000 [ 248.425563] GPR16: 0000000000000000 0000000000000100 0000000000000001 0000000000000010 [ 248.425563] GPR20: c000000001712208 0000000000200042 0000000000000040 0000003fedbc0000 [ 248.425563] GPR24: c0000000011d8580 c00000000171dd78 c000003feed98580 0000000000000001 [ 248.425563] GPR28: 0000000000000004 c000003fd7a44be4 0000000000000008 c000003feed98580 [ 248.425575] NIP [c000000000d02480] _raw_spin_lock+0x60/0xe0 [ 248.425576] LR [c00000000014f32c] try_to_wake_up+0x1fc/0x660 [ 248.425576] Call Trace: [ 248.425577] [c000003fed1ef750] [0000000000000004] 0x4 (unreliable) [ 248.425579] [c000003fed1ef780] [c00000000014f2ec] try_to_wake_up+0x1bc/0x660 [ 248.425580] [c000003fed1ef800] [c0000000001b6a04] process_timeout+0x24/0x40 [ 248.425582] [c000003fed1ef820] [c0000000001b6a70] call_timer_fn+0x50/0x1c0 [ 248.425583] [c000003fed1ef8a0] [c0000000001b6d18] expire_timers+0x138/0x1f0 [ 248.425585] [c000003fed1ef910] [c0000000001b6ed8] run_timer_softirq+0x108/0x270 [ 248.425586] [c000003fed1ef9b0] [c000000000d03268] do_softirq+0x158/0x3e4 [ 248.425588] [c000003fed1efa90] [c0000000001168c8] irq_exit+0xe8/0x120 [ 248.425590] [c000003fed1efab0] [c000000000024bcc] timer_interrupt+0x9c/0xe0 [ 248.425591] [c000003fed1efae0] [c000000000009104] decrementer_common+0x114/0x120 [ 248.425594] --- interrupt: 901 at replay_interrupt_return+0x0/0x4 [ 248.425594] LR = arch_local_irq_restore+0x74/0x90 [ 248.425595] [c000003fed1efdd0] [000000000000000c] 0xc (unreliable) [ 248.425597] [c000003fed1efdf0] [c000000000ac4500] cpuidle_enter_state+0xf0/0x450 [ 248.425599] [c000003fed1efe50] [c00000000017415c] call_cpuidle+0x4c/0x90 [ 248.425600] [c000003fed1efe70] [c000000000174570] do_idle+0x2b0/0x330 [ 248.425602] [c000003fed1efec0] [c00000000017482c] cpu_startup_entry+0x3c/0x50 [ 248.425603] [c000003fed1efef0] [c00000000004a540] start_secondary+0x4f0/0x510 [ 248.425605] [c000003fed1eff90] [c00000000000ab6c] start_secondary_prolog+0x10/0x14 [ 248.425606] Instruction dump: [ 248.425607] 40c20010 7d40192d 40c2fff0 7c2004ac 2fa90000 409e001c 38210030 ebe1fff8 [ 248.425609] 4e800020 60000000 60000000 60420000 <7c210b78> e92d0000 89290009 71290002 [ 252.645789] Watchdog CPU:15 detected Hard LOCKUP other CPUS:9-11,48,56-59,93-95,132-135 [ 252.647887] Watchdog CPU:9 Hard LOCKUP [ 252.647888] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 252.647914] CPU: 9 PID: 63 Comm: watchdog/9 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 252.647915] NIP: c000000000d02480 LR: c000000000151318 CTR: c000000000444d00 [ 252.647917] REGS: c00000003ff8fd80 TRAP: 0100 Tainted: G D (4.15.0-33-generic) [ 252.647918] MSR: 9000000002089033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE> CR: 28008484 XER: 00000000 [ 252.647926] CFAR: c000000000d0249c SOFTE: 0 [ 252.647926] GPR00: c000000000151318 c000003fe5047050 c0000000016eb000 c000003feedd8580 [ 252.647926] GPR04: 0000000000000000 0000000000000000 0000000000000000 000000337dc6f7ba [ 252.647926] GPR08: c000003fce7ed9e0 0000000080000009 0000000080000009 c000003ebf1ee730 [ 252.647926] GPR12: 0000000000008000 c00000000fa86300 c00000000013d748 c000003fed0e5a80 [ 252.647926] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000003fedc00000 [ 252.647926] GPR20: c000003feedc35c0 0000000000000000 0000000000000000 c0000000001d0650 [ 252.647926] GPR24: 0000000000000000 c000003fe51c4380 0000003fedc00000 c00000000171dd78 [ 252.647926] GPR28: c0000000011d8580 0000000000000048 c0000000011d8580 c000003feedd8580 [ 252.647954] NIP [c000000000d02480] _raw_spin_lock+0x60/0xe0 [ 252.647957] LR [c000000000151318] scheduler_tick+0x68/0x170 [ 252.647957] Call Trace: [ 252.647959] [c000003fe5047050] [0000000000000004] 0x4 (unreliable) [ 252.647961] [c000003fe5047080] [0000000000000000] (null) [ 252.647964] [c000003fe50470e0] [c0000000001b8888] update_process_times+0x68/0x90 [ 252.647967] [c000003fe5047110] [c0000000001d05b4] tick_sched_handle.isra.5+0x34/0xd0 [ 252.647970] [c000003fe5047140] [c0000000001d06b0] tick_sched_timer+0x60/0xe0 [ 252.647974] [c000003fe5047180] [c0000000001b9414] hrtimer_run_queues+0x144/0x370 [ 252.647977] [c000003fe5047200] [c0000000001ba36c] hrtimer_interrupt+0xfc/0x350 [ 252.647981] [c000003fe50472d0] [c0000000000247b0] timer_interrupt+0x90/0x260 [ 252.647984] [c000003fe5047320] [c000000000024bc8] timer_interrupt+0x98/0xe0 [ 252.647988] [c000003fe5047350] [c000000000009104] decrementer_common+0x114/0x120 [ 252.647994] --- interrupt: 901 at replay_interrupt_return+0x0/0x4 [ 252.647994] LR = arch_local_irq_restore+0x74/0x90 [ 252.647996] [c000003fe5047640] [000000000000000b] 0xb (unreliable) [ 252.647999] [c000003fe5047660] [c0000000001142d4] do_exit+0xf4/0xc80 [ 252.648003] [c000003fe5047720] [c00000000002606c] oops_end+0x17c/0x210 [ 252.648007] [c000003fe50477a0] [c00000000006b2a0] bad_page_fault+0xe0/0x160 [ 252.648009] [c000003fe5047810] [c000000000027654] slb_miss_bad_addr+0x34/0x70 [ 252.648013] [c000003fe5047830] [c000000000008b0c] bad_addr_slb+0x11c/0x120 [ 252.648018] --- interrupt: 380 at dbs_update_util_handler+0x18/0xf0 [ 252.648018] LR = update_curr_rt+0xd0/0x3f0 [ 252.648020] [c000003fe5047b20] [0000000000000000] (null) (unreliable) [ 252.648023] [c000003fe5047bb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 252.648026] [c000003fe5047bf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 252.648030] [c000003fe5047c70] [c000000000cfb9dc] __schedule+0x3bc/0xaf0 [ 252.648033] [c000003fe5047d40] [c000000000cfc150] schedule+0x40/0xc0 [ 252.648037] [c000003fe5047d60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 252.648040] [c000003fe5047dc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 252.648044] [c000003fe5047e30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 252.648045] Instruction dump: [ 252.648046] 40c20010 7d40192d 40c2fff0 7c2004ac 2fa90000 409e001c 38210030 ebe1fff8 [ 252.648052] 4e800020 60000000 60000000 60420000 <7c210b78> e92d0000 89290009 71290002 [ 252.657996] Watchdog CPU:10 Hard LOCKUP [ 252.657997] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 252.658022] CPU: 10 PID: 69 Comm: watchdog/10 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 252.658024] NIP: c000000000d02480 LR: c000000000151318 CTR: c000000000444d00 [ 252.658025] REGS: c00000003ff83d80 TRAP: 0100 Tainted: G D (4.15.0-33-generic) [ 252.658026] MSR: 9000000002089033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE> CR: 28008484 XER: 00000000 [ 252.658033] CFAR: c000000000d0249c SOFTE: 0 [ 252.658033] GPR00: c000000000151318 c000003fe504f050 c0000000016eb000 c000003feee18580 [ 252.658033] GPR04: 0000000000000000 0000000000000000 0000000000000000 ffffffffffffffff [ 252.658033] GPR08: c000003fce7ed9e0 000000008000000a 000000008000000a 00000000000001b9 [ 252.658033] GPR12: 0000000000008000 c00000000fa86e00 c00000000013d748 c000003fed0e5b00 [ 252.658033] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000003fedc40000 [ 252.658033] GPR20: c000003feee035c0 0000000000000000 0000000000000000 c0000000001d0650 [ 252.658033] GPR24: 0000000000000000 c000003fe51e8c80 0000003fedc40000 c00000000171dd78 [ 252.658033] GPR28: c0000000011d8580 0000000000000050 c0000000011d8580 c000003feee18580 [ 252.658061] NIP [c000000000d02480] _raw_spin_lock+0x60/0xe0 [ 252.658064] LR [c000000000151318] scheduler_tick+0x68/0x170 [ 252.658064] Call Trace: [ 252.658066] [c000003fe504f050] [0000000000000004] 0x4 (unreliable) [ 252.658068] [c000003fe504f080] [0000000000000000] (null) [ 252.658071] [c000003fe504f0e0] [c0000000001b8888] update_process_times+0x68/0x90 [ 252.658074] [c000003fe504f110] [c0000000001d05b4] tick_sched_handle.isra.5+0x34/0xd0 [ 252.658077] [c000003fe504f140] [c0000000001d06b0] tick_sched_timer+0x60/0xe0 [ 252.658080] [c000003fe504f180] [c0000000001b9414] hrtimer_run_queues+0x144/0x370 [ 252.658084] [c000003fe504f200] [c0000000001ba36c] hrtimer_interrupt+0xfc/0x350 [ 252.658087] [c000003fe504f2d0] [c0000000000247b0] timer_interrupt+0x90/0x260 [ 252.658091] [c000003fe504f320] [c000000000024bc8] timer_interrupt+0x98/0xe0 [ 252.658095] [c000003fe504f350] [c000000000009104] decrementer_common+0x114/0x120 [ 252.658101] --- interrupt: 901 at replay_interrupt_return+0x0/0x4 [ 252.658101] LR = arch_local_irq_restore+0x74/0x90 [ 252.658102] [c000003fe504f640] [000000000000000b] 0xb (unreliable) [ 252.658105] [c000003fe504f660] [c0000000001142d4] do_exit+0xf4/0xc80 [ 252.658109] [c000003fe504f720] [c00000000002606c] oops_end+0x17c/0x210 [ 252.658112] [c000003fe504f7a0] [c00000000006b2a0] bad_page_fault+0xe0/0x160 [ 252.658115] [c000003fe504f810] [c000000000027654] slb_miss_bad_addr+0x34/0x70 [ 252.658119] [c000003fe504f830] [c000000000008b0c] bad_addr_slb+0x11c/0x120 [ 252.658124] --- interrupt: 380 at dbs_update_util_handler+0x18/0xf0 [ 252.658124] LR = update_curr_rt+0xd0/0x3f0 [ 252.658125] [c000003fe504fb20] [0000000000000000] (null) (unreliable) [ 252.658128] [c000003fe504fbb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 252.658131] [c000003fe504fbf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 252.658135] [c000003fe504fc70] [c000000000cfb9dc] __schedule+0x3bc/0xaf0 [ 252.658139] [c000003fe504fd40] [c000000000cfc150] schedule+0x40/0xc0 [ 252.658142] [c000003fe504fd60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 252.658145] [c000003fe504fdc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 252.658149] [c000003fe504fe30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 252.658150] Instruction dump: [ 252.658151] 40c20010 7d40192d 40c2fff0 7c2004ac 2fa90000 409e001c 38210030 ebe1fff8 [ 252.658157] 4e800020 60000000 60000000 60420000 <7c210b78> e92d0000 89290009 71290002 [ 252.668106] Watchdog CPU:11 Hard LOCKUP [ 252.668106] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 252.668131] CPU: 11 PID: 75 Comm: watchdog/11 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 252.668133] NIP: c000000000d02480 LR: c000000000151318 CTR: c000000000444d00 [ 252.668135] REGS: c00000003ff77d80 TRAP: 0100 Tainted: G D (4.15.0-33-generic) [ 252.668135] MSR: 9000000002089033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE> CR: 28008484 XER: 00000000 [ 252.668143] CFAR: c000000000d0249c SOFTE: 0 [ 252.668143] GPR00: c000000000151318 c000003fe5043050 c0000000016eb000 c000003feee58580 [ 252.668143] GPR04: 0000000000000000 0000000000000000 0000000000000000 00000034a0572c5e [ 252.668143] GPR08: c000003fce7ed9e0 000000008000000b 000000008000000b c000003ebf1ee730 [ 252.668143] GPR12: 0000000000008000 c00000000fa87900 c00000000013d748 c000003fed0e5b80 [ 252.668143] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000003fedc80000 [ 252.668143] GPR20: c000003feee435c0 0000000000000000 0000000000000000 c0000000001d0650 [ 252.668143] GPR24: 0000000000000000 c000003fe51e7600 0000003fedc80000 c00000000171dd78 [ 252.668143] GPR28: c0000000011d8580 0000000000000058 c0000000011d8580 c000003feee58580 [ 252.668171] NIP [c000000000d02480] _raw_spin_lock+0x60/0xe0 [ 252.668173] LR [c000000000151318] scheduler_tick+0x68/0x170 [ 252.668174] Call Trace: [ 252.668175] [c000003fe5043050] [0000000000000004] 0x4 (unreliable) [ 252.668177] [c000003fe5043080] [0000000000000000] (null) [ 252.668181] [c000003fe50430e0] [c0000000001b8888] update_process_times+0x68/0x90 [ 252.668184] [c000003fe5043110] [c0000000001d05b4] tick_sched_handle.isra.5+0x34/0xd0 [ 252.668186] [c000003fe5043140] [c0000000001d06b0] tick_sched_timer+0x60/0xe0 [ 252.668190] [c000003fe5043180] [c0000000001b9414] hrtimer_run_queues+0x144/0x370 [ 252.668194] [c000003fe5043200] [c0000000001ba36c] hrtimer_interrupt+0xfc/0x350 [ 252.668197] [c000003fe50432d0] [c0000000000247b0] timer_interrupt+0x90/0x260 [ 252.668201] [c000003fe5043320] [c000000000024bc8] timer_interrupt+0x98/0xe0 [ 252.668204] [c000003fe5043350] [c000000000009104] decrementer_common+0x114/0x120 [ 252.668210] --- interrupt: 901 at replay_interrupt_return+0x0/0x4 [ 252.668210] LR = arch_local_irq_restore+0x74/0x90 [ 252.668211] [c000003fe5043640] [000000000000000b] 0xb (unreliable) [ 252.668215] [c000003fe5043660] [c0000000001142d4] do_exit+0xf4/0xc80 [ 252.668218] [c000003fe5043720] [c00000000002606c] oops_end+0x17c/0x210 [ 252.668222] [c000003fe50437a0] [c00000000006b2a0] bad_page_fault+0xe0/0x160 [ 252.668224] [c000003fe5043810] [c000000000027654] slb_miss_bad_addr+0x34/0x70 [ 252.668228] [c000003fe5043830] [c000000000008b0c] bad_addr_slb+0x11c/0x120 [ 252.668233] --- interrupt: 380 at dbs_update_util_handler+0x18/0xf0 [ 252.668233] LR = update_curr_rt+0xd0/0x3f0 [ 252.668234] [c000003fe5043b20] [0000000000000000] (null) (unreliable) [ 252.668237] [c000003fe5043bb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 252.668240] [c000003fe5043bf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 252.668244] [c000003fe5043c70] [c000000000cfb9dc] __schedule+0x3bc/0xaf0 [ 252.668247] [c000003fe5043d40] [c000000000cfc150] schedule+0x40/0xc0 [ 252.668251] [c000003fe5043d60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 252.668254] [c000003fe5043dc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 252.668258] [c000003fe5043e30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 252.668259] Instruction dump: [ 252.668260] 40c20010 7d40192d 40c2fff0 7c2004ac 2fa90000 409e001c 38210030 ebe1fff8 [ 252.668266] 4e800020 60000000 60000000 60420000 <7c210b78> e92d0000 89290009 71290002 [ 252.678216] Watchdog CPU:48 Hard LOCKUP [ 252.678216] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 252.678228] CPU: 48 PID: 297 Comm: watchdog/48 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 252.678229] NIP: c000000000d02484 LR: c000000000151318 CTR: c000000000444d00 [ 252.678230] REGS: c00000003fdbbd80 TRAP: 0100 Tainted: G D (4.15.0-33-generic) [ 252.678230] MSR: 9000000002089033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE> CR: 28000484 XER: 00000000 [ 252.678233] CFAR: c000000000d0249c SOFTE: 0 [ 252.678233] GPR00: c000000000151318 c000003fe4d2f070 c0000000016eb000 c000003fef798580 [ 252.678233] GPR04: 0000000000000000 0000000000000000 0000000000000000 00000034d059aff2 [ 252.678233] GPR08: c000003fce7ed9e0 0000000080000030 0000000080000030 0000000000000000 [ 252.678233] GPR12: 0000000000000000 c00000000faa1000 c00000000013d748 c000003fed0e6e00 [ 252.678233] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000003fee5c0000 [ 252.678233] GPR20: c000003fef7835c0 0000000000000000 0000000000000000 c0000000001d0650 [ 252.678233] GPR24: 0000000000000000 c000003fe4e84380 0000003fee5c0000 c00000000171dd78 [ 252.678233] GPR28: c0000000011d8580 0000000000000180 c0000000011d8580 c000003fef798580 [ 252.678246] NIP [c000000000d02484] _raw_spin_lock+0x64/0xe0 [ 252.678247] LR [c000000000151318] scheduler_tick+0x68/0x170 [ 252.678247] Call Trace: [ 252.678248] [c000003fe4d2f070] [0000000000000004] 0x4 (unreliable) [ 252.678249] [c000003fe4d2f0a0] [0000000000000000] (null) [ 252.678251] [c000003fe4d2f100] [c0000000001b8888] update_process_times+0x68/0x90 [ 252.678253] [c000003fe4d2f130] [c0000000001d05b4] tick_sched_handle.isra.5+0x34/0xd0 [ 252.678254] [c000003fe4d2f160] [c0000000001d06b0] tick_sched_timer+0x60/0xe0 [ 252.678256] [c000003fe4d2f1a0] [c0000000001b9414] hrtimer_run_queues+0x144/0x370 [ 252.678258] [c000003fe4d2f220] [c0000000001ba36c] hrtimer_interrupt+0xfc/0x350 [ 252.678260] [c000003fe4d2f2f0] [c0000000000247b0] timer_interrupt+0x90/0x260 [ 252.678261] [c000003fe4d2f340] [c000000000024bc8] timer_interrupt+0x98/0xe0 [ 252.678263] [c000003fe4d2f370] [c000000000009104] decrementer_common+0x114/0x120 [ 252.678266] --- interrupt: 901 at replay_interrupt_return+0x0/0x4 [ 252.678266] LR = arch_local_irq_restore+0x74/0x90 [ 252.678267] [c000003fe4d2f660] [000000000000000b] 0xb (unreliable) [ 252.678268] [c000003fe4d2f680] [c0000000001142d4] do_exit+0xf4/0xc80 [ 252.678270] [c000003fe4d2f740] [c00000000002606c] oops_end+0x17c/0x210 [ 252.678271] [c000003fe4d2f7c0] [c00000000006b2a0] bad_page_fault+0xe0/0x160 [ 252.678273] [c000003fe4d2f830] [c00000000000a650] handle_page_fault+0x34/0x38 [ 252.678275] --- interrupt: 400 at (null) [ 252.678275] LR = update_curr_rt+0xd0/0x3f0 [ 252.678275] [c000003fe4d2fb20] [0000000000000000] (null) (unreliable) [ 252.678277] [c000003fe4d2fbb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 252.678278] [c000003fe4d2fbf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 252.678280] [c000003fe4d2fc70] [c000000000cfb9dc] __schedule+0x3bc/0xaf0 [ 252.678282] [c000003fe4d2fd40] [c000000000cfc150] schedule+0x40/0xc0 [ 252.678283] [c000003fe4d2fd60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 252.678285] [c000003fe4d2fdc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 252.678286] [c000003fe4d2fe30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 252.678287] Instruction dump: [ 252.678288] 7d40192d 40c2fff0 7c2004ac 2fa90000 409e001c 38210030 ebe1fff8 4e800020 [ 252.678290] 60000000 60000000 60420000 7c210b78 89290009 71290002 40820050 [ 252.688327] Watchdog CPU:56 Hard LOCKUP [ 252.688329] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 252.688371] CPU: 56 PID: 345 Comm: watchdog/56 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 252.688375] NIP: c000000000d02480 LR: c000000000151318 CTR: c000000000444d00 [ 252.688378] REGS: c00000003fd5bd80 TRAP: 0100 Tainted: G D (4.15.0-33-generic) [ 252.688379] MSR: 9000000002089033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE> CR: 28008484 XER: 00000000 [ 252.688392] CFAR: c000000000d0249c SOFTE: 0 [ 252.688392] GPR00: c000000000151318 c000003fe35fb050 c0000000016eb000 c000003fef998580 [ 252.688392] GPR04: 0000000000000000 0000000000000000 0000000000000000 00000036586c12e0 [ 252.688392] GPR08: c000003fce7ed9e0 0000000080000038 0000000080000038 c000003ebf1ee730 [ 252.688392] GPR12: 0000000000008000 c00000000faa6800 c00000000013d748 c000003fed0e7200 [ 252.688392] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000003fee7c0000 [ 252.688392] GPR20: c000003fef9835c0 0000000000000000 0000000000000000 c0000000001d0650 [ 252.688392] GPR24: 0000000000000000 c000003fe34a7600 0000003fee7c0000 c00000000171dd78 [ 252.688392] GPR28: c0000000011d8580 00000000000001c0 c0000000011d8580 c000003fef998580 [ 252.688440] NIP [c000000000d02480] _raw_spin_lock+0x60/0xe0 [ 252.688445] LR [c000000000151318] scheduler_tick+0x68/0x170 [ 252.688446] Call Trace: [ 252.688449] [c000003fe35fb050] [0000000000000004] 0x4 (unreliable) [ 252.688453] [c000003fe35fb080] [0000000000000000] (null) [ 252.688459] [c000003fe35fb0e0] [c0000000001b8888] update_process_times+0x68/0x90 [ 252.688464] [c000003fe35fb110] [c0000000001d05b4] tick_sched_handle.isra.5+0x34/0xd0 [ 252.688469] [c000003fe35fb140] [c0000000001d06b0] tick_sched_timer+0x60/0xe0 [ 252.688475] [c000003fe35fb180] [c0000000001b9414] hrtimer_run_queues+0x144/0x370 [ 252.688481] [c000003fe35fb200] [c0000000001ba36c] hrtimer_interrupt+0xfc/0x350 [ 252.688487] [c000003fe35fb2d0] [c0000000000247b0] timer_interrupt+0x90/0x260 [ 252.688494] [c000003fe35fb320] [c000000000024bc8] timer_interrupt+0x98/0xe0 [ 252.688500] [c000003fe35fb350] [c000000000009104] decrementer_common+0x114/0x120 [ 252.688510] --- interrupt: 901 at replay_interrupt_return+0x0/0x4 [ 252.688510] LR = arch_local_irq_restore+0x74/0x90 [ 252.688513] [c000003fe35fb640] [000000000000000b] 0xb (unreliable) [ 252.688519] [c000003fe35fb660] [c0000000001142d4] do_exit+0xf4/0xc80 [ 252.688525] [c000003fe35fb720] [c00000000002606c] oops_end+0x17c/0x210 [ 252.688530] [c000003fe35fb7a0] [c00000000006b2a0] bad_page_fault+0xe0/0x160 [ 252.688535] [c000003fe35fb810] [c000000000027654] slb_miss_bad_addr+0x34/0x70 [ 252.688541] [c000003fe35fb830] [c000000000008b0c] bad_addr_slb+0x11c/0x120 [ 252.688550] --- interrupt: 380 at dbs_update_util_handler+0x18/0xf0 [ 252.688550] LR = update_curr_rt+0xd0/0x3f0 [ 252.688552] [c000003fe35fbb20] [0000000000000000] (null) (unreliable) [ 252.688558] [c000003fe35fbbb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 252.688563] [c000003fe35fbbf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 252.688569] [c000003fe35fbc70] [c000000000cfb9dc] __schedule+0x3bc/0xaf0 [ 252.688575] [c000003fe35fbd40] [c000000000cfc150] schedule+0x40/0xc0 [ 252.688582] [c000003fe35fbd60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 252.688586] [c000003fe35fbdc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 252.688593] [c000003fe35fbe30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 252.688594] Instruction dump: [ 252.688597] 40c20010 7d40192d 40c2fff0 7c2004ac 2fa90000 409e001c 38210030 ebe1fff8 [ 252.688607] 4e800020 60000000 60000000 60420000 <7c210b78> e92d0000 89290009 71290002 [ 252.698436] Watchdog CPU:57 Hard LOCKUP [ 252.698437] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 252.698479] CPU: 57 PID: 351 Comm: watchdog/57 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 252.698482] NIP: c000000000d02480 LR: c000000000151318 CTR: c000000000444d00 [ 252.698485] REGS: c00000003fd4fd80 TRAP: 0100 Tainted: G D (4.15.0-33-generic) [ 252.698486] MSR: 9000000002089033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE> CR: 28008484 XER: 00000000 [ 252.698498] CFAR: c000000000d0249c SOFTE: 0 [ 252.698498] GPR00: c000000000151318 c000003fe3587050 c0000000016eb000 c000003fef9d8580 [ 252.698498] GPR04: 0000000000000000 0000000000000000 0000000000000000 00000035f654704a [ 252.698498] GPR08: c000003fce7ed9e0 0000000080000039 0000000080000039 c000003ebf1ee730 [ 252.698498] GPR12: 0000000000008000 c00000000faa7300 c00000000013d748 c000003fed0e7280 [ 252.698498] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000003fee800000 [ 252.698498] GPR20: c000003fef9c35c0 0000000000000000 0000000000000000 c0000000001d0650 [ 252.698498] GPR24: 0000000000000000 c000003fe3481680 0000003fee800000 c00000000171dd78 [ 252.698498] GPR28: c0000000011d8580 00000000000001c8 c0000000011d8580 c000003fef9d8580 [ 252.698544] NIP [c000000000d02480] _raw_spin_lock+0x60/0xe0 [ 252.698549] LR [c000000000151318] scheduler_tick+0x68/0x170 [ 252.698550] Call Trace: [ 252.698552] [c000003fe3587050] [0000000000000004] 0x4 (unreliable) [ 252.698556] [c000003fe3587080] [0000000000000000] (null) [ 252.698561] [c000003fe35870e0] [c0000000001b8888] update_process_times+0x68/0x90 [ 252.698566] [c000003fe3587110] [c0000000001d05b4] tick_sched_handle.isra.5+0x34/0xd0 [ 252.698571] [c000003fe3587140] [c0000000001d06b0] tick_sched_timer+0x60/0xe0 [ 252.698577] [c000003fe3587180] [c0000000001b9414] hrtimer_run_queues+0x144/0x370 [ 252.698583] [c000003fe3587200] [c0000000001ba36c] hrtimer_interrupt+0xfc/0x350 [ 252.698589] [c000003fe35872d0] [c0000000000247b0] timer_interrupt+0x90/0x260 [ 252.698595] [c000003fe3587320] [c000000000024bc8] timer_interrupt+0x98/0xe0 [ 252.698601] [c000003fe3587350] [c000000000009104] decrementer_common+0x114/0x120 [ 252.698611] --- interrupt: 901 at replay_interrupt_return+0x0/0x4 [ 252.698611] LR = arch_local_irq_restore+0x74/0x90 [ 252.698613] [c000003fe3587640] [000000000000000b] 0xb (unreliable) [ 252.698619] [c000003fe3587660] [c0000000001142d4] do_exit+0xf4/0xc80 [ 252.698625] [c000003fe3587720] [c00000000002606c] oops_end+0x17c/0x210 [ 252.698630] [c000003fe35877a0] [c00000000006b2a0] bad_page_fault+0xe0/0x160 [ 252.698635] [c000003fe3587810] [c000000000027654] slb_miss_bad_addr+0x34/0x70 [ 252.698640] [c000003fe3587830] [c000000000008b0c] bad_addr_slb+0x11c/0x120 [ 252.698649] --- interrupt: 380 at dbs_update_util_handler+0x18/0xf0 [ 252.698649] LR = update_curr_rt+0xd0/0x3f0 [ 252.698651] [c000003fe3587b20] [0000000000000000] (null) (unreliable) [ 252.698657] [c000003fe3587bb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 252.698662] [c000003fe3587bf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 252.698668] [c000003fe3587c70] [c000000000cfb9dc] __schedule+0x3bc/0xaf0 [ 252.698674] [c000003fe3587d40] [c000000000cfc150] schedule+0x40/0xc0 [ 252.698680] [c000003fe3587d60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 252.698684] [c000003fe3587dc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 252.698691] [c000003fe3587e30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 252.698692] Instruction dump: [ 252.698695] 40c20010 7d40192d 40c2fff0 7c2004ac 2fa90000 409e001c 38210030 ebe1fff8 [ 252.698705] 4e800020 60000000 60000000 60420000 <7c210b78> e92d0000 89290009 71290002 [ 252.708545] Watchdog CPU:58 Hard LOCKUP [ 252.708546] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 252.708587] CPU: 58 PID: 357 Comm: watchdog/58 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 252.708590] NIP: c000000000d02484 LR: c000000000151318 CTR: c000000000444d00 [ 252.708593] REGS: c00000003fd43d80 TRAP: 0100 Tainted: G D (4.15.0-33-generic) [ 252.708594] MSR: 9000000002089033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE> CR: 28008484 XER: 00000000 [ 252.708606] CFAR: c000000000d0249c SOFTE: 0 [ 252.708606] GPR00: c000000000151318 c000003fe359b050 c0000000016eb000 c000003fefa18580 [ 252.708606] GPR04: 0000000000000000 0000000000000000 0000000000000000 00000036275f98ce [ 252.708606] GPR08: c000003fce7ed9e0 000000008000003a 000000008000003a c000003ebf1ee730 [ 252.708606] GPR12: 0000000000008000 c00000000faa7e00 c00000000013d748 c000003fed0e7300 [ 252.708606] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000003fee840000 [ 252.708606] GPR20: c000003fefa035c0 0000000000000000 0000000000000000 c0000000001d0650 [ 252.708606] GPR24: 0000000000000000 c000003fe34b1380 0000003fee840000 c00000000171dd78 [ 252.708606] GPR28: c0000000011d8580 00000000000001d0 c0000000011d8580 c000003fefa18580 [ 252.708652] NIP [c000000000d02484] _raw_spin_lock+0x64/0xe0 [ 252.708657] LR [c000000000151318] scheduler_tick+0x68/0x170 [ 252.708658] Call Trace: [ 252.708660] [c000003fe359b050] [0000000000000004] 0x4 (unreliable) [ 252.708663] [c000003fe359b080] [0000000000000000] (null) [ 252.708669] [c000003fe359b0e0] [c0000000001b8888] update_process_times+0x68/0x90 [ 252.708674] [c000003fe359b110] [c0000000001d05b4] tick_sched_handle.isra.5+0x34/0xd0 [ 252.708678] [c000003fe359b140] [c0000000001d06b0] tick_sched_timer+0x60/0xe0 [ 252.708684] [c000003fe359b180] [c0000000001b9414] hrtimer_run_queues+0x144/0x370 [ 252.708690] [c000003fe359b200] [c0000000001ba36c] hrtimer_interrupt+0xfc/0x350 [ 252.708696] [c000003fe359b2d0] [c0000000000247b0] timer_interrupt+0x90/0x260 [ 252.708702] [c000003fe359b320] [c000000000024bc8] timer_interrupt+0x98/0xe0 [ 252.708708] [c000003fe359b350] [c000000000009104] decrementer_common+0x114/0x120 [ 252.708717] --- interrupt: 901 at replay_interrupt_return+0x0/0x4 [ 252.708717] LR = arch_local_irq_restore+0x74/0x90 [ 252.708720] [c000003fe359b640] [000000000000000b] 0xb (unreliable) [ 252.708725] [c000003fe359b660] [c0000000001142d4] do_exit+0xf4/0xc80 [ 252.708731] [c000003fe359b720] [c00000000002606c] oops_end+0x17c/0x210 [ 252.708737] [c000003fe359b7a0] [c00000000006b2a0] bad_page_fault+0xe0/0x160 [ 252.708741] [c000003fe359b810] [c000000000027654] slb_miss_bad_addr+0x34/0x70 [ 252.708747] [c000003fe359b830] [c000000000008b0c] bad_addr_slb+0x11c/0x120 [ 252.708755] --- interrupt: 380 at dbs_update_util_handler+0x18/0xf0 [ 252.708755] LR = update_curr_rt+0xd0/0x3f0 [ 252.708757] [c000003fe359bb20] [0000000000000000] (null) (unreliable) [ 252.708763] [c000003fe359bbb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 252.708768] [c000003fe359bbf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 252.708774] [c000003fe359bc70] [c000000000cfb9dc] __schedule+0x3bc/0xaf0 [ 252.708780] [c000003fe359bd40] [c000000000cfc150] schedule+0x40/0xc0 [ 252.708785] [c000003fe359bd60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 252.708789] [c000003fe359bdc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 252.708796] [c000003fe359be30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 252.708798] Instruction dump: [ 252.708800] 7d40192d 40c2fff0 7c2004ac 2fa90000 409e001c 38210030 ebe1fff8 4e800020 [ 252.708810] 60000000 60000000 60420000 7c210b78 89290009 71290002 40820050 [ 252.718655] Watchdog CPU:59 Hard LOCKUP [ 252.718656] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 252.718697] CPU: 59 PID: 363 Comm: watchdog/59 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 252.718700] NIP: c000000000d02480 LR: c000000000151318 CTR: c000000000444d00 [ 252.718702] REGS: c00000003fd37d80 TRAP: 0100 Tainted: G D (4.15.0-33-generic) [ 252.718704] MSR: 9000000002089033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE> CR: 28008484 XER: 00000000 [ 252.718716] CFAR: c000000000d0249c SOFTE: 0 [ 252.718716] GPR00: c000000000151318 c000003fe35b7050 c0000000016eb000 c000003fefa58580 [ 252.718716] GPR04: 0000000000000000 0000000000000000 0000000000000000 00000035c5423674 [ 252.718716] GPR08: c000003fce7ed9e0 000000008000003b 000000008000003b c000003ebf1ee730 [ 252.718716] GPR12: 0000000000008000 c00000000faa8900 c00000000013d748 c000003fed0e7380 [ 252.718716] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000003fee880000 [ 252.718716] GPR20: c000003fefa435c0 0000000000000000 0000000000000000 c0000000001d0650 [ 252.718716] GPR24: 0000000000000000 c000003fe3499500 0000003fee880000 c00000000171dd78 [ 252.718716] GPR28: c0000000011d8580 00000000000001d8 c0000000011d8580 c000003fefa58580 [ 252.718761] NIP [c000000000d02480] _raw_spin_lock+0x60/0xe0 [ 252.718765] LR [c000000000151318] scheduler_tick+0x68/0x170 [ 252.718766] Call Trace: [ 252.718769] [c000003fe35b7050] [0000000000000004] 0x4 (unreliable) [ 252.718772] [c000003fe35b7080] [0000000000000000] (null) [ 252.718777] [c000003fe35b70e0] [c0000000001b8888] update_process_times+0x68/0x90 [ 252.718782] [c000003fe35b7110] [c0000000001d05b4] tick_sched_handle.isra.5+0x34/0xd0 [ 252.718787] [c000003fe35b7140] [c0000000001d06b0] tick_sched_timer+0x60/0xe0 [ 252.718792] [c000003fe35b7180] [c0000000001b9414] hrtimer_run_queues+0x144/0x370 [ 252.718798] [c000003fe35b7200] [c0000000001ba36c] hrtimer_interrupt+0xfc/0x350 [ 252.718804] [c000003fe35b72d0] [c0000000000247b0] timer_interrupt+0x90/0x260 [ 252.718810] [c000003fe35b7320] [c000000000024bc8] timer_interrupt+0x98/0xe0 [ 252.718816] [c000003fe35b7350] [c000000000009104] decrementer_common+0x114/0x120 [ 252.718825] --- interrupt: 901 at replay_interrupt_return+0x0/0x4 [ 252.718825] LR = arch_local_irq_restore+0x74/0x90 [ 252.718828] [c000003fe35b7640] [000000000000000b] 0xb (unreliable) [ 252.718833] [c000003fe35b7660] [c0000000001142d4] do_exit+0xf4/0xc80 [ 252.718839] [c000003fe35b7720] [c00000000002606c] oops_end+0x17c/0x210 [ 252.718844] [c000003fe35b77a0] [c00000000006b2a0] bad_page_fault+0xe0/0x160 [ 252.718849] [c000003fe35b7810] [c000000000027654] slb_miss_bad_addr+0x34/0x70 [ 252.718854] [c000003fe35b7830] [c000000000008b0c] bad_addr_slb+0x11c/0x120 [ 252.718863] --- interrupt: 380 at dbs_update_util_handler+0x18/0xf0 [ 252.718863] LR = update_curr_rt+0xd0/0x3f0 [ 252.718864] [c000003fe35b7b20] [0000000000000000] (null) (unreliable) [ 252.718870] [c000003fe35b7bb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 252.718875] [c000003fe35b7bf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 252.718881] [c000003fe35b7c70] [c000000000cfb9dc] __schedule+0x3bc/0xaf0 [ 252.718886] [c000003fe35b7d40] [c000000000cfc150] schedule+0x40/0xc0 [ 252.718892] [c000003fe35b7d60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 252.718896] [c000003fe35b7dc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 252.718903] [c000003fe35b7e30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 252.718905] Instruction dump: [ 252.718907] 40c20010 7d40192d 40c2fff0 7c2004ac 2fa90000 409e001c 38210030 ebe1fff8 [ 252.718917] 4e800020 60000000 60000000 60420000 <7c210b78> e92d0000 89290009 71290002 [ 252.728871] Watchdog CPU:93 Hard LOCKUP [ 252.728872] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 252.728914] CPU: 93 PID: 568 Comm: watchdog/93 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 252.728917] NIP: c000000000d02480 LR: c000000000151318 CTR: c000000000444d00 [ 252.728920] REGS: c00000003fb9fd80 TRAP: 0100 Tainted: G D (4.15.0-33-generic) [ 252.728922] MSR: 9000000002089033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE> CR: 28008484 XER: 00000000 [ 252.728934] CFAR: c000000000d0249c SOFTE: 0 [ 252.728934] GPR00: c000000000151318 c00020397bfa3050 c0000000016eb000 c000203993ba8580 [ 252.728934] GPR04: 0000000000000000 0000000000000000 0000000000000000 0000000000004000 [ 252.728934] GPR08: c000003fce7ed9e0 000000008000005d 000000008000005d 00000000000001b9 [ 252.728934] GPR12: 0000000000008000 c00000000fabff00 c00000000013d748 c000003fed0e8480 [ 252.728934] GPR16: 0000000000000000 0000000000000000 0000000000000000 00002039929d0000 [ 252.728934] GPR20: c000203993b935c0 0000000000000000 0000000000000000 c0000000001d0650 [ 252.728934] GPR24: 0000000000000000 c00020397bcb5700 00002039929d0000 c00000000171dd78 [ 252.728934] GPR28: c0000000011d8580 00000000000002e8 c0000000011d8580 c000203993ba8580 [ 252.728980] NIP [c000000000d02480] _raw_spin_lock+0x60/0xe0 [ 252.728985] LR [c000000000151318] scheduler_tick+0x68/0x170 [ 252.728986] Call Trace: [ 252.728988] [c00020397bfa3050] [0000000000000004] 0x4 (unreliable) [ 252.728992] [c00020397bfa3080] [0000000000000000] (null) [ 252.728998] [c00020397bfa30e0] [c0000000001b8888] update_process_times+0x68/0x90 [ 252.729003] [c00020397bfa3110] [c0000000001d05b4] tick_sched_handle.isra.5+0x34/0xd0 [ 252.729007] [c00020397bfa3140] [c0000000001d06b0] tick_sched_timer+0x60/0xe0 [ 252.729013] [c00020397bfa3180] [c0000000001b9414] hrtimer_run_queues+0x144/0x370 [ 252.729019] [c00020397bfa3200] [c0000000001ba36c] hrtimer_interrupt+0xfc/0x350 [ 252.729025] [c00020397bfa32d0] [c0000000000247b0] timer_interrupt+0x90/0x260 [ 252.729032] [c00020397bfa3320] [c000000000024bc8] timer_interrupt+0x98/0xe0 [ 252.729038] [c00020397bfa3350] [c000000000009104] decrementer_common+0x114/0x120 [ 252.729048] --- interrupt: 901 at replay_interrupt_return+0x0/0x4 [ 252.729048] LR = arch_local_irq_restore+0x74/0x90 [ 252.729050] [c00020397bfa3640] [000000000000000b] 0xb (unreliable) [ 252.729056] [c00020397bfa3660] [c0000000001142d4] do_exit+0xf4/0xc80 [ 252.729062] [c00020397bfa3720] [c00000000002606c] oops_end+0x17c/0x210 [ 252.729068] [c00020397bfa37a0] [c00000000006b2a0] bad_page_fault+0xe0/0x160 [ 252.729073] [c00020397bfa3810] [c000000000027654] slb_miss_bad_addr+0x34/0x70 [ 252.729079] [c00020397bfa3830] [c000000000008b0c] bad_addr_slb+0x11c/0x120 [ 252.729088] --- interrupt: 380 at dbs_update_util_handler+0x18/0xf0 [ 252.729088] LR = update_curr_rt+0xd0/0x3f0 [ 252.729089] [c00020397bfa3b20] [0000000000000000] (null) (unreliable) [ 252.729095] [c00020397bfa3bb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 252.729100] [c00020397bfa3bf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 252.729107] [c00020397bfa3c70] [c000000000cfb9dc] __schedule+0x3bc/0xaf0 [ 252.729112] [c00020397bfa3d40] [c000000000cfc150] schedule+0x40/0xc0 [ 252.729119] [c00020397bfa3d60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 252.729123] [c00020397bfa3dc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 252.729130] [c00020397bfa3e30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 252.729132] Instruction dump: [ 252.729134] 40c20010 7d40192d 40c2fff0 7c2004ac 2fa90000 409e001c 38210030 ebe1fff8 [ 252.729144] 4e800020 60000000 60000000 60420000 <7c210b78> e92d0000 89290009 71290002 [ 252.738991] Watchdog CPU:94 Hard LOCKUP [ 252.738992] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 252.739033] CPU: 94 PID: 574 Comm: watchdog/94 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 252.739036] NIP: c000000000d02484 LR: c000000000151318 CTR: c000000000444d00 [ 252.739039] REGS: c00000003fb93d80 TRAP: 0100 Tainted: G D (4.15.0-33-generic) [ 252.739040] MSR: 9000000002089033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE> CR: 28008484 XER: 00000000 [ 252.739052] CFAR: c000000000d0249c SOFTE: 0 [ 252.739052] GPR00: c000000000151318 c00020397bf83050 c0000000016eb000 c000203993be8580 [ 252.739052] GPR04: 0000000000000000 0000000000000000 0000000000000000 0000000000004000 [ 252.739052] GPR08: c000003fce7ed9e0 000000008000005e 000000008000005e 00000000000001b9 [ 252.739052] GPR12: 0000000000008000 c00000000fac0a00 c00000000013d748 c000003fed0e8500 [ 252.739052] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000203992a10000 [ 252.739052] GPR20: c000203993bd35c0 0000000000000000 0000000000000000 c0000000001d0650 [ 252.739052] GPR24: 0000000000000000 c00020397bc9c200 0000203992a10000 c00000000171dd78 [ 252.739052] GPR28: c0000000011d8580 00000000000002f0 c0000000011d8580 c000203993be8580 [ 252.739098] NIP [c000000000d02484] _raw_spin_lock+0x64/0xe0 [ 252.739102] LR [c000000000151318] scheduler_tick+0x68/0x170 [ 252.739104] Call Trace: [ 252.739106] [c00020397bf83050] [0000000000000004] 0x4 (unreliable) [ 252.739109] [c00020397bf83080] [0000000000000000] (null) [ 252.739115] [c00020397bf830e0] [c0000000001b8888] update_process_times+0x68/0x90 [ 252.739120] [c00020397bf83110] [c0000000001d05b4] tick_sched_handle.isra.5+0x34/0xd0 [ 252.739124] [c00020397bf83140] [c0000000001d06b0] tick_sched_timer+0x60/0xe0 [ 252.739130] [c00020397bf83180] [c0000000001b9414] hrtimer_run_queues+0x144/0x370 [ 252.739136] [c00020397bf83200] [c0000000001ba36c] hrtimer_interrupt+0xfc/0x350 [ 252.739142] [c00020397bf832d0] [c0000000000247b0] timer_interrupt+0x90/0x260 [ 252.739148] [c00020397bf83320] [c000000000024bc8] timer_interrupt+0x98/0xe0 [ 252.739154] [c00020397bf83350] [c000000000009104] decrementer_common+0x114/0x120 [ 252.739164] --- interrupt: 901 at replay_interrupt_return+0x0/0x4 [ 252.739164] LR = arch_local_irq_restore+0x74/0x90 [ 252.739166] [c00020397bf83640] [000000000000000b] 0xb (unreliable) [ 252.739172] [c00020397bf83660] [c0000000001142d4] do_exit+0xf4/0xc80 [ 252.739178] [c00020397bf83720] [c00000000002606c] oops_end+0x17c/0x210 [ 252.739183] [c00020397bf837a0] [c00000000006b2a0] bad_page_fault+0xe0/0x160 [ 252.739188] [c00020397bf83810] [c000000000027654] slb_miss_bad_addr+0x34/0x70 [ 252.739194] [c00020397bf83830] [c000000000008b0c] bad_addr_slb+0x11c/0x120 [ 252.739203] --- interrupt: 380 at dbs_update_util_handler+0x18/0xf0 [ 252.739203] LR = update_curr_rt+0xd0/0x3f0 [ 252.739204] [c00020397bf83b20] [0000000000000000] (null) (unreliable) [ 252.739210] [c00020397bf83bb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 252.739215] [c00020397bf83bf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 252.739221] [c00020397bf83c70] [c000000000cfb9dc] __schedule+0x3bc/0xaf0 [ 252.739227] [c00020397bf83d40] [c000000000cfc150] schedule+0x40/0xc0 [ 252.739233] [c00020397bf83d60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 252.739237] [c00020397bf83dc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 252.739244] [c00020397bf83e30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 252.739245] Instruction dump: [ 252.739247] 7d40192d 40c2fff0 7c2004ac 2fa90000 409e001c 38210030 ebe1fff8 4e800020 [ 252.739258] 60000000 60000000 60420000 7c210b78 89290009 71290002 40820050 [ 252.749112] Watchdog CPU:95 Hard LOCKUP [ 252.749113] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 252.749153] CPU: 95 PID: 580 Comm: watchdog/95 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 252.749156] NIP: c000000000d02480 LR: c000000000151318 CTR: c000000000444d00 [ 252.749159] REGS: c00000003fb87d80 TRAP: 0100 Tainted: G D (4.15.0-33-generic) [ 252.749160] MSR: 9000000002089033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE> CR: 28008484 XER: 00000000 [ 252.749172] CFAR: c000000000d0249c SOFTE: 0 [ 252.749172] GPR00: c000000000151318 c00020397bfa7050 c0000000016eb000 c000203993c28580 [ 252.749172] GPR04: 0000000000000000 0000000000000000 0000000000000000 0000000000004000 [ 252.749172] GPR08: c000003fce7ed9e0 000000008000005f 000000008000005f 00000000000001b9 [ 252.749172] GPR12: 0000000000008000 c00000000fac1500 c00000000013d748 c000003fed0e8580 [ 252.749172] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000203992a50000 [ 252.749172] GPR20: c000203993c135c0 0000000000000000 0000000000000000 c0000000001d0650 [ 252.749172] GPR24: 0000000000000000 c00020397bc84380 0000203992a50000 c00000000171dd78 [ 252.749172] GPR28: c0000000011d8580 00000000000002f8 c0000000011d8580 c000203993c28580 [ 252.749217] NIP [c000000000d02480] _raw_spin_lock+0x60/0xe0 [ 252.749222] LR [c000000000151318] scheduler_tick+0x68/0x170 [ 252.749223] Call Trace: [ 252.749225] [c00020397bfa7050] [0000000000000004] 0x4 (unreliable) [ 252.749228] [c00020397bfa7080] [0000000000000000] (null) [ 252.749234] [c00020397bfa70e0] [c0000000001b8888] update_process_times+0x68/0x90 [ 252.749239] [c00020397bfa7110] [c0000000001d05b4] tick_sched_handle.isra.5+0x34/0xd0 [ 252.749243] [c00020397bfa7140] [c0000000001d06b0] tick_sched_timer+0x60/0xe0 [ 252.749249] [c00020397bfa7180] [c0000000001b9414] hrtimer_run_queues+0x144/0x370 [ 252.749254] [c00020397bfa7200] [c0000000001ba36c] hrtimer_interrupt+0xfc/0x350 [ 252.749261] [c00020397bfa72d0] [c0000000000247b0] timer_interrupt+0x90/0x260 [ 252.749267] [c00020397bfa7320] [c000000000024bc8] timer_interrupt+0x98/0xe0 [ 252.749273] [c00020397bfa7350] [c000000000009104] decrementer_common+0x114/0x120 [ 252.749282] --- interrupt: 901 at replay_interrupt_return+0x0/0x4 [ 252.749282] LR = arch_local_irq_restore+0x74/0x90 [ 252.749284] [c00020397bfa7640] [000000000000000b] 0xb (unreliable) [ 252.749290] [c00020397bfa7660] [c0000000001142d4] do_exit+0xf4/0xc80 [ 252.749296] [c00020397bfa7720] [c00000000002606c] oops_end+0x17c/0x210 [ 252.749301] [c00020397bfa77a0] [c00000000006b2a0] bad_page_fault+0xe0/0x160 [ 252.749306] [c00020397bfa7810] [c000000000027654] slb_miss_bad_addr+0x34/0x70 [ 252.749312] [c00020397bfa7830] [c000000000008b0c] bad_addr_slb+0x11c/0x120 [ 252.749320] --- interrupt: 380 at dbs_update_util_handler+0x18/0xf0 [ 252.749320] LR = update_curr_rt+0xd0/0x3f0 [ 252.749322] [c00020397bfa7b20] [0000000000000000] (null) (unreliable) [ 252.749328] [c00020397bfa7bb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 252.749332] [c00020397bfa7bf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 252.749338] [c00020397bfa7c70] [c000000000cfb9dc] __schedule+0x3bc/0xaf0 [ 252.749344] [c00020397bfa7d40] [c000000000cfc150] schedule+0x40/0xc0 [ 252.749350] [c00020397bfa7d60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 252.749354] [c00020397bfa7dc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 252.749361] [c00020397bfa7e30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 252.749363] Instruction dump: [ 252.749365] 40c20010 7d40192d 40c2fff0 7c2004ac 2fa90000 409e001c 38210030 ebe1fff8 [ 252.749375] 4e800020 60000000 60000000 60420000 <7c210b78> e92d0000 89290009 71290002 [ 252.759237] Watchdog CPU:132 Hard LOCKUP [ 252.759238] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 252.759281] CPU: 132 PID: 802 Comm: watchdog/132 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 252.759284] NIP: c000000000d02484 LR: c000000000151318 CTR: c000000000444d00 [ 252.759287] REGS: c00000003f9cbd80 TRAP: 0100 Tainted: G D (4.15.0-33-generic) [ 252.759289] MSR: 9000000002089033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE> CR: 28008484 XER: 00000000 [ 252.759302] CFAR: c000000000d0249c SOFTE: 0 [ 252.759302] GPR00: c000000000151318 c000203978bdb050 c0000000016eb000 c000203994568580 [ 252.759302] GPR04: 0000000000000000 0000000000000000 0000000000000000 0000000000000004 [ 252.759302] GPR08: c000003fce7ed9e0 0000000080000084 0000000080000084 00000000000001b9 [ 252.759302] GPR12: 0000000000008000 c00000000fadac00 c00000000013d748 c000003fed0e9800 [ 252.759302] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000203993390000 [ 252.759302] GPR20: c0002039945535c0 0000000000000000 0000000000000000 c0000000001d0650 [ 252.759302] GPR24: 0000000000000000 c000203978895180 0000203993390000 c00000000171dd78 [ 252.759302] GPR28: c0000000011d8580 0000000000000420 c0000000011d8580 c000203994568580 [ 252.759350] NIP [c000000000d02484] _raw_spin_lock+0x64/0xe0 [ 252.759355] LR [c000000000151318] scheduler_tick+0x68/0x170 [ 252.759356] Call Trace: [ 252.759359] [c000203978bdb050] [0000000000000004] 0x4 (unreliable) [ 252.759363] [c000203978bdb080] [0000000000000000] (null) [ 252.759369] [c000203978bdb0e0] [c0000000001b8888] update_process_times+0x68/0x90 [ 252.759374] [c000203978bdb110] [c0000000001d05b4] tick_sched_handle.isra.5+0x34/0xd0 [ 252.759379] [c000203978bdb140] [c0000000001d06b0] tick_sched_timer+0x60/0xe0 [ 252.759385] [c000203978bdb180] [c0000000001b9414] hrtimer_run_queues+0x144/0x370 [ 252.759391] [c000203978bdb200] [c0000000001ba36c] hrtimer_interrupt+0xfc/0x350 [ 252.759398] [c000203978bdb2d0] [c0000000000247b0] timer_interrupt+0x90/0x260 [ 252.759404] [c000203978bdb320] [c000000000024bc8] timer_interrupt+0x98/0xe0 [ 252.759411] [c000203978bdb350] [c000000000009104] decrementer_common+0x114/0x120 [ 252.759421] --- interrupt: 901 at replay_interrupt_return+0x0/0x4 [ 252.759421] LR = arch_local_irq_restore+0x74/0x90 [ 252.759424] [c000203978bdb640] [000000000000000b] 0xb (unreliable) [ 252.759430] [c000203978bdb660] [c0000000001142d4] do_exit+0xf4/0xc80 [ 252.759436] [c000203978bdb720] [c00000000002606c] oops_end+0x17c/0x210 [ 252.759442] [c000203978bdb7a0] [c00000000006b2a0] bad_page_fault+0xe0/0x160 [ 252.759447] [c000203978bdb810] [c000000000027654] slb_miss_bad_addr+0x34/0x70 [ 252.759453] [c000203978bdb830] [c000000000008b0c] bad_addr_slb+0x11c/0x120 [ 252.759462] --- interrupt: 380 at dbs_update_util_handler+0x18/0xf0 [ 252.759462] LR = update_curr_rt+0xd0/0x3f0 [ 252.759464] [c000203978bdbb20] [0000000000000000] (null) (unreliable) [ 252.759470] [c000203978bdbbb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 252.759475] [c000203978bdbbf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 252.759481] [c000203978bdbc70] [c000000000cfb9dc] __schedule+0x3bc/0xaf0 [ 252.759487] [c000203978bdbd40] [c000000000cfc150] schedule+0x40/0xc0 [ 252.759493] [c000203978bdbd60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 252.759497] [c000203978bdbdc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 252.759504] [c000203978bdbe30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 252.759506] Instruction dump: [ 252.759509] 7d40192d 40c2fff0 7c2004ac 2fa90000 409e001c 38210030 ebe1fff8 4e800020 [ 252.759519] 60000000 60000000 60420000 7c210b78 89290009 71290002 40820050 [ 252.769357] Watchdog CPU:133 Hard LOCKUP [ 252.769358] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 252.769399] CPU: 133 PID: 808 Comm: watchdog/133 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 252.769402] NIP: c000000000d02484 LR: c000000000151318 CTR: c000000000444d00 [ 252.769405] REGS: c00000003f9bfd80 TRAP: 0100 Tainted: G D (4.15.0-33-generic) [ 252.769407] MSR: 9000000002089033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE> CR: 28008484 XER: 00000000 [ 252.769419] CFAR: c000000000d0249c SOFTE: 0 [ 252.769419] GPR00: c000000000151318 c000203978bcb050 c0000000016eb000 c0002039945a8580 [ 252.769419] GPR04: 0000000000000000 0000000000000000 0000000000000000 0010000000000000 [ 252.769419] GPR08: c000003fce7ed9e0 0000000080000085 0000000080000085 00000000000001b9 [ 252.769419] GPR12: 0000000000008000 c00000000fadb700 c00000000013d748 c000003fed0e9880 [ 252.769419] GPR16: 0000000000000000 0000000000000000 0000000000000000 00002039933d0000 [ 252.769419] GPR20: c0002039945935c0 0000000000000000 0000000000000000 c0000000001d0650 [ 252.769419] GPR24: 0000000000000000 c0002039788a5f80 00002039933d0000 c00000000171dd78 [ 252.769419] GPR28: c0000000011d8580 0000000000000428 c0000000011d8580 c0002039945a8580 [ 252.769465] NIP [c000000000d02484] _raw_spin_lock+0x64/0xe0 [ 252.769470] LR [c000000000151318] scheduler_tick+0x68/0x170 [ 252.769471] Call Trace: [ 252.769473] [c000203978bcb050] [0000000000000004] 0x4 (unreliable) [ 252.769477] [c000203978bcb080] [0000000000000000] (null) [ 252.769483] [c000203978bcb0e0] [c0000000001b8888] update_process_times+0x68/0x90 [ 252.769487] [c000203978bcb110] [c0000000001d05b4] tick_sched_handle.isra.5+0x34/0xd0 [ 252.769492] [c000203978bcb140] [c0000000001d06b0] tick_sched_timer+0x60/0xe0 [ 252.769498] [c000203978bcb180] [c0000000001b9414] hrtimer_run_queues+0x144/0x370 [ 252.769504] [c000203978bcb200] [c0000000001ba36c] hrtimer_interrupt+0xfc/0x350 [ 252.769511] [c000203978bcb2d0] [c0000000000247b0] timer_interrupt+0x90/0x260 [ 252.769517] [c000203978bcb320] [c000000000024bc8] timer_interrupt+0x98/0xe0 [ 252.769523] [c000203978bcb350] [c000000000009104] decrementer_common+0x114/0x120 [ 252.769533] --- interrupt: 901 at replay_interrupt_return+0x0/0x4 [ 252.769533] LR = arch_local_irq_restore+0x74/0x90 [ 252.769535] [c000203978bcb640] [000000000000000b] 0xb (unreliable) [ 252.769541] [c000203978bcb660] [c0000000001142d4] do_exit+0xf4/0xc80 [ 252.769547] [c000203978bcb720] [c00000000002606c] oops_end+0x17c/0x210 [ 252.769552] [c000203978bcb7a0] [c00000000006b2a0] bad_page_fault+0xe0/0x160 [ 252.769556] [c000203978bcb810] [c000000000027654] slb_miss_bad_addr+0x34/0x70 [ 252.769562] [c000203978bcb830] [c000000000008b0c] bad_addr_slb+0x11c/0x120 [ 252.769571] --- interrupt: 380 at dbs_update_util_handler+0x18/0xf0 [ 252.769571] LR = update_curr_rt+0xd0/0x3f0 [ 252.769573] [c000203978bcbb20] [0000000000000000] (null) (unreliable) [ 252.769578] [c000203978bcbbb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 252.769583] [c000203978bcbbf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 252.769589] [c000203978bcbc70] [c000000000cfb9dc] __schedule+0x3bc/0xaf0 [ 252.769595] [c000203978bcbd40] [c000000000cfc150] schedule+0x40/0xc0 [ 252.769601] [c000203978bcbd60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 252.769605] [c000203978bcbdc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 252.769612] [c000203978bcbe30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 252.769614] Instruction dump: [ 252.769616] 7d40192d 40c2fff0 7c2004ac 2fa90000 409e001c 38210030 ebe1fff8 4e800020 [ 252.769626] 60000000 60000000 60420000 7c210b78 89290009 71290002 40820050 [ 252.779478] Watchdog CPU:134 Hard LOCKUP [ 252.779479] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 252.779521] CPU: 134 PID: 814 Comm: watchdog/134 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 252.779524] NIP: c000000000d02480 LR: c000000000151318 CTR: c000000000444d00 [ 252.779526] REGS: c00000003f9b3d80 TRAP: 0100 Tainted: G D (4.15.0-33-generic) [ 252.779528] MSR: 9000000002089033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE> CR: 28008484 XER: 00000000 [ 252.779540] CFAR: c000000000d0249c SOFTE: 0 [ 252.779540] GPR00: c000000000151318 c000203978b93050 c0000000016eb000 c0002039945e8580 [ 252.779540] GPR04: 0000000000000000 0000000000000000 0000000000000000 0000000000000004 [ 252.779540] GPR08: c000003fce7ed9e0 0000000080000086 0000000080000086 00000000000001b9 [ 252.779540] GPR12: 0000000000008000 c00000000fadc200 c00000000013d748 c000003fed0e9900 [ 252.779540] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000203993410000 [ 252.779540] GPR20: c0002039945d35c0 0000000000000000 0000000000000000 c0000000001d0650 [ 252.779540] GPR24: 0000000000000000 c000203978e78400 0000203993410000 c00000000171dd78 [ 252.779540] GPR28: c0000000011d8580 0000000000000430 c0000000011d8580 c0002039945e8580 [ 252.779586] NIP [c000000000d02480] _raw_spin_lock+0x60/0xe0 [ 252.779591] LR [c000000000151318] scheduler_tick+0x68/0x170 [ 252.779591] Call Trace: [ 252.779594] [c000203978b93050] [0000000000000004] 0x4 (unreliable) [ 252.779597] [c000203978b93080] [0000000000000000] (null) [ 252.779603] [c000203978b930e0] [c0000000001b8888] update_process_times+0x68/0x90 [ 252.779608] [c000203978b93110] [c0000000001d05b4] tick_sched_handle.isra.5+0x34/0xd0 [ 252.779612] [c000203978b93140] [c0000000001d06b0] tick_sched_timer+0x60/0xe0 [ 252.779618] [c000203978b93180] [c0000000001b9414] hrtimer_run_queues+0x144/0x370 [ 252.779624] [c000203978b93200] [c0000000001ba36c] hrtimer_interrupt+0xfc/0x350 [ 252.779630] [c000203978b932d0] [c0000000000247b0] timer_interrupt+0x90/0x260 [ 252.779636] [c000203978b93320] [c000000000024bc8] timer_interrupt+0x98/0xe0 [ 252.779642] [c000203978b93350] [c000000000009104] decrementer_common+0x114/0x120 [ 252.779652] --- interrupt: 901 at replay_interrupt_return+0x0/0x4 [ 252.779652] LR = arch_local_irq_restore+0x74/0x90 [ 252.779654] [c000203978b93640] [000000000000000b] 0xb (unreliable) [ 252.779660] [c000203978b93660] [c0000000001142d4] do_exit+0xf4/0xc80 [ 252.779666] [c000203978b93720] [c00000000002606c] oops_end+0x17c/0x210 [ 252.779671] [c000203978b937a0] [c00000000006b2a0] bad_page_fault+0xe0/0x160 [ 252.779676] [c000203978b93810] [c000000000027654] slb_miss_bad_addr+0x34/0x70 [ 252.779681] [c000203978b93830] [c000000000008b0c] bad_addr_slb+0x11c/0x120 [ 252.779690] --- interrupt: 380 at dbs_update_util_handler+0x18/0xf0 [ 252.779690] LR = update_curr_rt+0xd0/0x3f0 [ 252.779692] [c000203978b93b20] [0000000000000000] (null) (unreliable) [ 252.779698] [c000203978b93bb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 252.779703] [c000203978b93bf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 252.779709] [c000203978b93c70] [c000000000cfb9dc] __schedule+0x3bc/0xaf0 [ 252.779715] [c000203978b93d40] [c000000000cfc150] schedule+0x40/0xc0 [ 252.779721] [c000203978b93d60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 252.779725] [c000203978b93dc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 252.779732] [c000203978b93e30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 252.779734] Instruction dump: [ 252.779736] 40c20010 7d40192d 40c2fff0 7c2004ac 2fa90000 409e001c 38210030 ebe1fff8 [ 252.779746] 4e800020 60000000 60000000 60420000 <7c210b78> e92d0000 89290009 71290002 [ 252.789600] Watchdog CPU:135 Hard LOCKUP [ 252.789601] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 252.789642] CPU: 135 PID: 820 Comm: watchdog/135 Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 252.789645] NIP: c000000000d02484 LR: c000000000151318 CTR: c000000000444d00 [ 252.789647] REGS: c00000003f9a7d80 TRAP: 0100 Tainted: G D (4.15.0-33-generic) [ 252.789649] MSR: 9000000002089033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE> CR: 28008484 XER: 00000000 [ 252.789661] CFAR: c000000000d0249c SOFTE: 0 [ 252.789661] GPR00: c000000000151318 c000203978bef050 c0000000016eb000 c000203994628580 [ 252.789661] GPR04: 0000000000000000 0000000000000000 0000000000000000 0000000000000004 [ 252.789661] GPR08: c000003fce7ed9e0 0000000080000087 0000000080000087 00000000000001b9 [ 252.789661] GPR12: 0000000000008000 c00000000fadcd00 c00000000013d748 c000003fed0e9980 [ 252.789661] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000203993450000 [ 252.789661] GPR20: c0002039946135c0 0000000000000000 0000000000000000 c0000000001d0650 [ 252.789661] GPR24: 0000000000000000 c000203978e76d80 0000203993450000 c00000000171dd78 [ 252.789661] GPR28: c0000000011d8580 0000000000000438 c0000000011d8580 c000203994628580 [ 252.789706] NIP [c000000000d02484] _raw_spin_lock+0x64/0xe0 [ 252.789710] LR [c000000000151318] scheduler_tick+0x68/0x170 [ 252.789711] Call Trace: [ 252.789713] [c000203978bef050] [0000000000000004] 0x4 (unreliable) [ 252.789716] [c000203978bef080] [0000000000000000] (null) [ 252.789722] [c000203978bef0e0] [c0000000001b8888] update_process_times+0x68/0x90 [ 252.789727] [c000203978bef110] [c0000000001d05b4] tick_sched_handle.isra.5+0x34/0xd0 [ 252.789731] [c000203978bef140] [c0000000001d06b0] tick_sched_timer+0x60/0xe0 [ 252.789737] [c000203978bef180] [c0000000001b9414] hrtimer_run_queues+0x144/0x370 [ 252.789743] [c000203978bef200] [c0000000001ba36c] hrtimer_interrupt+0xfc/0x350 [ 252.789749] [c000203978bef2d0] [c0000000000247b0] __timer_interrupt+0x90/0x260 [ 252.789755] [c000203978bef320] [c000000000024bc8] timer_interrupt+0x98/0xe0 [ 252.789761] [c000203978bef350] [c000000000009104] decrementer_common+0x114/0x120 [ 252.789770] --- interrupt: 901 at replay_interrupt_return+0x0/0x4 [ 252.789770] LR = arch_local_irq_restore+0x74/0x90 [ 252.789772] [c000203978bef640] [000000000000000b] 0xb (unreliable) [ 252.789778] [c000203978bef660] [c0000000001142d4] do_exit+0xf4/0xc80 [ 252.789784] [c000203978bef720] [c00000000002606c] oops_end+0x17c/0x210 [ 252.789789] [c000203978bef7a0] [c00000000006b2a0] bad_page_fault+0xe0/0x160 [ 252.789793] [c000203978bef810] [c000000000027654] slb_miss_bad_addr+0x34/0x70 [ 252.789799] [c000203978bef830] [c000000000008b0c] bad_addr_slb+0x11c/0x120 [ 252.789808] --- interrupt: 380 at dbs_update_util_handler+0x18/0xf0 [ 252.789808] LR = update_curr_rt+0xd0/0x3f0 [ 252.789810] [c000203978befb20] [0000000000000000] (null) (unreliable) [ 252.789815] [c000203978befbb0] [c00000000016b76c] dequeue_task_rt+0x3c/0xf0 [ 252.789820] [c000203978befbf0] [c00000000014da50] deactivate_task+0xb0/0x160 [ 252.789826] [c000203978befc70] [c000000000cfb9dc] __schedule+0x3bc/0xaf0 [ 252.789832] [c000203978befd40] [c000000000cfc150] schedule+0x40/0xc0 [ 252.789838] [c000203978befd60] [c000000000143cd4] smpboot_thread_fn+0x284/0x290 [ 252.789842] [c000203978befdc0] [c00000000013d8e8] kthread+0x1a8/0x1b0 [ 252.789849] [c000203978befe30] [c00000000000b658] ret_from_kernel_thread+0x5c/0x84 [ 252.789850] Instruction dump: [ 252.789852] 7d40192d 40c2fff0 7c2004ac 2fa90000 409e001c 38210030 ebe1fff8 4e800020 [ 252.789862] 60000000 60000000 60420000 7c210b78 89290009 71290002 40820050 [ 256.680280] Watchdog CPU:38 Hard LOCKUP [ 256.680281] Modules linked in: idt_89hpesx joydev input_leds mac_hid vmx_crypto crct10dif_vpmsum at24 ofpart ipmi_powernv cmdlinepart ipmi_devintf powernv_flash ipmi_msghandler opal_prd ibmpowernv uio_pdrv_genirq mtd uio sch_fq_codel ip_tables x_tables autofs4 ses enclosure scsi_transport_sas hid_generic usbhid hid ast ttm drm_kms_helper igb syscopyarea sysfillrect sysimgblt fb_sys_fops crc32c_vpmsum drm i40e i2c_algo_bit aacraid [ 256.680296] CPU: 38 PID: 3494 Comm: (agetty) Tainted: G D 4.15.0-33-generic #36-Ubuntu [ 256.680297] NIP: c000000000d023a4 LR: c000000000168010 CTR: c000000000ac0cd0 [ 256.680298] REGS: c00000003fe37d80 TRAP: 0900 Tainted: G D (4.15.0-33-generic) [ 256.680299] MSR: 9000000000009033 <SF,HV,EE,ME,IR,DR,RI,LE> CR: 24848844 XER: 20040000 [ 256.680302] CFAR: c000000000d023bc SOFTE: 0 [ 256.680302] GPR00: c000000000168010 c000003fb44a7c00 c0000000016eb000 c000003feed98580 [ 256.680302] GPR04: 0000000000000008 0000000000000008 0000000000000000 c0000000018b4f00 [ 256.680302] GPR08: 0000000000000008 0000000080000029 0000000080000026 0000000000000000 [ 256.680302] GPR12: c000000000ac0cd0 c00000000fa9a200 0000000000000004 0000000000000003 [ 256.680302] GPR16: c000003fd4bc1000 c00000000171dd78 c000003feed98580 0000003fedbc0000 [ 256.680302] GPR20: 0000000000000001 c000000000f75968 c00000000163527b 0000000000000000 [ 256.680302] GPR24: 0000000000000000 c0000000017220e0 c000000001722494 c000003ebfe5e600 [ 256.680302] GPR28: 0000000000000008 0000000000000040 c0000000011d8580 c000003feed98580 [ 256.680314] NIP [c000000000d023a4] _raw_spin_lock_irq+0x64/0xe0 [ 256.680316] LR [c000000000168010] online_fair_sched_group+0xd0/0x210 [ 256.680316] Call Trace: [ 256.680318] [c000003fb44a7c00] [c000003fb44a7c30] 0xc000003fb44a7c30 (unreliable) [ 256.680319] [c000003fb44a7c30] [c000000000168058] online_fair_sched_group+0x118/0x210 [ 256.680321] [c000003fb44a7d10] [c0000000001544f4] sched_online_group+0xc4/0xf0 [ 256.680323] [c000003fb44a7d50] [c000000000179258] sched_autogroup_create_attach+0xe8/0x200 [ 256.680324] [c000003fb44a7de0] [c00000000012c55c] sys_setsid+0x1bc/0x1f0 [ 256.680326] [c000003fb44a7e30] [c00000000000b284] system_call+0x58/0x6c [ 256.680327] Instruction dump: [ 256.680328] 7d201829 2c090000 40c20010 7d40192d 40c2fff0 7c2004ac 2fa90000 409e0010 [ 256.680330] 38210030 ebe1fff8 4e800020 7c210b78 89290009 71290002 40820050 [ 269.298623] aacraid: Host adapter abort request (0,2,0,0) [ 280.243007] i40e 0002:01:00.0: TX driver issue detected, PF reset issued

--== Welcome to Hostboot ==--

2.79640|secure|SecureROM valid - enabling functionality 5.47562|Ignoring boot flags, incorrect version 0x0 5.47922|Booting from SBE side 0 on master proc=00050000 5.67457|ISTEP 6. 5 - host_init_fsi 5.78212|ISTEP 6. 6 - host_set_ipl_parms 6.02715|ISTEP 6. 7 - host_discover_targets 6.72080|HWAS|PRESENT> DIMM[03]=A0A0000000000000 6.72085|HWAS|PRESENT> Proc[05]=8800000000000000 6.72086|HWAS|PRESENT> Core[07]=FFF3CFC3FFFF0000 6.92463|ISTEP 6. 8 - host_update_master_tpm 14.38499|SECURE|Security Access Bit> 0xC000000000000000 14.38500|SECURE|Secure Mode Disable (via Jumper)> 0x0000000000000000 14.38547|ISTEP 6. 9 - host_gard 14.93579|================================================ 14.93579|Error reported by prdf (0xE500) PLID 0x90000004 14.93580| PRD Signature : 0x50000 0xFE680000 14.94124| Signature Description : pu:k0:n0:s0:p00 (PBCENTFIR[0]) pb protocol_error 14.94291| UserData1 : 0x0005000000000101 14.94292| UserData2 : 0xfe68000000000000 14.94292|------------------------------------------------ 14.94502| Callout type : Procedure Callout 14.94503| Procedure : EPUB_PRC_LVL_SUPP 14.94503| Priority : SRCI_PRIORITY_MED 14.94504|------------------------------------------------ 14.94505| Callout type : Hardware Callout 14.94506| CPU id : 0 14.94507| Target : Physical:/Sys0/Node0/Proc0 14.94508| Deconfig State : NO_DECONFIG 14.94508| GARD Error Type : GARD_NULL 14.94509| Priority : SRCI_PRIORITY_LOW 14.94510|------------------------------------------------ 14.94510| 14.94511|------------------------------------------------ 14.94512| System checkstop occurred during runtime on previous boot 14.94513|------------------------------------------------ 14.94513| 14.94514|------------------------------------------------ 14.94514| Hostboot Build ID: 14.94515|================================================ 14.94647|HWAS|FUNCTIONAL> DIMM[03]=A0A0000000000000 14.94648|HWAS|FUNCTIONAL> Proc[05]=8800000000000000 14.94649|HWAS|FUNCTIONAL> Core[07]=FFF3CFC3FFFF0000 15.17252|ISTEP 6.10 - host_revert_sbe_mcs_setup 15.18312|ISTEP 6.11 - host_start_occ_xstop_handler 16.27499|ISTEP 6.12 - host_voltage_config

mikey commented 6 years ago

scoms can do all sorts of nasty things to your computer. Hitting a random scom and it crashing your machine is not a skiboot bug. Please ask the hardware team what the problem is here.

abalib commented 6 years ago

It was not a random but a documented scom register. I will check with the hardware team.