Open fszyang opened 7 years ago
Still happening on
commit f621bae58cfe467e4cbe5c1dd8d630c3c67dc3e7
Tried the stop/start test on invader2(same invader as previous) with a longer wait cycle (7 seconds) between stop and start, and it still hung, the CPU but did take longer this time (10 hours).
I see this messages repeated on console quite frequently throughout. Not sure if related to hang.
DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0b000 [fault reason 05] PTE Write access is not set
Only able to capture the last snippet on the console this time when CPU hung:
...
[\
Did the same test on invader7, and it did NOT hang over 13 hours.
Bother invader7 and invader2 have the same rev of goes f621bae58cfe467e4cbe5c1dd8d630c3c67dc3e7
and the same kernel 4.9.8-platina-mk1-amd64
SRIOV enabled
Saw similar messages on console as invader2. Less frequent and seem to happen in burst. DMAR: [DMA Write] Request device [04:00.0] fault addr ffc0e000 [fault reason 05] PTE Write access is not set DMAR: DRHD: handling fault status reg 500
Tested with newer ixgbevf driver version 4.4 on kernel 4.9.8. Do not see the crash/hang anymore on invader2 which was hanging previously.
After 2+ hours of repeated goes stop/start loops, cpu hangs Need power cycle to recover
tested on commit 73003a9d242d24005d1c6c222221447678a333eb
console shows:
BUG: unable to handle kernel NULL pointer dereference at 0000000000000008 IP: [\] ixgbevf_alloc_rx_buffers+0x88/0x1f0 [ixgbevf]
PGD 0
Oops: 0000 [#1] SMP
Modules linked in: xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 ixgbevf iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay iptable_raw nls_utf8 nls_cp437 vfat fat kvm_intel kvm uio_pci_dma i2c_i801 i2c_smbus autofs4 dm_mod ixgbe mdio
CPU: 7 PID: 8609 Comm: kworker/u16:2 Tainted: G W 4.9.8-platina-mk1-amd64 #2
Hardware name: Intel Camelback Mountain Platina DC/Camelback Mountain Platina DC, BIOS 6a724f1-dirty 03/14/2017
Workqueue: ixgbevf ixgbevf_service_task [ixgbevf]
task: ffff88046b0d0000 task.stack: ffffc90006f5c000
RIP: 0010:[\] [\] ixgbevf_alloc_rx_buffers+0x88/0x1f0 [ixgbevf]
RSP: 0018:ffffc90006f5fd28 EFLAGS: 00010287
RAX: 0000000000000200 RBX: 0000000000000000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 00000000000001ff RDI: ffff88044e1f9000
RBP: ffffc90006f5fd60 R08: 0000000000000000 R09: 0000000000100000
R10: 00000001006cecce R11: 0000000000000001 R12: 0000000000000000
R13: 00000000fffffe00 R14: 00000000000001ff R15: ffff88044e1f9000
FS: 0000000000000000(0000) GS:ffff88047fdc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 000000007ea07000 CR4: 00000000003406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000000 ffff880460919b28 ffff880460919840 0000000000000009
ffff880460919b28 ffff88044e1f9000 0000000000001028 ffffc90006f5fdc0
ffffffffa060b1d3 ffff880400001028 0000000000000000 ffff880460919ab0
Call Trace:
[\] ixgbevf_configure+0x5a3/0x710 [ixgbevf]
[\] ixgbevf_reinit_locked+0x38/0x70 [ixgbevf]
[\] ixgbevf_service_task+0x1a6/0x330 [ixgbevf]
[\] process_one_work+0x12a/0x330
[\] worker_thread+0x61/0x490
[\] ? process_one_work+0x330/0x330
[\] kthread+0xd7/0xf0
[\] ? kthread_park+0x60/0x60
[\] ret_from_fork+0x22/0x30
Code: 49 83 c4 10 48 83 c3 18 49 89 44 24 f0 41 83 c5 01 0f 84 14 01 00 00 66 41 83 ee 01 49 c7 44 24 08 00 00 00 00 0f 84 c8 00 00 00 \<48> 83 7b 08 00 75 c5 31 c9 31 f6 48 c7 c2 00 0d 87 81 bf 20 63
RIP [\] ixgbevf_alloc_rx_buffers+0x88/0x1f0 [ixgbevf]
RSP \
CR2: 0000000000000008
---[ end trace 41feb073b5b2881a ]---
BUG: unable to handle kernel NULL pointer dereference at 0000000000000009
IP: [\] wake_up_common+0x26/0x80
PGD 0
Oops: 0000 [#2] SMP
Modules linked in: xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 ixgbevf iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay iptable_raw nls_utf8 nls_cp437 vfat fat kvm_intel kvm uio_pci_dma i2c_i801 i2c_smbus autofs4 dm_mod ixgbe mdio
CPU: 7 PID: 8609 Comm: kworker/u16:2 Tainted: G D W 4.9.8-platina-mk1-amd64 #2
Hardware name: Intel Camelback Mountain Platina DC/Camelback Mountain Platina DC, BIOS 6a724f1-dirty 03/14/2017
task: ffff88046b0d0000 task.stack: ffffc90006f5c000
RIP: 0010:[\] [\] wake_up_common+0x26/0x80
RSP: 0018:ffffc90006f5fe50 EFLAGS: 00010086
RAX: 0000000000000082 RBX: ffffc90006f5ff18 RCX: 0000000000000000
RDX: 0000000000000009 RSI: 0000000000000003 RDI: ffffc90006f5ff18
RBP: ffffc90006f5fe88 R08: 0000000000000000 R09: 0000000000000000
R10: 000000000007e952 R11: 000000000007e952 R12: ffffc90006f5ff20
R13: 0000000000000082 R14: 0000000000000000 R15: 0000000000000003
FS: 0000000000000000(0000) GS:ffff88047fdc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000028 CR3: 000000007ea07000 CR4: 00000000003406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000100000046 0000000000000000 ffffc90006f5ff18 ffffc90006f5ff10
0000000000000082 0000000000000000 0000000000000000 ffffc90006f5fe98
ffffffff810b0e6e ffffc90006f5fec0 ffffffff810b17e2 0000000000000000
Call Trace:
[\] wake_up_locked+0xe/0x10
[\] complete+0x32/0x50
[\] mm_release+0x9e/0x100
[\] do_exit+0x284/0xab0
[\] rewind_stack_do_exit+0x17/0x19
Code: 00 00 00 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 41 89 f7 53 4c 8d 67 08 41 89 ce 48 83 ec 10 89 55 cc 48 8b 57 08 4c 89 45 d0 \<48> 8b 32 49 39 d4 48 8d 42 e8 4c 8d 6e e8 75 05 eb 38 49 89 d5
RIP [\] wake_up_common+0x26/0x80
RSP \
CR2: 0000000000000009
---[ end trace 41feb073b5b2881b ]---
Fixing recursive fault but reboot is needed!
BUG: unable to handle kernel paging request at ffffffffffffffd8
IP: [\] kthread_data+0xb/0x20
PGD 7ea08067 PUD 7ea0a067
PMD 0
Oops: 0000 [#3] SMP
Modules linked in: xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 ixgbevf iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay iptable_raw nls_utf8 nls_cp437 vfat fat kvm_intel kvm uio_pci_dma i2c_i801 i2c_smbus autofs4 dm_mod ixgbe mdio
CPU: 7 PID: 8609 Comm: kworker/u16:2 Tainted: G D W 4.9.8-platina-mk1-amd64 #2
Hardware name: Intel Camelback Mountain Platina DC/Camelback Mountain Platina DC, BIOS 6a724f1-dirty 03/14/2017
task: ffff88046b0d0000 task.stack: ffffc90006f5c000
RIP: 0010:[\] [\] kthread_data+0xb/0x20
RSP: 0018:ffffc90006f5fe78 EFLAGS: 00010002
RAX: 0000000000000000 RBX: 0000000000015300 RCX: 0000000000000007
RDX: ffff88046d805000 RSI: ffff88046b0d0000 RDI: ffff88046b0d0000
RBP: ffffc90006f5fe78 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000001 R11: 00000000000001da R12: ffff88047fdd5300
R13: ffff88046b0d0000 R14: ffff88046b0d04b0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff88047fdc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000028 CR3: 000000007ea07000 CR4: 00000000003406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffc90006f5fe88 ffffffff81092a79 ffffc90006f5fec8 ffffffff814c328e
ffffc90006f5fef0 ffff88046b0d0000 0000000000000009 0000000000000046
ffff88046b0d0000 0000000000000000 ffffc90006f5fee0 ffffffff814c3571
Call Trace:
[\] wq_worker_sleeping+0x9/0x80
[\] schedule+0x2ce/0x580
[\] schedule+0x31/0x80
[\] do_exit+0x884/0xab0
[\] rewind_stack_do_exit+0x17/0x19
Code: be 7d 04 00 00 48 c7 c7 b8 75 6d 81 e8 af 5c fe ff eb ca 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 48 8b 87 50 04 00 00 55 48 89 e5 \<48> 8b 40 d8 5d c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00
RIP [\] kthread_data+0xb/0x20
RSP \
CR2: ffffffffffffffd8
---[ end trace 41feb073b5b2881c ]---
Fixing recursive fault but reboot is needed!
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=794
(detected by 4, t=5255 jiffies, g=985507, c=985506, q=549)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[\] ? oops_exit+0x22/0x30
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] wait_for_xmitr+0x32/0x90
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] ? wait_for_xmitr+0x32/0x90
[\] ? serial8250_console_write+0x204/0x250
[\] ? bad_area_nosemaphore+0xf/0x20
[\] ? univ8250_console_write+0x19/0x20
[\] ? irq_work_queue+0xf/0x70
[\] ? wake_up_klogd+0x2f/0x40
[\] ? console_unlock+0x51f/0x550
[\] ? vprintk_emit+0x273/0x430
[\] ? _raw_spin_lock+0x1b/0x20
[\] ? schedule+0x6b/0x580
[\] ? schedule+0x31/0x80
[\] ? do_exit+0x884/0xab0
[\] ? rewind_stack_do_exit+0x17/0x19
rcu_sched kthread starved for 3604 jiffies! g985507 c985506 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
rcu_sched S 0 7 2 0x00000000
0000000000000000 ffff88046182a580 ffff88046d8f4800 ffff88047fc55300
ffff88046b1e3c00 ffffc9000003bdb8 ffffffff814c314e 0000000000000282
ffff88046d8f4800 ffff88047fc4eb80 ffff88047fc4eb80 ffffc9000003bdf0
Call Trace:
[\] ? schedule+0x18e/0x580
[\] schedule+0x31/0x80
[\] schedule_timeout+0xf9/0x180
[\] ? del_timer_sync+0x50/0x50
[\] rcu_gp_kthread+0x3be/0x7d0
[\] ? force_qs_rnp+0x180/0x180
[\] kthread+0xd7/0xf0
[\] ? kthread_park+0x60/0x60
[\] ? kthread_park+0x60/0x60
[\] ret_from_fork+0x22/0x30
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=5824
(detected by 0, t=21007 jiffies, g=985507, c=985506, q=568)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[\] ? oops_exit+0x22/0x30
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] wait_for_xmitr+0x32/0x90
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] ? wait_for_xmitr+0x32/0x90
[\] ? serial8250_console_write+0x204/0x250
[\] ? bad_area_nosemaphore+0xf/0x20
[\] ? univ8250_console_write+0x19/0x20
[\] ? irq_work_queue+0xf/0x70
[\] ? wake_up_klogd+0x2f/0x40
[\] ? console_unlock+0x51f/0x550
[\] ? vprintk_emit+0x273/0x430
[\] ? _raw_spin_lock+0x1b/0x20
[\] ? schedule+0x6b/0x580
[\] ? schedule+0x31/0x80
[\] ? do_exit+0x884/0xab0
[\] ? rewind_stack_do_exit+0x17/0x19
rcu_sched kthread starved for 5402 jiffies! g985507 c985506 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x1
rcu_sched S 0 7 2 0x00000000
0000000000000000 ffff88046182a580 ffff88046d8f4800 ffff88047fd15300
ffff88046d96a400 ffffc9000003bdb8 ffffffff814c314e 0000000000000282
ffff88046d8f4800 ffff88047fd0eb80 ffff88047fd0eb80 ffffc9000003bdf0
Call Trace:
[\] ? schedule+0x18e/0x580
[\] schedule+0x31/0x80
[\] schedule_timeout+0xf9/0x180
[\] ? del_timer_sync+0x50/0x50
[\] rcu_gp_kthread+0x3be/0x7d0
[\] ? force_qs_rnp+0x180/0x180
[\] kthread+0xd7/0xf0
[\] ? kthread_park+0x60/0x60
[\] ? kthread_park+0x60/0x60
[\] ret_from_fork+0x22/0x30
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=13674
(detected by 0, t=36762 jiffies, g=985507, c=985506, q=570)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[\] ? oops_exit+0x22/0x30
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] wait_for_xmitr+0x32/0x90
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] ? wait_for_xmitr+0x32/0x90
[\] ? serial8250_console_write+0x204/0x250
[\] ? bad_area_nosemaphore+0xf/0x20
[\] ? univ8250_console_write+0x19/0x20
[\] ? irq_work_queue+0xf/0x70
[\] ? wake_up_klogd+0x2f/0x40
[\] ? console_unlock+0x51f/0x550
[\] ? vprintk_emit+0x273/0x430
[\] ? _raw_spin_lock+0x1b/0x20
[\] ? schedule+0x6b/0x580
[\] ? schedule+0x31/0x80
[\] ? do_exit+0x884/0xab0
[\] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=21535
(detected by 0, t=52517 jiffies, g=985507, c=985506, q=574)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[\] ? oops_exit+0x22/0x30
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] wait_for_xmitr+0x32/0x90
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] ? wait_for_xmitr+0x32/0x90
[\] ? serial8250_console_write+0x204/0x250
[\] ? bad_area_nosemaphore+0xf/0x20
[\] ? univ8250_console_write+0x19/0x20
[\] ? irq_work_queue+0xf/0x70
[\] ? wake_up_klogd+0x2f/0x40
[\] ? console_unlock+0x51f/0x550
[\] ? vprintk_emit+0x273/0x430
[\] ? _raw_spin_lock+0x1b/0x20
[\] ? schedule+0x6b/0x580
[\] ? schedule+0x31/0x80
[\] ? do_exit+0x884/0xab0
[\] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=29396
(detected by 0, t=68272 jiffies, g=985507, c=985506, q=576)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[\] ? oops_exit+0x22/0x30
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] wait_for_xmitr+0x32/0x90
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] ? wait_for_xmitr+0x32/0x90
[\] ? serial8250_console_write+0x204/0x250
[\] ? bad_area_nosemaphore+0xf/0x20
[\] ? univ8250_console_write+0x19/0x20
[\] ? irq_work_queue+0xf/0x70
[\] ? wake_up_klogd+0x2f/0x40
[\] ? console_unlock+0x51f/0x550
[\] ? vprintk_emit+0x273/0x430
[\] ? _raw_spin_lock+0x1b/0x20
[\] ? schedule+0x6b/0x580
[\] ? schedule+0x31/0x80
[\] ? do_exit+0x884/0xab0
[\] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=37257
(detected by 0, t=84027 jiffies, g=985507, c=985506, q=580)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[\] ? oops_exit+0x22/0x30
[\] ? delay+0xa/0x10
[\] ? __const_udelay+0x2a/0x30
[\] wait_for_xmitr+0x32/0x90
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] ? wait_for_xmitr+0x32/0x90
[\] ? serial8250_console_write+0x204/0x250
[\] ? bad_area_nosemaphore+0xf/0x20
[\] ? univ8250_console_write+0x19/0x20
[\] ? irq_work_queue+0xf/0x70
[\] ? wake_up_klogd+0x2f/0x40
[\] ? console_unlock+0x51f/0x550
[\] ? vprintk_emit+0x273/0x430
[\] ? _raw_spin_lock+0x1b/0x20
[\] ? schedule+0x6b/0x580
[\] ? schedule+0x31/0x80
[\] ? do_exit+0x884/0xab0
[\] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=45118
(detected by 0, t=99782 jiffies, g=985507, c=985506, q=582)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[\] ? oops_exit+0x22/0x30
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] wait_for_xmitr+0x32/0x90
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] ? wait_for_xmitr+0x32/0x90
[\] ? serial8250_console_write+0x204/0x250
[\] ? bad_area_nosemaphore+0xf/0x20
[\] ? univ8250_console_write+0x19/0x20
[\] ? irq_work_queue+0xf/0x70
[\] ? wake_up_klogd+0x2f/0x40
[\] ? console_unlock+0x51f/0x550
[\] ? vprintk_emit+0x273/0x430
[\] ? _raw_spin_lock+0x1b/0x20
[\] ? schedule+0x6b/0x580
[\] ? schedule+0x31/0x80
[\] ? do_exit+0x884/0xab0
[\] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=52979
(detected by 0, t=115537 jiffies, g=985507, c=985506, q=586)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[\] ? oops_exit+0x22/0x30
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] wait_for_xmitr+0x32/0x90
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] ? wait_for_xmitr+0x32/0x90
[\] ? serial8250_console_write+0x204/0x250
[\] ? bad_area_nosemaphore+0xf/0x20
[\] ? univ8250_console_write+0x19/0x20
[\] ? irq_work_queue+0xf/0x70
[\] ? wake_up_klogd+0x2f/0x40
[\] ? console_unlock+0x51f/0x550
[\] ? vprintk_emit+0x273/0x430
[\] ? _raw_spin_lock+0x1b/0x20
[\] ? schedule+0x6b/0x580
[\] ? schedule+0x31/0x80
[\] ? do_exit+0x884/0xab0
[\] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=60840
(detected by 0, t=131292 jiffies, g=985507, c=985506, q=588)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[\] ? oops_exit+0x22/0x30
[\] ? delay+0xa/0x10
[\] ? __const_udelay+0x2a/0x30
[\] wait_for_xmitr+0x32/0x90
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] ? wait_for_xmitr+0x32/0x90
[\] ? serial8250_console_write+0x204/0x250
[\] ? bad_area_nosemaphore+0xf/0x20
[\] ? univ8250_console_write+0x19/0x20
[\] ? irq_work_queue+0xf/0x70
[\] ? wake_up_klogd+0x2f/0x40
[\] ? console_unlock+0x51f/0x550
[\] ? vprintk_emit+0x273/0x430
[\] ? _raw_spin_lock+0x1b/0x20
[\] ? schedule+0x6b/0x580
[\] ? schedule+0x31/0x80
[\] ? do_exit+0x884/0xab0
[\] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=68701
(detected by 0, t=147047 jiffies, g=985507, c=985506, q=588)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[\] ? oops_exit+0x22/0x30
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] wait_for_xmitr+0x32/0x90
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] ? wait_for_xmitr+0x32/0x90
[\] ? serial8250_console_write+0x204/0x250
[\] ? bad_area_nosemaphore+0xf/0x20
[\] ? univ8250_console_write+0x19/0x20
[\] ? irq_work_queue+0xf/0x70
[\] ? wake_up_klogd+0x2f/0x40
[\] ? console_unlock+0x51f/0x550
[\] ? vprintk_emit+0x273/0x430
[\] ? _raw_spin_lock+0x1b/0x20
[\] ? schedule+0x6b/0x580
[\] ? schedule+0x31/0x80
[\] ? do_exit+0x884/0xab0
[\] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=76562
(detected by 0, t=162802 jiffies, g=985507, c=985506, q=588)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[\] ? oops_exit+0x22/0x30
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] wait_for_xmitr+0x32/0x90
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] ? wait_for_xmitr+0x32/0x90
[\] ? serial8250_console_write+0x204/0x250
[\] ? bad_area_nosemaphore+0xf/0x20
[\] ? univ8250_console_write+0x19/0x20
[\] ? irq_work_queue+0xf/0x70
[\] ? wake_up_klogd+0x2f/0x40
[\] ? console_unlock+0x51f/0x550
[\] ? vprintk_emit+0x273/0x430
[\] ? _raw_spin_lock+0x1b/0x20
[\] ? schedule+0x6b/0x580
[\] ? schedule+0x31/0x80
[\] ? do_exit+0x884/0xab0
[\] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=84423
(detected by 0, t=178557 jiffies, g=985507, c=985506, q=588)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[\] ? oops_exit+0x22/0x30
[\] ? delay+0xa/0x10
[\] ? __const_udelay+0x2a/0x30
[\] wait_for_xmitr+0x32/0x90
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] ? wait_for_xmitr+0x32/0x90
[\] ? serial8250_console_write+0x204/0x250
[\] ? bad_area_nosemaphore+0xf/0x20
[\] ? univ8250_console_write+0x19/0x20
[\] ? irq_work_queue+0xf/0x70
[\] ? wake_up_klogd+0x2f/0x40
[\] ? console_unlock+0x51f/0x550
[\] ? vprintk_emit+0x273/0x430
[\] ? _raw_spin_lock+0x1b/0x20
[\] ? schedule+0x6b/0x580
[\] ? schedule+0x31/0x80
[\] ? do_exit+0x884/0xab0
[\] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=92284
(detected by 0, t=194312 jiffies, g=985507, c=985506, q=588)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[\] ? oops_exit+0x22/0x30
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] wait_for_xmitr+0x32/0x90
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] ? wait_for_xmitr+0x32/0x90
[\] ? serial8250_console_write+0x204/0x250
[\] ? bad_area_nosemaphore+0xf/0x20
[\] ? univ8250_console_write+0x19/0x20
[\] ? irq_work_queue+0xf/0x70
[\] ? wake_up_klogd+0x2f/0x40
[\] ? console_unlock+0x51f/0x550
[\] ? vprintk_emit+0x273/0x430
[\] ? _raw_spin_lock+0x1b/0x20
[\] ? schedule+0x6b/0x580
[\] ? schedule+0x31/0x80
[\] ? do_exit+0x884/0xab0
[\] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=100145
(detected by 0, t=210067 jiffies, g=985507, c=985506, q=588)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[\] ? oops_exit+0x22/0x30
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] wait_for_xmitr+0x32/0x90
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] ? wait_for_xmitr+0x32/0x90
[\] ? serial8250_console_write+0x204/0x250
[\] ? bad_area_nosemaphore+0xf/0x20
[\] ? univ8250_console_write+0x19/0x20
[\] ? irq_work_queue+0xf/0x70
[\] ? wake_up_klogd+0x2f/0x40
[\] ? console_unlock+0x51f/0x550
[\] ? vprintk_emit+0x273/0x430
[\] ? _raw_spin_lock+0x1b/0x20
[\] ? schedule+0x6b/0x580
[\] ? schedule+0x31/0x80
[\] ? do_exit+0x884/0xab0
[\] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=108006
(detected by 0, t=225822 jiffies, g=985507, c=985506, q=588)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[\] ? oops_exit+0x22/0x30
[\] ? delay+0xa/0x10
[\] ? __const_udelay+0x2a/0x30
[\] wait_for_xmitr+0x32/0x90
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] ? wait_for_xmitr+0x32/0x90
[\] ? serial8250_console_write+0x204/0x250
[\] ? bad_area_nosemaphore+0xf/0x20
[\] ? univ8250_console_write+0x19/0x20
[\] ? irq_work_queue+0xf/0x70
[\] ? wake_up_klogd+0x2f/0x40
[\] ? console_unlock+0x51f/0x550
[\] ? vprintk_emit+0x273/0x430
[\] ? _raw_spin_lock+0x1b/0x20
[\] ? schedule+0x6b/0x580
[\] ? schedule+0x31/0x80
[\] ? do_exit+0x884/0xab0
[\] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=115867
(detected by 0, t=241577 jiffies, g=985507, c=985506, q=588)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[\] ? oops_exit+0x22/0x30
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] wait_for_xmitr+0x32/0x90
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] ? wait_for_xmitr+0x32/0x90
[\] ? serial8250_console_write+0x204/0x250
[\] ? bad_area_nosemaphore+0xf/0x20
[\] ? univ8250_console_write+0x19/0x20
[\] ? irq_work_queue+0xf/0x70
[\] ? wake_up_klogd+0x2f/0x40
[\] ? console_unlock+0x51f/0x550
[\] ? vprintk_emit+0x273/0x430
[\] ? _raw_spin_lock+0x1b/0x20
[\] ? schedule+0x6b/0x580
[\] ? schedule+0x31/0x80
[\] ? do_exit+0x884/0xab0
[\] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=123728
(detected by 0, t=257332 jiffies, g=985507, c=985506, q=588)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[\] ? oops_exit+0x22/0x30
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] wait_for_xmitr+0x32/0x90
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] ? wait_for_xmitr+0x32/0x90
[\] ? serial8250_console_write+0x204/0x250
[\] ? bad_area_nosemaphore+0xf/0x20
[\] ? univ8250_console_write+0x19/0x20
[\] ? irq_work_queue+0xf/0x70
[\] ? wake_up_klogd+0x2f/0x40
[\] ? console_unlock+0x51f/0x550
[\] ? vprintk_emit+0x273/0x430
[\] ? _raw_spin_lock+0x1b/0x20
[\] ? schedule+0x6b/0x580
[\] ? schedule+0x31/0x80
[\] ? do_exit+0x884/0xab0
[\] ? rewind_stack_do_exit+0x17/0x19
INFO: rcu_sched detected stalls on CPUs/tasks:
7-...: (2 GPs behind) idle=b43/140000000000000/0 softirq=7775986/7775986 fqs=131589
(detected by 0, t=273087 jiffies, g=985507, c=985506, q=588)
Task dump for CPU 7:
kworker/u16:2 D 0 8609 2 0x00000008
ffffc90006f5fc58 ffffffff8107d522 ffffc90006f5fc80 ffffc90006f5fc70
ffffffff812653ba ffffc90006f5fc80 ffffffff812653ea ffffc90006f5fca8
ffffffff812f6842 ffffffff8199e200 000000000000000a ffffffff81968c2d
Call Trace:
[\] ? oops_exit+0x22/0x30
[\] ? delay+0xa/0x10
[\] ? __const_udelay+0x2a/0x30
[\] wait_for_xmitr+0x32/0x90
[\] ? delay+0xa/0x10
[\] ? const_udelay+0x2a/0x30
[\] ? wait_for_xmitr+0x32/0x90
[\] ? serial8250_console_write+0x204/0x250
[\] ? bad_area_nosemaphore+0xf/0x20
[\] ? univ8250_console_write+0x19/0x20
[\] ? irq_work_queue+0xf/0x70
[\] ? wake_up_klogd+0x2f/0x40
[\] ? console_unlock+0x51f/0x550
[\] ? vprintk_emit+0x273/0x430
[\] ? _raw_spin_lock+0x1b/0x20
[\] ? schedule+0x6b/0x580
[\] ? schedule+0x31/0x80
[\] ? do_exit+0x884/0xab0
[\] ? rewind_stack_do_exit+0x17/0x19