kernel oops under high load

[mooinglemur]

We have a number of machines using enhanceio, and we've been trying to nail down some lockups that happen under load. The load has increased substantially, and now we're seeing frequent lockups. I changed the kernel to panic on oops, so that these machines will reboot themselves, so it's not a crisis. :)

I was finally able to capture a lockup that happened early in a boot. Since this oops seems to render the machine locked up and useless, panicking seems like the right thing. Let me know if I can provide more info.

[  115.652110] kernel BUG at /usr/src/enhanceio/EnhanceIO/Driver/enhanceio/eio_main.c:1518!

[  115.660334] invalid opcode: 0000 [#1] SMP 
[  115.664866] Modules linked in: enhanceio_rand(O) enhanceio_lru(O) enhanceio_fifo(O) enhanceio(O) i2c_i801 coretemp
[  115.675747] CPU: 0 PID: 1350 Comm: kworker/u48:3 Tainted: G        W  O 3.10.7-gentoo #3
[  115.683923] Hardware name: Supermicro X8DTU/X8DTU, BIOS 2.1a       08/27/11  
[  115.691131] Workqueue: eio_callback eio_post_io_callback [enhanceio]
[  115.697613] task: ffff88081f336190 ti: ffff88081e6ce000 task.ti: ffff88081e6ce000
[  115.705174] RIP: 0010:[<ffffffffa002b6f2>]  [<ffffffffa002b6f2>] eio_md_write+0x232/0x240 [enhanceio]
[  115.714548] RSP: 0018:ffff88081e6cfdc8  EFLAGS: 00010286
[  115.719922] RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000001
[  115.727120] RDX: 0000000000000020 RSI: 0000000000000046 RDI: 0000000000000282
[  115.734323] RBP: ffffc90074ccc640 R08: ffffffff81820fd8 R09: 0000000000000000
[  115.741526] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880750671ac0
[  115.748726] R13: 0000000000001888 R14: 0000000000000213 R15: ffff880819588000
[  115.755930] FS:  0000000000000000(0000) GS:ffff88083f200000(0000) knlGS:0000000000000000
[  115.764104] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  115.769915] CR2: 00007fa90863a7b2 CR3: 0000000001b28000 CR4: 00000000000007f0
[  115.777122] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  115.784322] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  115.791526] Stack:
[  115.793604]  ffff880750a729c0 0000000100000282 ffff88083bde57c0 ffff88083bde57c0
[  115.801347]  ffff88083bc97c00 ffff880818ec0900 0000000000000000 0000000000000000
[  115.809079]  ffff8807505cdaa0 ffffffff810ae1d3 ffff880818ec0900 0000000000000000
[  115.816818] Call Trace:
[  115.819333]  [<ffffffff810ae1d3>] ? process_one_work+0x153/0x420
[  115.825405]  [<ffffffff810af074>] ? worker_thread+0x114/0x370
[  115.831219]  [<ffffffff810aef60>] ? manage_workers.isra.26+0x2c0/0x2c0
[  115.837819]  [<ffffffff810b4743>] ? kthread+0xb3/0xc0
[  115.842936]  [<ffffffff810b4690>] ? __kthread_parkme+0x80/0x80
[  115.848840]  [<ffffffff81810d3c>] ? ret_from_fork+0x7c/0xb0
[  115.854482]  [<ffffffff810b4690>] ? __kthread_parkme+0x80/0x80
[  115.860383] Code: 4c 89 7a 30 48 89 4a 38 48 89 55 40 48 c7 43 38 00 00 00 00 e9 16 ff ff ff 48 89 4d 38 49 89 6c 24 18 e9 3b fe ff ff 0f 0b 0f 0b <0f> 0b 0f 0b 66 2e 0f 1f 84 00 00 00 00 00 48 83 ec 48 48 89 5c 
[  115.883270] RIP  [<ffffffffa002b6f2>] eio_md_write+0x232/0x240 [enhanceio]
[  115.890274]  RSP <ffff88081e6cfdc8>
[  115.894458] ---[ end trace 5bd50fb860d73366 ]---
[  115.901224] Kernel panic - not syncing: Fatal exception
[  116.895030] Rebooting in 300 seconds..

[mooinglemur]

I'm going to add here that I haven't seen this happen on boxes that exclusively have drives 2TB and smaller. It has happened on machines with 3TB and 4TB drives. I don't know if it's another one of these 32 bit LBA bugs, but I thought it was noteworthy.

[grexi]

I've seen the same behaviour:

Feb  6 20:05:57 wirt kernel: [907547.614898] ------------[ cut here ]------------
Feb  6 20:05:57 wirt kernel: [907547.615168] kernel BUG at /var/lib/dkms/enhanceio/0.1/build/eio_main.c:1518!
Feb  6 20:05:57 wirt kernel: [907547.615580] invalid opcode: 0000 [#1] SMP 
Feb  6 20:05:57 wirt kernel: [907547.615834] Modules linked in: btrfs(F) zlib_deflate(F) ufs(F) qnx4(F) hfsplus(F) hfs(F) minix(F) ntfs(F) msdos(F) jfs(F) xfs(F) libcrc32c(F) reiserfs(F) ext2(F) vesaf
b(F) pci_stub vboxpci(OF) vboxnetadp(OF) vboxnetflt(OF) vboxdrv(OF) enhanceio_lru(OF) enhanceio_fifo(OF) enhanceio(OF) gpio_ich x86_pkg_temp_thermal coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmu
l ghash_clmulni_intel aesni_intel mxm_wmi aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd microcode joydev mei_me lpc_ich mei ioatdma wmi lp mac_hid parport raid10 raid456 async_raid6_recov asy
nc_memcpy async_pq async_xor async_tx xor hid_generic raid6_pq raid1 raid0 igb isci multipath usbhid i2c_algo_bit ahci libsas usb_storage dca hid linear libahci ptp scsi_transport_sas pps_core
Feb  6 20:05:57 wirt kernel: [907547.653858] CPU: 0 PID: 20999 Comm: kworker/u49:0 Tainted: GF          O 3.11.0-15-generic #23-Ubuntu
Feb  6 20:05:57 wirt kernel: [907547.676584] Hardware name: Thomas-Krenn.AG X9DR3-F/X9DR3-F, BIOS 3.0a 07/31/2013
Feb  6 20:05:57 wirt kernel: [907547.700132] Workqueue: eio_callback eio_post_io_callback [enhanceio]
Feb  6 20:05:57 wirt kernel: [907547.710692] task: ffff8808511faee0 ti: ffff880b448ac000 task.ti: ffff880b448ac000
Feb  6 20:05:57 wirt kernel: [907547.731170] RIP: 0010:[<ffffffffa03c54bc>]  [<ffffffffa03c54bc>] eio_md_write+0x24c/0x250 [enhanceio]
Feb  6 20:05:57 wirt kernel: [907547.752336] RSP: 0018:ffff880b448add90  EFLAGS: 00010206
Feb  6 20:05:57 wirt kernel: [907547.762933] RAX: 0000000000000001 RBX: 000000000001facf RCX: 0000000000000001
Feb  6 20:05:57 wirt kernel: [907547.784307] RDX: 0000000000000000 RSI: 0000000000000046 RDI: 0000000000000282
Feb  6 20:05:57 wirt kernel: [907547.806288] RBP: ffff880b448addd0 R08: ffff88085fc14580 R09: 0000000000000001
Feb  6 20:05:57 wirt kernel: [907547.829061] R10: 0000000000000000 R11: 0000000000014580 R12: ffff88033cbb8a80
Feb  6 20:05:57 wirt kernel: [907547.852220] R13: 0000000000000001 R14: 0000000000000000 R15: ffffc9002eeaca38
Feb  6 20:05:57 wirt kernel: [907547.875239] FS:  0000000000000000(0000) GS:ffff88085fc00000(0000) knlGS:0000000000000000
Feb  6 20:05:57 wirt kernel: [907547.899255] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb  6 20:05:57 wirt kernel: [907547.910856] CR2: 00007f35579804e0 CR3: 0000000001c0e000 CR4: 00000000001427f0
Feb  6 20:05:57 wirt kernel: [907547.933950] Stack:
Feb  6 20:05:57 wirt kernel: [907547.944918]  ffff88085190c000 0000000000000203 ffff88033cbb8700 ffff880cbbe16e70
Feb  6 20:05:57 wirt kernel: [907547.967166]  ffff88085190c000 ffff880289f91960 0000000000000000 000000000001facf
Feb  6 20:05:57 wirt kernel: [907547.989756]  ffff880b448ade20 ffffffffa03c5fc9 ffff880800000062 0000000001facf11
Feb  6 20:05:57 wirt kernel: [907548.012107] Call Trace:
Feb  6 20:05:57 wirt kernel: [907548.023179]  [<ffffffffa03c5fc9>] eio_post_io_callback+0x739/0x950 [enhanceio]
Feb  6 20:05:57 wirt kernel: [907548.045135]  [<ffffffff8107cfec>] process_one_work+0x17c/0x430
Feb  6 20:05:57 wirt kernel: [907548.056189]  [<ffffffff8107dfac>] worker_thread+0x11c/0x3c0
Feb  6 20:05:57 wirt kernel: [907548.066831]  [<ffffffff8107de90>] ? manage_workers.isra.25+0x2a0/0x2a0
Feb  6 20:05:57 wirt kernel: [907548.077500]  [<ffffffff81084740>] kthread+0xc0/0xd0
Feb  6 20:05:57 wirt kernel: [907548.088052]  [<ffffffff81084680>] ? kthread_create_on_node+0x120/0x120
Feb  6 20:05:57 wirt kernel: [907548.098605]  [<ffffffff816f716c>] ret_from_fork+0x7c/0xb0
Feb  6 20:05:57 wirt kernel: [907548.108893]  [<ffffffff81084680>] ? kthread_create_on_node+0x120/0x120
Feb  6 20:05:57 wirt kernel: [907548.119173] Code: 48 89 72 38 48 89 42 30 49 89 57 40 49 c7 46 38 00 00 00 00 e9 05 ff ff ff 49 89 4d 38 4d 89 6c 24 18 e9 2a fe ff ff 0f 0b 0f 0b <0f> 0b 0f 0b 0f 1f 44 00 00 55 48 89 e5 41 57 49 89 ff 41 56 41 
Feb  6 20:05:57 wirt kernel: [907548.151012] RIP  [<ffffffffa03c54bc>] eio_md_write+0x24c/0x250 [enhanceio]
Feb  6 20:05:57 wirt kernel: [907548.161026]  RSP <ffff880b448add90>
Feb  6 20:06:01 wirt kernel: [907548.183458] ---[ end trace 838f3d5ef81c5266 ]---

[devsk]

any comments on this by the devs? Is this project even alive?

[bhansaliakhil]

I know it is bit too late to ask for details, but I was curious if you have details around following, I will try and reproduce the same on my local setup.

Could you please share following details...

1. Cache configuration details (mode, block size of cache, policy), HDD, SSD sizes.
2. Workload details, is it fio, vdbench or anything else for load generation?
3. I presume it is writeback cache because the stack trace of panic show it is "eio_md_write" function stack which could only be run as part of write back cache.
4. I assume the workload might be random and write intensive which lead to metadata writes in large numbers. I could not speculate what lead to CPU lockup but would be good to have a machine reproducing this issue.

Thank you.

[enealDC]

I have seen the same issue on the ticket I just opened. I guess ill have to look at an alternative.

[spctm]

I have hit the same issue as noted here. I had set up Linux host as a iSCSI target on which there were about 10 Virtual machines running on VMware vsphere. Of the 10 VMs 5 were running vdbench, dt, and sqliosim. I have 3 SSDs in raid0 doing the duty of WB cache for a raid5 set of 5 HDDs.

~# eio_cli info Cache Name : raidvol1_cache Source Device : /dev/md126 SSD Device : /dev/md127 Policy : lru Mode : Write Back Block Size : 4096 Associativity : 256 State : normal

kernel: [2573885.309343] kernel BUG at /root/enhanceio/EnhanceIO/Driver/enhanceio/eio_main.c:1518! kernel: [2573885.309445] invalid opcode: 0000 [#1] SMP kernel: [2573885.309505] Modules linked in: ib_srpt ib_cm ib_sa ib_mad ib_core tcm_qla2xxx qla2xxx tcm_loop tcm_fc libfc scsi_transport_fc scsi_tgt iscsi_target_mod target_core_pscsi target_core_file target_c ore_iblock target_core_mod configfs nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache enhanceio_rand(OF) enhanceio_lru(OF) enhanceio_fifo(OF) enhanceio(OF) snd_hda_codec_hdmi eeepc_wmi asus_wmi sparse_keymap kvmamd kvm crct10dif pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd snd_hda_codec_realtek nouveau serio_raw snd_hda_intel edac_core k10temp edac_mce_amd fam15h_power snd_hda_codec snd_hwdep m xm_wmi video ttm joydev drm_kms_helper snd_pcm snd_page_alloc drm snd_timer sp5100_tco i2c_piix4 snd i2c_algo_bit soundcore 8021q garp stp mrp llc mac_hid wmi lp parport raid10 raid456 async_raid6_recov async_memcpy async_pq async _xor async_tx hid_generic usbhid hid xor raid6_pq psmouse raid1 r8169 mii ixgbe raid0 dca mpt2sas ptp multipath ahci pps_core raid_class libahci linear scsi_transport_sas mdio kernel: [2573885.310644] CPU: 2 PID: 976 Comm: kworker/u12:2 Tainted: GF O 3.13.0-19-generic #40-Ubuntu kernel: [2573885.310750] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A99FX PRO R2.0, BIOS 2005 09/16/2013 kernel: [2573885.310865] Workqueue: eio_callback eio_post_io_callback [enhanceio] kernel: [2573885.310931] task: ffff8807b6465fc0 ti: ffff8807ba7ce000 task.ti: ffff8807ba7ce000 kernel: [2573885.311031] RIP: 0010:[] [] eio_md_write+0x24c/0x250 [enhanceio] kernel: [2573885.311140] RSP: 0018:ffff8807ba7cfd88 EFLAGS: 00010202 kernel: [2573885.311198] RAX: 0000000000000001 RBX: 00000000000088aa RCX: 00000000ffffffff kernel: [2573885.311296] RDX: 0000000000000001 RSI: 000000002b1e2b1c RDI: 0000000000000286 kernel: [2573885.311394] RBP: ffff8807ba7cfdc8 R08: ffff88082ec94440 R09: 0000000000000001 kernel: [2573885.311493] R10: ffffffff8114f747 R11: ffffea001ee657c0 R12: ffff88065f53ae80 kernel: [2573885.311591] R13: 0000000000000001 R14: 0000000000000000 R15: ffffc900528b5fd0 kernel: [2573885.311690] FS: 00007f35f4fe9700(0000) GS:ffff88082ec80000(0000) knlGS:0000000000000000 kernel: [2573885.311792] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b kernel: [2573885.311853] CR2: 00007fac050e4000 CR3: 00000007bd69b000 CR4: 00000000000407e0 kernel: [2573885.311953] Stack: kernel: [2573885.312001] ffff8808019bb000 0000000000000213 ffff88065f53a280 ffff8807b995fe70 kernel: [2573885.312109] ffff8808019bb000 ffff8807ae03fc60 0000000000000000 000000000088aaad kernel: [2573885.312217] ffff8807ba7cfe20 ffffffffa0521fb9 0000000000000000 ffff8808000088aa kernel: [2573885.312325] Call Trace: kernel: [2573885.312376] [] eio_post_io_callback+0x729/0x950 [enhanceio] kernel: [2573885.312479] [] process_one_work+0x182/0x450 kernel: [2573885.312567] [] worker_thread+0x121/0x410 kernel: [2573885.312629] [] ? rescuer_thread+0x3e0/0x3e0 kernel: [2573885.312691] [] kthread+0xd2/0xf0 kernel: [2573885.312750] [] ? kthread_create_on_node+0x190/0x190 kernel: [2573885.312816] [] ret_from_fork+0x7c/0xb0 kernel: [2573885.312876] [] ? kthread_create_on_node+0x190/0x190 kernel: [2573885.312940] Code: 48 89 72 38 48 89 42 30 49 89 57 40 49 c7 46 38 00 00 00 00 e9 05 ff ff ff 49 89 4d 38 4d 89 6c 24 18 e9 2a fe ff ff 0f 0b 0f 0b <0f> 0b 0f 0b 66 66 66 66 90 55 48 89 e5 41 57 4

kernel: [2573885.313286] RIP [] eio_md_write+0x24c/0x250 [enhanceio] kernel: [2573885.313353] RSP kernel: [2573885.314363] ---[ end trace 833c0a3658ba1937 ]--- kernel: [2573885.314589] BUG: unable to handle kernel paging request at ffffffffffffffd8 kernel: [2573885.314788] IP: [] kthread_data+0x10/0x20 kernel: [2573885.314935] PGD 1c11067 PUD 1c13067 PMD 0 kernel: [2573885.315192] Oops: 0000 [#2] SMP kernel: [2573885.315372] Modules linked in: ib_srpt ib_cm ib_sa ib_mad ib_core tcm_qla2xxx qla2xxx tcm_loop tcm_fc libfc scsi_transport_fc scsi_tgt iscsi_target_mod target_core_pscsi target_core_file target_c ore_iblock target_core_mod configfs nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache enhanceio_rand(OF) enhanceio_lru(OF) enhanceio_fifo(OF) enhanceio(OF) snd_hda_codec_hdmi eeepc_wmi asus_wmi sparse_keymap kvmamd kvm crct10dif pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd snd_hda_codec_realtek nouveau serio_raw snd_hda_intel edac_core k10temp edac_mce_amd fam15h_power snd_hda_codec snd_hwdep m xm_wmi video ttm joydev drm_kms_helper snd_pcm snd_page_alloc drm snd_timer sp5100_tco i2c_piix4 snd i2c_algo_bit soundcore 8021q garp stp mrp llc mac_hid wmi lp parport raid10 raid456 async_raid6_recov async_memcpy async_pq async _xor async_tx hid_generic usbhid hid xor raid6_pq psmouse raid1 r8169 mii ixgbe raid0 dca mpt2sas ptp multipath ahci pps_core raid_class libahci linear scsi_transport_sas mdio kernel: [2573885.321176] CPU: 2 PID: 976 Comm: kworker/u12:2 Tainted: GF D O 3.13.0-19-generic #40-Ubuntu kernel: [2573885.321176] CPU: 2 PID: 976 Comm: kworker/u12:2 Tainted: GF D O 3.13.0-19-generic #40-Ubuntu kernel: [2573885.321323] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A99FX PRO R2.0, BIOS 2005 09/16/2013 kernel: [2573885.321483] task: ffff8807b6465fc0 ti: ffff8807ba7ce000 task.ti: ffff8807ba7ce000 kernel: [2573885.321623] RIP: 0010:[] [] kthread_data+0x10/0x20 kernel: [2573885.321811] RSP: 0018:ffff8807ba7cfa78 EFLAGS: 00010002 kernel: [2573885.321910] RAX: 0000000000000000 RBX: 0000000000000002 RCX: 0000000000000002 kernel: [2573885.322049] RDX: 0000000000000001 RSI: 0000000000000002 RDI: ffff8807b6465fc0 kernel: [2573885.322188] RBP: ffff8807ba7cfa78 R08: 0000000000000000 R09: 0000000000000001 kernel: [2573885.322351] R10: ffffffff8133443a R11: ffffea001ee73780 R12: ffff88082ec94440 kernel: [2573885.322490] R13: 0000000000000002 R14: ffff8807b6465fb0 R15: ffff8807b6465fc0 kernel: [2573885.322630] FS: 00007f35f4fe9700(0000) GS:ffff88082ec80000(0000) knlGS:0000000000000000 kernel: [2573885.322772] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b kernel: [2573885.322874] CR2: 0000000000000028 CR3: 00000007bd69b000 CR4: 00000000000407e0 kernel: [2573885.323013] Stack: kernel: [2573885.323098] ffff8807ba7cfa90 ffffffff81083951 ffff8807b6465fc0 ffff8807ba7cfaf0 kernel: [2573885.323417] ffffffff81715249 ffff8807b6465fc0 ffff8807ba7cffd8 0000000000014440 kernel: [2573885.323765] 0000000000014440 ffff8807b6465fc0 ffff8807b64665f8 ffff8807b6465fb0 kernel: [2573885.324086] Call Trace: kernel: [2573885.324177] [] wq_worker_sleeping+0x11/0x90 kernel: [2573885.324282] [] __schedule+0x589/0x7d0 kernel: [2573885.324384] [] schedule+0x29/0x70 kernel: [2573885.324485] [] do_exit+0x6df/0xa50 kernel: [2573885.324588] [] oops_end+0xa9/0x150 kernel: [2573885.324690] [] die+0x4b/0x70 kernel: [2573885.324823] [] do_trap+0x60/0x170 kernel: [2573885.324925] [] do_invalid_op+0xa2/0x100 kernel: [2573885.325030] [] ? eio_md_write+0x24c/0x250 [enhanceio] kernel: [2573885.325139] [] ? ttwu_do_activate.constprop.74+0x5d/0x70 kernel: [2573885.325247] [] ? try_to_wake_up+0x1fa/0x2c0 kernel: [2573885.325352] [] ? kfree+0xfd/0x130 kernel: [2573885.325452] [] invalid_op+0x1e/0x30 kernel: [2573885.325554] [] ? mempool_free_slab+0x17/0x20 kernel: [2573885.325661] [] ? eio_md_write+0x24c/0x250 [enhanceio] kernel: [2573885.325771] [] ? eio_md_write+0x1ab/0x250 [enhanceio] kernel: [2573885.325882] [] eio_post_io_callback+0x729/0x950 [enhanceio] kernel: [2573885.326024] [] process_one_work+0x182/0x450 kernel: [2573885.326129] [] worker_thread+0x121/0x410 kernel: [2573885.326232] [] ? rescuer_thread+0x3e0/0x3e0 kernel: [2573885.326341] [] kthread+0xd2/0xf0 kernel: [2573885.326446] [] ? kthread_create_on_node+0x190/0x190 kernel: [2573885.326553] [] ret_from_fork+0x7c/0xb0 kernel: [2573885.326654] [] ? kthread_create_on_node+0x190/0x190 kernel: [2573885.326760] Code: 00 48 89 e5 5d 48 8b 40 c8 48 c1 e8 02 83 e0 01 c3 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 48 8b 87 a8 03 00 00 55 48 89 e5 <48> 8b 40 d8 5d c3 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 kernel: [2573885.329987] RIP [] kthread_data+0x10/0x20 kernel: [2573885.330134] RSP kernel: [2573885.330227] CR2: ffffffffffffffd8 kernel: [2573885.330319] ---[ end trace 833c0a3658ba1938 ]--- kernel: [2573885.330416] Fixing recursive fault but reboot is needed! kernel: [2573905.625082] ------------[ cut here ]------------ kernel: [2573905.625186] WARNING: CPU: 2 PID: 976 at /build/buildd/linux-3.13.0/kernel/watchdog.c:245 watchdog_overflow_callback+0x9c/0xd0() kernel: [2573905.626983] Watchdog detected hard LOCKUP on cpu 2 kernel: [2573905.627048] Modules linked in: ib_srpt ib_cm ib_sa ib_mad ib_core tcm_qla2xxx qla2xxx tcm_loop tcm_fc libfc scsi_transport_fc scsi_tgt iscsi_target_mod target_core_pscsi target_core_file target_core_iblock target_core_mod configfs nfsd auth_rpcgss nfs_acl nfs lockd sunrpc fscache enhanceio_rand(OF) enhanceio_lru(OF) enhanceio_fifo(OF) enhanceio(OF) snd_hda_codec_hdmi eeepc_wmi asus_wmi sparse_keymap kvm_amd kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd snd_hda_codec_realtek nouveau serio_raw snd_hda_intel edac_core k10temp edac_mce_amd fam15h_power snd_hda_codec snd_hwdep mxm_wmi video ttm joydev drm_kms_helper snd_pcm snd_page_alloc drm snd_timer sp5100_tco i2c_piix4 snd i2c_algo_bit soundcore 8021q garp stp mrp llc mac_hid wmi lp parport raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx hid_generic usbhid hid xor raid6_pq psmouse raid1 r8169 mii ixgbe raid0 dca mpt2sas ptp multipath ahci pps_core raid_class libahci linear scsi_transport_sas mdio kernel: [2573905.632902] CPU: 2 PID: 976 Comm: kworker/u12:2 Tainted: GF D O 3.13.0-19-generic #40-Ubuntu kernel: [2573905.633049] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A99FX PRO R2.0, BIOS 2005 09/16/2013 kernel: [2573905.633209] 0000000000000009 ffff88082ec86c38 ffffffff81711075 ffff88082ec86c80 kernel: [2573905.633526] ffff88082ec86c70 ffffffff810662cd ffff880802e68000 0000000000000000 kernel: [2573905.633842] ffff88082ec86d88 0000000000000000 ffff88082ec86ef8 ffff88082ec86cd0 kernel: [2573905.634185] Call Trace: kernel: [2573905.634272] [] dump_stack+0x45/0x56 kernel: [2573905.634467] [] warn_slowpath_common+0x7d/0xa0 kernel: [2573905.634571] [] warn_slowpath_fmt+0x4c/0x50 kernel: [2573905.634674] [] ? restart_watchdog_hrtimer+0x50/0x50 kernel: [2573905.634780] [] watchdog_overflow_callback+0x9c/0xd0 kernel: [2573905.634886] [] __perf_event_overflow+0x8e/0x240 kernel: [2573905.634990] [] perf_event_overflow+0x14/0x20