Soft lockups - Githubissues

I had the message running across my screen yesterday.

kernel: BUG: soft lockup - CPU#0 stuck for 22s! [z_ioctl_int/0:2778]

Logs were full ofkernel messages.

I run Scientific Linux 6.5 with kernel-lt 3.10.58 ZFS version is from git, 0.6.3-139_gbb9d808

Here is the sample syslog entry.

2014-11-04T14:57:15.529841-08:00 logserver1-chi kernel: BUG: soft lockup - CPU#0 stuck for 22s! [z_ioctl_int/0:2778] 2014-11-04T14:57:15.529851-08:00 logserver1-chi kernel: Modules linked in: ipmi_si vfat fat usb_storage mpt3sas mpt2sas scsi_transport_sas raid_class mptctl mptbase ipmi_devintf ipmi_msghandler dell_rbu nfsd nfs_acl auth_rpcgss autofs4 bnx2fc cnic uio fcoe libfcoe libfc scsi_transport_fc scsi_tgt 8021q garp stp llc lockd sunrpc xt_multiport xt_limit xt_LOG nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_netlink nfnetlink nf_conntrack_irc nf_conntrack_pptp nf_conntrack_snmp xt_conntrack nf_conntrack_proto_gre nf_conntrack_sane nf_conntrack_proto_udplite nf_conntrack_proto_sctp nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_sip nf_conntrack_tftp nf_conntrack_proto_dccp ts_kmp nf_conntrack_amanda ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 uinput gpio_ich iTCO_wdt iTCO_vendor_support dcdbas zfs(PO) zcommon(PO) znvpair(PO) zavl(PO) zunicode(PO) spl(O) coretemp intel_powerclamp kvm_intel kvm crc32_pclmul crc32c_intel ghash_clmulni_intel microcode joydev serio_raw pcspkr sg ses enclosure bnx2 lpc_ich i7core_edac edac_core acpi_power_meter hwmon ext4 jbd2 mbcache sd_mod crc_t10dif aesni_intel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 wmi megaraid_sas mgag200 ttm drm_kms_helper sysimgblt sysfillrect syscopyarea dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_si] 2014-11-04T14:57:15.529874-08:00 logserver1-chi kernel: CPU: 0 PID: 2778 Comm: z_ioctl_int/0 Tainted: P IO 3.10.58-1.el6.elrepo.x86_64 #1 2014-11-04T14:57:15.529875-08:00 logserver1-chi kernel: Hardware name: Dell Inc. PowerEdge R510/0DPRKF, BIOS 1.12.0 07/26/2013 2014-11-04T14:57:15.529876-08:00 logserver1-chi kernel: task: ffff8802207fe340 ti: ffff880220006000 task.ti: ffff880220006000 2014-11-04T14:57:15.529877-08:00 logserver1-chi kernel: RIP: 0010:[] [] _raw_spin_lock+0x22/0x30 2014-11-04T14:57:15.529877-08:00 logserver1-chi kernel: RSP: 0018:ffff880220007c68 EFLAGS: 00000293 2014-11-04T14:57:15.529878-08:00 logserver1-chi kernel: RAX: 0000000000000002 RBX: 0000000000000282 RCX: ffff8802744d84d8 2014-11-04T14:57:15.529879-08:00 logserver1-chi kernel: RDX: 0000000000000000 RSI: 0000000000000282 RDI: ffff8802744d84a4 2014-11-04T14:57:15.529879-08:00 logserver1-chi kernel: RBP: ffff880220007c68 R08: ffff88042fc14480 R09: 0000000000000000 2014-11-04T14:57:15.529880-08:00 logserver1-chi kernel: R10: 0000000000000001 R11: 0000000000000000 R12: ffff8802744d84d8 2014-11-04T14:57:15.529881-08:00 logserver1-chi kernel: R13: ffff8802744d84d8 R14: ffff88042fc14480 R15: 0000000000000000 2014-11-04T14:57:15.529882-08:00 logserver1-chi kernel: FS: 0000000000000000(0000) GS:ffff88042fc00000(0000) knlGS:0000000000000000 2014-11-04T14:57:15.529882-08:00 logserver1-chi kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b 2014-11-04T14:57:15.529883-08:00 logserver1-chi kernel: CR2: 00007f46745a8058 CR3: 0000000001a0c000 CR4: 00000000000007f0 2014-11-04T14:57:15.529884-08:00 logserver1-chi kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 2014-11-04T14:57:15.529885-08:00 logserver1-chi kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 2014-11-04T14:57:15.529885-08:00 logserver1-chi kernel: Stack: 2014-11-04T14:57:15.529886-08:00 logserver1-chi kernel: ffff880220007c88 ffffffff815ec3ba ffff880220007c98 ffff8802744d8190 2014-11-04T14:57:15.529887-08:00 logserver1-chi kernel: ffff880220007c98 ffffffff815ec39b ffff880220007d18 ffffffffa044915d 2014-11-04T14:57:15.529887-08:00 logserver1-chi kernel: ffff880225b10a38 ffff880225b10000 ffff8802744d8190 ffff8802b4c4ccf8 2014-11-04T14:57:15.529888-08:00 logserver1-chi kernel: Call Trace: 2014-11-04T14:57:15.529889-08:00 logserver1-chi kernel: [] __mutex_unlock_slowpath+0x1a/0x50 2014-11-04T14:57:15.529889-08:00 logserver1-chi kernel: [] mutex_unlock+0x1b/0x20 2014-11-04T14:57:15.529890-08:00 logserver1-chi kernel: [] zio_done+0x5ed/0xe50 [zfs] 2014-11-04T14:57:15.529891-08:00 logserver1-chi kernel: [] ? zio_remove_child+0xcb/0xf0 [zfs] 2014-11-04T14:57:15.529891-08:00 logserver1-chi kernel: [] zio_done+0x7c7/0xe50 [zfs] 2014-11-04T14:57:15.529892-08:00 logserver1-chi kernel: [] zio_execute+0xb5/0x140 [zfs] 2014-11-04T14:57:15.529893-08:00 logserver1-chi kernel: [] taskq_thread+0x22d/0x4f0 [spl] 2014-11-04T14:57:15.529893-08:00 logserver1-chi kernel: [] ? try_to_wake_up+0x2c0/0x2c0 2014-11-04T14:57:15.529894-08:00 logserver1-chi kernel: [] ? task_expire+0x120/0x120 [spl] 2014-11-04T14:57:15.529895-08:00 logserver1-chi kernel: [] kthread+0xce/0xe0 2014-11-04T14:57:15.529895-08:00 logserver1-chi kernel: [] ? kthread_freezable_should_stop+0x70/0x70 2014-11-04T14:57:15.529896-08:00 logserver1-chi kernel: [] ret_from_fork+0x7c/0xb0 2014-11-04T14:57:15.529897-08:00 logserver1-chi kernel: [] ? kthread_freezable_should_stop+0x70/0x70 2014-11-04T14:57:15.529897-08:00 logserver1-chi kernel: Code: 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 66 66 66 66 90 b8 00 00 01 00 f0 0f c1 07 89 c2 c1 ea 10 66 39 c2 74 0e 0f 1f 40 00 f3 90 <0f> b7 07 66 39 c2 75 f6 c9 c3 0f 1f 40 00 55 48 89 e5 66 66 66

openzfs / zfs

Soft lockups #2870