search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.32k stars 1.72k forks source link

BUG: scheduling while atomic: swapper/4/0/0x10000100 #8137

Closed devZer0 closed 5 years ago

devZer0 commented 5 years ago

Distribution Name | CentOS7 Distribution Version | CentOS Linux release 7.5.1804 (Core) Linux Kernel | 3.10.0-862.14.4.el7.x86_64 Architecture | x86_64 ZFS Version | 0.8.0-rc2 ( zfs-0.8.0-rc2.el7.x86_64 ) SPL Version | 0.8.0-rc2

Describe the problem you're observing

While playing with metadata allocation classes and filling disks/pool with some test data, one of the pool's ordinary disks faulted (using old, flaky ones), leaving kernel BUG/trace below. not sure if this has anything to do with special device/metadata separation... oh, and ignore there is double fault, the second drive faulted while resilvering...

[root@localhost ~]# zpool status -v 1
  pool: zfspool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat Nov 17 16:41:06 2018
    941G scanned at 208M/s, 604G issued at 134M/s, 1,04T total
    97,1G resilvered, 56,69% done, 0 days 00:58:58 to go
config:

    NAME             STATE     READ WRITE CKSUM
    zfspool          DEGRADED     0     0     0
      raidz2-0       DEGRADED     0     0     0
        sda          FAULTED      0     0     0  too many errors  (resilvering)
        sdb          ONLINE       0     0     0
        replacing-2  DEGRADED     0     0     0
          old        OFFLINE      0     4     0
          sdc        ONLINE       0     0     0  (resilvering)
        sdd          ONLINE       0     0     0
        sde          ONLINE       0     0     0
        sdf          ONLINE       0     0     0
    special
      sdg        ONLINE       0     0     0

errors: No known data errors

[17398.865285] sd 0:1:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE
[17398.865303] sd 0:1:2:0: [sdc] tag#0 Sense Key : Hardware Error [current]
[17398.865309] sd 0:1:2:0: [sdc] tag#0 Add. Sense: No additional sense information
[17398.865316] sd 0:1:2:0: [sdc] tag#0 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00
[17398.865326] blk_update_request: I/O error, dev sdc, sector 0
[17398.865377] BUG: scheduling while atomic: swapper/4/0/0x10000100
[17398.865416] Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter zfs(POE) zunicode(POE) zlua(POE) intel_powerclamp coretemp kvm_intel ipmi_ssif kvm irqbypass crc32_pclmul ghash_clmulni_intel iTCO_wdt iTCO_vendor_support aesni_intel lrw gf128mul glue_helper ablk_helper zcommon(POE) cryptd znvpair(POE) zavl(POE) icp(POE) spl(OE) pcspkr joydev i2c_i801 lpc_ich ipmi_si sg ipmi_devintf ipmi_msghandler tpm_infineon
[17398.865478]  acpi_power_meter ioatdma i7core_edac shpchp acpi_cpufreq ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper qla2xxx syscopyarea sysfillrect ahci sysimgblt fb_sys_fops ttm libahci crct10dif_pclmul crct10dif_common drm crc32c_intel libata igb aacraid ptp pps_core dca i2c_algo_bit i2c_core scsi_transport_fc scsi_tgt dm_mirror dm_region_hash dm_log dm_mod
[17398.865514] CPU: 4 PID: 0 Comm: swapper/4 Kdump: loaded Tainted: P          IOE  ------------   3.10.0-862.14.4.el7.x86_64 #1
[17398.865517] Hardware name: FUJITSU                          PRIMERGY RX300 S6             /D2619, BIOS 6.00 Rev. 1.13.2619.N1           01/19/2012
[17398.865520] Call Trace:
[17398.865523]  <IRQ>  [<ffffffff9bb13754>] dump_stack+0x19/0x1b
[17398.865535]  [<ffffffff9bb0d862>] __schedule_bug+0x64/0x72
[17398.865540]  [<ffffffff9bb18eeb>] __schedule+0x9fb/0xa20
[17398.865546]  [<ffffffff9b7c6ffb>] ? fbcon_putcs+0x12b/0x160
[17398.865553]  [<ffffffff9b4ceb76>] __cond_resched+0x26/0x30
[17398.865557]  [<ffffffff9bb191da>] _cond_resched+0x3a/0x50
[17398.865562]  [<ffffffff9b5f92fc>] __kmalloc_node+0x5c/0x2b0
[17398.865575]  [<ffffffffc05644cf>] ? spl_kmem_alloc+0xdf/0x140 [spl]
[17398.865584]  [<ffffffffc05644cf>] spl_kmem_alloc+0xdf/0x140 [spl]
[17398.865656]  [<ffffffffc0cdb629>] __dprintf+0x69/0x150 [zfs]
[17398.865662]  [<ffffffff9b5fa282>] ? kmem_cache_free+0x1e2/0x200
[17398.865720]  [<ffffffffc0cb1abf>] vdev_disk_error.part.15+0x5f/0x70 [zfs]
[17398.865778]  [<ffffffffc0cb1b18>] vdev_disk_io_flush_completion+0x48/0x70 [zfs]
[17398.865783]  [<ffffffff9b65cbd7>] bio_endio+0x67/0xb0
[17398.865789]  [<ffffffff9b71f8e0>] blk_update_request+0x90/0x360
[17398.865793]  [<ffffffff9b71fbcc>] blk_update_bidi_request+0x1c/0x80
[17398.865797]  [<ffffffff9b720797>] __blk_end_bidi_request+0x17/0x40
[17398.865801]  [<ffffffff9b72089f>] __blk_end_request_all+0x1f/0x30
[17398.865805]  [<ffffffff9b722fe5>] blk_flush_complete_seq+0x345/0x360
[17398.865809]  [<ffffffff9b7233e0>] flush_end_io+0x1f0/0x2f0
[17398.865813]  [<ffffffff9b71fe43>] blk_finish_request+0x83/0x130
[17398.865819]  [<ffffffff9b8a99d6>] scsi_end_request+0x116/0x1e0
[17398.865823]  [<ffffffff9b8a9dd7>] scsi_io_completion+0x2d7/0x6a0
[17398.865827]  [<ffffffff9b89f14c>] scsi_finish_command+0xdc/0x140
[17398.865831]  [<ffffffff9b8a91b2>] scsi_softirq_done+0x132/0x160
[17398.865836]  [<ffffffff9b7272a6>] blk_done_softirq+0x96/0xc0
[17398.865841]  [<ffffffff9b49dba5>] __do_softirq+0xf5/0x280
[17398.865846]  [<ffffffff9bb28cec>] call_softirq+0x1c/0x30
[17398.865852]  [<ffffffff9b42e625>] do_softirq+0x65/0xa0
[17398.865855]  [<ffffffff9b49df25>] irq_exit+0x105/0x110
[17398.865859]  [<ffffffff9bb29fa6>] do_IRQ+0x56/0xf0
[17398.865863]  [<ffffffff9bb1c362>] common_interrupt+0x162/0x162
[17398.865865]  <EOI>  [<ffffffff9b96e704>] ? cpuidle_enter_state+0x54/0xd0
[17398.865873]  [<ffffffff9b96e85e>] cpuidle_idle_call+0xde/0x230
[17398.865877]  [<ffffffff9b4366ce>] arch_cpu_idle+0xe/0xb0
[17398.865882]  [<ffffffff9b4f5dea>] cpu_startup_entry+0x14a/0x1e0
[17398.865888]  [<ffffffff9b4571b7>] start_secondary+0x1f7/0x270
[17398.865893]  [<ffffffff9b4000d5>] start_cpu+0x5/0x14
[17399.072950] sd 0:1:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
[17399.072970] sd 0:1:2:0: [sdc] tag#1 CDB: Write(16) 8a 00 00 00 00 01 5c bf 67 7c 00 00 00 02 00 00
[17399.072978] blk_update_request: I/O error, dev sdc, sector 5851015036
[17399.073063] BUG: scheduling while atomic: swapper/4/0/0x10000100
[17399.073131] Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter zfs(POE) zunicode(POE) zlua(POE) intel_powerclamp coretemp kvm_intel ipmi_ssif kvm irqbypass crc32_pclmul ghash_clmulni_intel iTCO_wdt iTCO_vendor_support aesni_intel lrw gf128mul glue_helper ablk_helper zcommon(POE) cryptd znvpair(POE) zavl(POE) icp(POE) spl(OE) pcspkr joydev i2c_i801 lpc_ich ipmi_si sg ipmi_devintf ipmi_msghandler tpm_infineon
[17399.073224]  acpi_power_meter ioatdma i7core_edac shpchp acpi_cpufreq ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper qla2xxx syscopyarea sysfillrect ahci sysimgblt fb_sys_fops ttm libahci crct10dif_pclmul crct10dif_common drm crc32c_intel libata igb aacraid ptp pps_core dca i2c_algo_bit i2c_core scsi_transport_fc scsi_tgt dm_mirror dm_region_hash dm_log dm_mod
[17399.073278] CPU: 4 PID: 0 Comm: swapper/4 Kdump: loaded Tainted: P        W IOE  ------------   3.10.0-862.14.4.el7.x86_64 #1
[17399.073284] Hardware name: FUJITSU                          PRIMERGY RX300 S6             /D2619, BIOS 6.00 Rev. 1.13.2619.N1           01/19/2012
[17399.073289] Call Trace:
[17399.073295]  <IRQ>  [<ffffffff9bb13754>] dump_stack+0x19/0x1b
[17399.073312]  [<ffffffff9bb0d862>] __schedule_bug+0x64/0x72
[17399.073321]  [<ffffffff9bb18eeb>] __schedule+0x9fb/0xa20
[17399.073331]  [<ffffffff9b49defa>] ? irq_exit+0xda/0x110
[17399.073342]  [<ffffffff9b4ceb76>] __cond_resched+0x26/0x30
[17399.073351]  [<ffffffff9bb191da>] _cond_resched+0x3a/0x50
[17399.073359]  [<ffffffff9b5f92fc>] __kmalloc_node+0x5c/0x2b0
[17399.073379]  [<ffffffffc05644cf>] ? spl_kmem_alloc+0xdf/0x140 [spl]
[17399.073397]  [<ffffffffc05644cf>] spl_kmem_alloc+0xdf/0x140 [spl]
[17399.073496]  [<ffffffffc0cdb629>] __dprintf+0x69/0x150 [zfs]
[17399.073508]  [<ffffffff9b5fa282>] ? kmem_cache_free+0x1e2/0x200
[17399.073524]  [<ffffffffc05643e5>] ? spl_kmem_free+0x35/0x40 [spl]
[17399.073613]  [<ffffffffc0cb1abf>] vdev_disk_error.part.15+0x5f/0x70 [zfs]
[17399.073702]  [<ffffffffc0cb1ba0>] vdev_disk_dio_put+0x60/0x80 [zfs]
[17399.073792]  [<ffffffffc0cb1c05>] vdev_disk_physio_completion+0x45/0x50 [zfs]
[17399.073802]  [<ffffffff9b65cbd7>] bio_endio+0x67/0xb0
[17399.073813]  [<ffffffff9b71f8e0>] blk_update_request+0x90/0x360
[17399.073823]  [<ffffffff9b8a98f4>] scsi_end_request+0x34/0x1e0
[17399.073832]  [<ffffffff9b8a9dd7>] scsi_io_completion+0x2d7/0x6a0
[17399.073839]  [<ffffffff9b89f14c>] scsi_finish_command+0xdc/0x140
[17399.073846]  [<ffffffff9b8a91b2>] scsi_softirq_done+0x132/0x160
[17399.073855]  [<ffffffff9b7272a6>] blk_done_softirq+0x96/0xc0
[17399.073862]  [<ffffffff9b49dba5>] __do_softirq+0xf5/0x280
[17399.073870]  [<ffffffff9bb28cec>] call_softirq+0x1c/0x30
[17399.073879]  [<ffffffff9b42e625>] do_softirq+0x65/0xa0
[17399.073885]  [<ffffffff9b49df25>] irq_exit+0x105/0x110
[17399.073893]  [<ffffffff9bb29fa6>] do_IRQ+0x56/0xf0
[17399.073900]  [<ffffffff9bb1c362>] common_interrupt+0x162/0x162
[17399.073904]  <EOI>  [<ffffffff9b96e704>] ? cpuidle_enter_state+0x54/0xd0
[17399.073917]  [<ffffffff9b96e85e>] cpuidle_idle_call+0xde/0x230
[17399.073925]  [<ffffffff9b4366ce>] arch_cpu_idle+0xe/0xb0
[17399.073933]  [<ffffffff9b4f5dea>] cpu_startup_entry+0x14a/0x1e0
[17399.073942]  [<ffffffff9b4571b7>] start_secondary+0x1f7/0x270
[17399.073950]  [<ffffffff9b4000d5>] start_cpu+0x5/0x14
[17399.077515] sd 0:1:2:0: [sdc] tag#7 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE
[17399.077533] sd 0:1:2:0: [sdc] tag#7 Sense Key : Hardware Error [current]
[17399.077542] sd 0:1:2:0: [sdc] tag#7 Add. Sense: No additional sense information
[17399.077551] sd 0:1:2:0: [sdc] tag#7 CDB: Write(16) 8a 00 00 00 00 00 00 00 09 7c 00 00 00 02 00 00
[17399.077557] blk_update_request: I/O error, dev sdc, sector 2428
[17399.077633] BUG: scheduling while atomic: swapper/4/0/0x10000100
[17399.077702] Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter zfs(POE) zunicode(POE) zlua(POE) intel_powerclamp coretemp kvm_intel ipmi_ssif kvm irqbypass crc32_pclmul ghash_clmulni_intel iTCO_wdt iTCO_vendor_support aesni_intel lrw gf128mul glue_helper ablk_helper zcommon(POE) cryptd znvpair(POE) zavl(POE) icp(POE) spl(OE) pcspkr joydev i2c_i801 lpc_ich ipmi_si sg ipmi_devintf ipmi_msghandler tpm_infineon
[17399.077783]  acpi_power_meter ioatdma i7core_edac shpchp acpi_cpufreq ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper qla2xxx syscopyarea sysfillrect ahci sysimgblt fb_sys_fops ttm libahci crct10dif_pclmul crct10dif_common drm crc32c_intel libata igb aacraid ptp pps_core dca i2c_algo_bit i2c_core scsi_transport_fc scsi_tgt dm_mirror dm_region_hash dm_log dm_mod
[17399.077833] CPU: 4 PID: 0 Comm: swapper/4 Kdump: loaded Tainted: P        W IOE  ------------   3.10.0-862.14.4.el7.x86_64 #1
[17399.077839] Hardware name: FUJITSU                          PRIMERGY RX300 S6             /D2619, BIOS 6.00 Rev. 1.13.2619.N1           01/19/2012
[17399.077843] Call Trace:
[17399.077849]  <IRQ>  [<ffffffff9bb13754>] dump_stack+0x19/0x1b
[17399.077860]  [<ffffffff9bb0d862>] __schedule_bug+0x64/0x72
[17399.077868]  [<ffffffff9bb18eeb>] __schedule+0x9fb/0xa20
[17399.077876]  [<ffffffff9b49defa>] ? irq_exit+0xda/0x110
[17399.077884]  [<ffffffff9b4ceb76>] __cond_resched+0x26/0x30
[17399.077892]  [<ffffffff9bb191da>] _cond_resched+0x3a/0x50
[17399.077899]  [<ffffffff9b5f92fc>] __kmalloc_node+0x5c/0x2b0
[17399.077916]  [<ffffffffc05644cf>] ? spl_kmem_alloc+0xdf/0x140 [spl]
[17399.077931]  [<ffffffffc05644cf>] spl_kmem_alloc+0xdf/0x140 [spl]
[17399.078022]  [<ffffffffc0cdb629>] __dprintf+0x69/0x150 [zfs]
[17399.078035]  [<ffffffff9b5fa282>] ? kmem_cache_free+0x1e2/0x200
[17399.078051]  [<ffffffffc05643e5>] ? spl_kmem_free+0x35/0x40 [spl]
[17399.078138]  [<ffffffffc0cb1abf>] vdev_disk_error.part.15+0x5f/0x70 [zfs]
[17399.078224]  [<ffffffffc0cb1ba0>] vdev_disk_dio_put+0x60/0x80 [zfs]
[17399.078315]  [<ffffffffc0cb1c05>] vdev_disk_physio_completion+0x45/0x50 [zfs]
[17399.078324]  [<ffffffff9b65cbd7>] bio_endio+0x67/0xb0
[17399.078332]  [<ffffffff9b71f8e0>] blk_update_request+0x90/0x360
[17399.078340]  [<ffffffff9b8a98f4>] scsi_end_request+0x34/0x1e0
[17399.078348]  [<ffffffff9b8a9dd7>] scsi_io_completion+0x2d7/0x6a0
[17399.078355]  [<ffffffff9b89f14c>] scsi_finish_command+0xdc/0x140
[17399.078363]  [<ffffffff9b8a91b2>] scsi_softirq_done+0x132/0x160
[17399.078371]  [<ffffffff9b7272a6>] blk_done_softirq+0x96/0xc0
[17399.078379]  [<ffffffff9b49dba5>] __do_softirq+0xf5/0x280
[17399.078386]  [<ffffffff9bb28cec>] call_softirq+0x1c/0x30
[17399.078393]  [<ffffffff9b42e625>] do_softirq+0x65/0xa0
[17399.078399]  [<ffffffff9b49df25>] irq_exit+0x105/0x110
[17399.078406]  [<ffffffff9bb29fa6>] do_IRQ+0x56/0xf0
[17399.078417]  [<ffffffff9bb1c362>] common_interrupt+0x162/0x162
[17399.078421]  <EOI>  [<ffffffff9b96e704>] ? cpuidle_enter_state+0x54/0xd0
[17399.078432]  [<ffffffff9b96e85e>] cpuidle_idle_call+0xde/0x230
[17399.078439]  [<ffffffff9b4366ce>] arch_cpu_idle+0xe/0xb0
[17399.078446]  [<ffffffff9b4f5dea>] cpu_startup_entry+0x14a/0x1e0
[17399.078454]  [<ffffffff9b4571b7>] start_secondary+0x1f7/0x270
[17399.078460]  [<ffffffff9b4000d5>] start_cpu+0x5/0x14
[17399.217339] sd 0:1:2:0: [sdc] tag#9 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
[17399.217361] sd 0:1:2:0: [sdc] tag#9 CDB: Write(16) 8a 00 00 00 00 01 5c bf 65 7c 00 00 00 02 00 00
[17399.217369] blk_update_request: I/O error, dev sdc, sector 5851014524
[17399.217453] BUG: scheduling while atomic: swapper/4/0/0x10000100
[17399.217513] Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter zfs(POE) zunicode(POE) zlua(POE) intel_powerclamp coretemp kvm_intel ipmi_ssif kvm irqbypass crc32_pclmul ghash_clmulni_intel iTCO_wdt iTCO_vendor_support aesni_intel lrw gf128mul glue_helper ablk_helper zcommon(POE) cryptd znvpair(POE) zavl(POE) icp(POE) spl(OE) pcspkr joydev i2c_i801 lpc_ich ipmi_si sg ipmi_devintf ipmi_msghandler tpm_infineon
[17399.217602]  acpi_power_meter ioatdma i7core_edac shpchp acpi_cpufreq ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper qla2xxx syscopyarea sysfillrect ahci sysimgblt fb_sys_fops ttm libahci crct10dif_pclmul crct10dif_common drm crc32c_intel libata igb aacraid ptp pps_core dca i2c_algo_bit i2c_core scsi_transport_fc scsi_tgt dm_mirror dm_region_hash dm_log dm_mod
[17399.217652] CPU: 4 PID: 0 Comm: swapper/4 Kdump: loaded Tainted: P        W IOE  ------------   3.10.0-862.14.4.el7.x86_64 #1
[17399.217658] Hardware name: FUJITSU                          PRIMERGY RX300 S6             /D2619, BIOS 6.00 Rev. 1.13.2619.N1           01/19/2012
[17399.217662] Call Trace:
[17399.217669]  <IRQ>  [<ffffffff9bb13754>] dump_stack+0x19/0x1b
[17399.217686]  [<ffffffff9bb0d862>] __schedule_bug+0x64/0x72
[17399.217695]  [<ffffffff9bb18eeb>] __schedule+0x9fb/0xa20
[17399.217705]  [<ffffffff9b49defa>] ? irq_exit+0xda/0x110
[17399.217714]  [<ffffffff9b4ceb76>] __cond_resched+0x26/0x30
[17399.217722]  [<ffffffff9bb191da>] _cond_resched+0x3a/0x50
[17399.217731]  [<ffffffff9b5f92fc>] __kmalloc_node+0x5c/0x2b0
[17399.217753]  [<ffffffffc05644cf>] ? spl_kmem_alloc+0xdf/0x140 [spl]
[17399.217767]  [<ffffffffc05644cf>] spl_kmem_alloc+0xdf/0x140 [spl]
[17399.217867]  [<ffffffffc0cdb629>] __dprintf+0x69/0x150 [zfs]
[17399.217879]  [<ffffffff9b5fa282>] ? kmem_cache_free+0x1e2/0x200
[17399.217895]  [<ffffffffc05643e5>] ? spl_kmem_free+0x35/0x40 [spl]
[17399.217983]  [<ffffffffc0cb1abf>] vdev_disk_error.part.15+0x5f/0x70 [zfs]
[17399.218076]  [<ffffffffc0cb1ba0>] vdev_disk_dio_put+0x60/0x80 [zfs]
[17399.218165]  [<ffffffffc0cb1c05>] vdev_disk_physio_completion+0x45/0x50 [zfs]
[17399.218176]  [<ffffffff9b65cbd7>] bio_endio+0x67/0xb0
[17399.218185]  [<ffffffff9b71f8e0>] blk_update_request+0x90/0x360
[17399.218195]  [<ffffffff9b8a98f4>] scsi_end_request+0x34/0x1e0
[17399.218202]  [<ffffffff9b8a9dd7>] scsi_io_completion+0x2d7/0x6a0
[17399.218210]  [<ffffffff9b89f14c>] scsi_finish_command+0xdc/0x140
[17399.218217]  [<ffffffff9b8a91b2>] scsi_softirq_done+0x132/0x160
[17399.218225]  [<ffffffff9b7272a6>] blk_done_softirq+0x96/0xc0
[17399.218232]  [<ffffffff9b49dba5>] __do_softirq+0xf5/0x280
[17399.218241]  [<ffffffff9bb28cec>] call_softirq+0x1c/0x30
[17399.218249]  [<ffffffff9b42e625>] do_softirq+0x65/0xa0
[17399.218256]  [<ffffffff9b49df25>] irq_exit+0x105/0x110
[17399.218264]  [<ffffffff9bb29fa6>] do_IRQ+0x56/0xf0
[17399.218271]  [<ffffffff9bb1c362>] common_interrupt+0x162/0x162
[17399.218276]  <EOI>  [<ffffffff9b96e704>] ? cpuidle_enter_state+0x54/0xd0
[17399.218288]  [<ffffffff9b96e85e>] cpuidle_idle_call+0xde/0x230
[17399.218296]  [<ffffffff9b4366ce>] arch_cpu_idle+0xe/0xb0
[17399.218303]  [<ffffffff9b4f5dea>] cpu_startup_entry+0x14a/0x1e0
[17399.218312]  [<ffffffff9b4571b7>] start_secondary+0x1f7/0x270
[17399.218321]  [<ffffffff9b4000d5>] start_cpu+0x5/0x14
[17399.221823] sd 0:1:2:0: [sdc] tag#8 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE
[17399.221838] sd 0:1:2:0: [sdc] tag#8 Sense Key : Hardware Error [current]
[17399.221846] sd 0:1:2:0: [sdc] tag#8 Add. Sense: No additional sense information
[17399.221854] sd 0:1:2:0: [sdc] tag#8 CDB: Write(16) 8a 00 00 00 00 00 00 00 0b 7c 00 00 00 02 00 00
[17399.221860] blk_update_request: I/O error, dev sdc, sector 2940
[17399.221936] BUG: scheduling while atomic: swapper/4/0/0x10000100
[17399.222007] Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter zfs(POE) zunicode(POE) zlua(POE) intel_powerclamp coretemp kvm_intel ipmi_ssif kvm irqbypass crc32_pclmul ghash_clmulni_intel iTCO_wdt iTCO_vendor_support aesni_intel lrw gf128mul glue_helper ablk_helper zcommon(POE) cryptd znvpair(POE) zavl(POE) icp(POE) spl(OE) pcspkr joydev i2c_i801 lpc_ich ipmi_si sg ipmi_devintf ipmi_msghandler tpm_infineon
[17399.222086]  acpi_power_meter ioatdma i7core_edac shpchp acpi_cpufreq ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper qla2xxx syscopyarea sysfillrect ahci sysimgblt fb_sys_fops ttm libahci crct10dif_pclmul crct10dif_common drm crc32c_intel libata igb aacraid ptp pps_core dca i2c_algo_bit i2c_core scsi_transport_fc scsi_tgt dm_mirror dm_region_hash dm_log dm_mod
[17399.222134] CPU: 4 PID: 0 Comm: swapper/4 Kdump: loaded Tainted: P        W IOE  ------------   3.10.0-862.14.4.el7.x86_64 #1
[17399.222140] Hardware name: FUJITSU                          PRIMERGY RX300 S6             /D2619, BIOS 6.00 Rev. 1.13.2619.N1           01/19/2012
[17399.222144] Call Trace:
[17399.222150]  <IRQ>  [<ffffffff9bb13754>] dump_stack+0x19/0x1b
[17399.222162]  [<ffffffff9bb0d862>] __schedule_bug+0x64/0x72
[17399.222170]  [<ffffffff9bb18eeb>] __schedule+0x9fb/0xa20
[17399.222177]  [<ffffffff9b49defa>] ? irq_exit+0xda/0x110
[17399.222186]  [<ffffffff9b4ceb76>] __cond_resched+0x26/0x30
[17399.222194]  [<ffffffff9bb191da>] _cond_resched+0x3a/0x50
[17399.222202]  [<ffffffff9b5f92fc>] __kmalloc_node+0x5c/0x2b0
[17399.222220]  [<ffffffffc05644cf>] ? spl_kmem_alloc+0xdf/0x140 [spl]
[17399.222235]  [<ffffffffc05644cf>] spl_kmem_alloc+0xdf/0x140 [spl]
[17399.222329]  [<ffffffffc0cdb629>] __dprintf+0x69/0x150 [zfs]
[17399.222338]  [<ffffffff9b5fa282>] ? kmem_cache_free+0x1e2/0x200
[17399.222355]  [<ffffffffc05643e5>] ? spl_kmem_free+0x35/0x40 [spl]
[17399.222445]  [<ffffffffc0cb1abf>] vdev_disk_error.part.15+0x5f/0x70 [zfs]
[17399.222534]  [<ffffffffc0cb1ba0>] vdev_disk_dio_put+0x60/0x80 [zfs]
[17399.222626]  [<ffffffffc0cb1c05>] vdev_disk_physio_completion+0x45/0x50 [zfs]
[17399.222635]  [<ffffffff9b65cbd7>] bio_endio+0x67/0xb0
[17399.222642]  [<ffffffff9b71f8e0>] blk_update_request+0x90/0x360
[17399.222650]  [<ffffffff9b8a98f4>] scsi_end_request+0x34/0x1e0
[17399.222658]  [<ffffffff9b8a9dd7>] scsi_io_completion+0x2d7/0x6a0
[17399.222665]  [<ffffffff9b89f14c>] scsi_finish_command+0xdc/0x140
[17399.222673]  [<ffffffff9b8a91b2>] scsi_softirq_done+0x132/0x160
[17399.222680]  [<ffffffff9b7272a6>] blk_done_softirq+0x96/0xc0
[17399.222687]  [<ffffffff9b49dba5>] __do_softirq+0xf5/0x280
[17399.222695]  [<ffffffff9bb28cec>] call_softirq+0x1c/0x30
[17399.222702]  [<ffffffff9b42e625>] do_softirq+0x65/0xa0
[17399.222708]  [<ffffffff9b49df25>] irq_exit+0x105/0x110
[17399.222715]  [<ffffffff9bb29fa6>] do_IRQ+0x56/0xf0
[17399.222721]  [<ffffffff9bb1c362>] common_interrupt+0x162/0x162
[17399.222725]  <EOI>  [<ffffffff9b96e704>] ? cpuidle_enter_state+0x54/0xd0
[17399.222735]  [<ffffffff9b96e85e>] cpuidle_idle_call+0xde/0x230
[17399.222740]  [<ffffffff9b4366ce>] arch_cpu_idle+0xe/0xb0
[17399.222744]  [<ffffffff9b4f5dea>] cpu_startup_entry+0x14a/0x1e0
[17399.222750]  [<ffffffff9b4571b7>] start_secondary+0x1f7/0x270
[17399.222755]  [<ffffffff9b4000d5>] start_cpu+0x5/0x14
[17399.439569] sd 0:1:2:0: [sdc] tag#2 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
[17399.439579] sd 0:1:2:0: [sdc] tag#2 CDB: Read(16) 88 00 00 00 00 01 5c bf 66 10 00 00 00 10 00 00
[17399.439583] blk_update_request: I/O error, dev sdc, sector 5851014672
[17399.439638] BUG: scheduling while atomic: swapper/4/0/0x10000100
[17399.439677] Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter zfs(POE) zunicode(POE) zlua(POE) intel_powerclamp coretemp kvm_intel ipmi_ssif kvm irqbypass crc32_pclmul ghash_clmulni_intel iTCO_wdt iTCO_vendor_support aesni_intel lrw gf128mul glue_helper ablk_helper zcommon(POE) cryptd znvpair(POE) zavl(POE) icp(POE) spl(OE) pcspkr joydev i2c_i801 lpc_ich ipmi_si sg ipmi_devintf ipmi_msghandler tpm_infineon
[17399.439741]  acpi_power_meter ioatdma i7core_edac shpchp acpi_cpufreq ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper qla2xxx syscopyarea sysfillrect ahci sysimgblt fb_sys_fops ttm libahci crct10dif_pclmul crct10dif_common drm crc32c_intel libata igb aacraid ptp pps_core dca i2c_algo_bit i2c_core scsi_transport_fc scsi_tgt dm_mirror dm_region_hash dm_log dm_mod
[17399.439777] CPU: 4 PID: 0 Comm: swapper/4 Kdump: loaded Tainted: P        W IOE  ------------   3.10.0-862.14.4.el7.x86_64 #1
[17399.439779] Hardware name: FUJITSU                          PRIMERGY RX300 S6             /D2619, BIOS 6.00 Rev. 1.13.2619.N1           01/19/2012
[17399.439782] Call Trace:
[17399.439786]  <IRQ>  [<ffffffff9bb13754>] dump_stack+0x19/0x1b
[17399.439798]  [<ffffffff9bb0d862>] __schedule_bug+0x64/0x72
[17399.439804]  [<ffffffff9bb18eeb>] __schedule+0x9fb/0xa20
[17399.439811]  [<ffffffff9b4ceb76>] __cond_resched+0x26/0x30
[17399.439815]  [<ffffffff9bb191da>] _cond_resched+0x3a/0x50
[17399.439820]  [<ffffffff9b5f92fc>] __kmalloc_node+0x5c/0x2b0
[17399.439833]  [<ffffffffc05644cf>] ? spl_kmem_alloc+0xdf/0x140 [spl]
[17399.439842]  [<ffffffffc05644cf>] spl_kmem_alloc+0xdf/0x140 [spl]
[17399.439914]  [<ffffffffc0cdb629>] __dprintf+0x69/0x150 [zfs]
[17399.439920]  [<ffffffff9b5fa282>] ? kmem_cache_free+0x1e2/0x200
[17399.439929]  [<ffffffffc05643e5>] ? spl_kmem_free+0x35/0x40 [spl]
[17399.439987]  [<ffffffffc0cb1abf>] vdev_disk_error.part.15+0x5f/0x70 [zfs]
[17399.440046]  [<ffffffffc0cb1ba0>] vdev_disk_dio_put+0x60/0x80 [zfs]
[17399.440103]  [<ffffffffc0cb1c05>] vdev_disk_physio_completion+0x45/0x50 [zfs]
[17399.440108]  [<ffffffff9b65cbd7>] bio_endio+0x67/0xb0
[17399.440113]  [<ffffffff9b71f8e0>] blk_update_request+0x90/0x360
[17399.440120]  [<ffffffff9b8a98f4>] scsi_end_request+0x34/0x1e0
[17399.440124]  [<ffffffff9b8a9dd7>] scsi_io_completion+0x2d7/0x6a0
[17399.440128]  [<ffffffff9b89f14c>] scsi_finish_command+0xdc/0x140
[17399.440132]  [<ffffffff9b8a91b2>] scsi_softirq_done+0x132/0x160
[17399.440137]  [<ffffffff9b7272a6>] blk_done_softirq+0x96/0xc0
[17399.440142]  [<ffffffff9b49dba5>] __do_softirq+0xf5/0x280
[17399.440147]  [<ffffffff9bb28cec>] call_softirq+0x1c/0x30
[17399.440153]  [<ffffffff9b42e625>] do_softirq+0x65/0xa0
[17399.440156]  [<ffffffff9b49df25>] irq_exit+0x105/0x110
[17399.440160]  [<ffffffff9bb29fa6>] do_IRQ+0x56/0xf0
[17399.440164]  [<ffffffff9bb1c362>] common_interrupt+0x162/0x162
[17399.440166]  <EOI>  [<ffffffff9b96e704>] ? cpuidle_enter_state+0x54/0xd0
[17399.440174]  [<ffffffff9b96e85e>] cpuidle_idle_call+0xde/0x230
[17399.440178]  [<ffffffff9b4366ce>] arch_cpu_idle+0xe/0xb0
[17399.440183]  [<ffffffff9b4f5dea>] cpu_startup_entry+0x14a/0x1e0
[17399.440189]  [<ffffffff9b4571b7>] start_secondary+0x1f7/0x270
[17399.440194]  [<ffffffff9b4000d5>] start_cpu+0x5/0x14
[17399.784424] sd 0:1:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
[17399.784442] sd 0:1:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 00 00 0a 10 00 00 00 10 00 00
[17399.784448] blk_update_request: I/O error, dev sdc, sector 2576
[17399.784500] BUG: scheduling while atomic: swapper/4/0/0x10000100
[17399.784539] Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter zfs(POE) zunicode(POE) zlua(POE) intel_powerclamp coretemp kvm_intel ipmi_ssif kvm irqbypass crc32_pclmul ghash_clmulni_intel iTCO_wdt iTCO_vendor_support aesni_intel lrw gf128mul glue_helper ablk_helper zcommon(POE) cryptd znvpair(POE) zavl(POE) icp(POE) spl(OE) pcspkr joydev i2c_i801 lpc_ich ipmi_si sg ipmi_devintf ipmi_msghandler tpm_infineon
[17399.784603]  acpi_power_meter ioatdma i7core_edac shpchp acpi_cpufreq ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper qla2xxx syscopyarea sysfillrect ahci sysimgblt fb_sys_fops ttm libahci crct10dif_pclmul crct10dif_common drm crc32c_intel libata igb aacraid ptp pps_core dca i2c_algo_bit i2c_core scsi_transport_fc scsi_tgt dm_mirror dm_region_hash dm_log dm_mod
[17399.784639] CPU: 4 PID: 0 Comm: swapper/4 Kdump: loaded Tainted: P        W IOE  ------------   3.10.0-862.14.4.el7.x86_64 #1
[17399.784643] Hardware name: FUJITSU                          PRIMERGY RX300 S6             /D2619, BIOS 6.00 Rev. 1.13.2619.N1           01/19/2012
[17399.784647] Call Trace:
[17399.784652]  <IRQ>  [<ffffffff9bb13754>] dump_stack+0x19/0x1b
[17399.784665]  [<ffffffff9bb0d862>] __schedule_bug+0x64/0x72
[17399.784671]  [<ffffffff9bb18eeb>] __schedule+0x9fb/0xa20
[17399.784678]  [<ffffffff9b49defa>] ? irq_exit+0xda/0x110
[17399.784686]  [<ffffffff9b4ceb76>] __cond_resched+0x26/0x30
[17399.784691]  [<ffffffff9bb191da>] _cond_resched+0x3a/0x50
[17399.784697]  [<ffffffff9b5f92fc>] __kmalloc_node+0x5c/0x2b0
[17399.784711]  [<ffffffffc05644cf>] ? spl_kmem_alloc+0xdf/0x140 [spl]
[17399.784722]  [<ffffffffc05644cf>] spl_kmem_alloc+0xdf/0x140 [spl]
[17399.784795]  [<ffffffffc0cdb629>] __dprintf+0x69/0x150 [zfs]
[17399.784802]  [<ffffffff9b5fa282>] ? kmem_cache_free+0x1e2/0x200
[17399.784813]  [<ffffffffc05643e5>] ? spl_kmem_free+0x35/0x40 [spl]
[17399.784872]  [<ffffffffc0cb1abf>] vdev_disk_error.part.15+0x5f/0x70 [zfs]
[17399.784932]  [<ffffffffc0cb1ba0>] vdev_disk_dio_put+0x60/0x80 [zfs]
[17399.784991]  [<ffffffffc0cb1c05>] vdev_disk_physio_completion+0x45/0x50 [zfs]
[17399.784998]  [<ffffffff9b65cbd7>] bio_endio+0x67/0xb0
[17399.785005]  [<ffffffff9b71f8e0>] blk_update_request+0x90/0x360
[17399.785012]  [<ffffffff9b8a98f4>] scsi_end_request+0x34/0x1e0
[17399.785018]  [<ffffffff9b8a9dd7>] scsi_io_completion+0x2d7/0x6a0
[17399.785023]  [<ffffffff9b89f14c>] scsi_finish_command+0xdc/0x140
[17399.785028]  [<ffffffff9b8a91b2>] scsi_softirq_done+0x132/0x160
[17399.785034]  [<ffffffff9b7272a6>] blk_done_softirq+0x96/0xc0
[17399.785039]  [<ffffffff9b49dba5>] __do_softirq+0xf5/0x280
[17399.785045]  [<ffffffff9bb28cec>] call_softirq+0x1c/0x30
[17399.785052]  [<ffffffff9b42e625>] do_softirq+0x65/0xa0
[17399.785056]  [<ffffffff9b49df25>] irq_exit+0x105/0x110
[17399.785062]  [<ffffffff9bb29fa6>] do_IRQ+0x56/0xf0
[17399.785067]  [<ffffffff9bb1c362>] common_interrupt+0x162/0x162
[17399.785070]  <EOI>  [<ffffffff9b96e704>] ? cpuidle_enter_state+0x54/0xd0
[17399.785078]  [<ffffffff9b96e85e>] cpuidle_idle_call+0xde/0x230
[17399.785084]  [<ffffffff9b4366ce>] arch_cpu_idle+0xe/0xb0
[17399.785090]  [<ffffffff9b4f5dea>] cpu_startup_entry+0x14a/0x1e0
[17399.785097]  [<ffffffff9b4571b7>] start_secondary+0x1f7/0x270
[17399.785103]  [<ffffffff9b4000d5>] start_cpu+0x5/0x14
[17400.594446] sd 0:1:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_SENSE
[17400.594469] sd 0:1:2:0: [sdc] tag#0 Sense Key : Hardware Error [current]
[17400.594478] sd 0:1:2:0: [sdc] tag#0 Add. Sense: No additional sense information
[17400.594488] sd 0:1:2:0: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 01 5c bf 64 10 00 00 00 10 00 00
[17400.594495] blk_update_request: I/O error, dev sdc, sector 5851014160
[17400.594576] BUG: scheduling while atomic: swapper/4/0/0x10000100
[17400.594643] Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter zfs(POE) zunicode(POE) zlua(POE) intel_powerclamp coretemp kvm_intel ipmi_ssif kvm irqbypass crc32_pclmul ghash_clmulni_intel iTCO_wdt iTCO_vendor_support aesni_intel lrw gf128mul glue_helper ablk_helper zcommon(POE) cryptd znvpair(POE) zavl(POE) icp(POE) spl(OE) pcspkr joydev i2c_i801 lpc_ich ipmi_si sg ipmi_devintf ipmi_msghandler tpm_infineon
[17400.594730]  acpi_power_meter ioatdma i7core_edac shpchp acpi_cpufreq ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper qla2xxx syscopyarea sysfillrect ahci sysimgblt fb_sys_fops ttm libahci crct10dif_pclmul crct10dif_common drm crc32c_intel libata igb aacraid ptp pps_core dca i2c_algo_bit i2c_core scsi_transport_fc scsi_tgt dm_mirror dm_region_hash dm_log dm_mod
[17400.594782] CPU: 4 PID: 0 Comm: swapper/4 Kdump: loaded Tainted: P        W IOE  ------------   3.10.0-862.14.4.el7.x86_64 #1
[17400.594789] Hardware name: FUJITSU                          PRIMERGY RX300 S6             /D2619, BIOS 6.00 Rev. 1.13.2619.N1           01/19/2012
[17400.594793] Call Trace:
[17400.594800]  <IRQ>  [<ffffffff9bb13754>] dump_stack+0x19/0x1b
[17400.594817]  [<ffffffff9bb0d862>] __schedule_bug+0x64/0x72
[17400.594825]  [<ffffffff9bb18eeb>] __schedule+0x9fb/0xa20
[17400.594835]  [<ffffffff9b49defa>] ? irq_exit+0xda/0x110
[17400.594845]  [<ffffffff9b4ceb76>] __cond_resched+0x26/0x30
[17400.594852]  [<ffffffff9bb191da>] _cond_resched+0x3a/0x50
[17400.594861]  [<ffffffff9b5f92fc>] __kmalloc_node+0x5c/0x2b0
[17400.594881]  [<ffffffffc05644cf>] ? spl_kmem_alloc+0xdf/0x140 [spl]
[17400.594898]  [<ffffffffc05644cf>] spl_kmem_alloc+0xdf/0x140 [spl]
[17400.594999]  [<ffffffffc0cdb629>] __dprintf+0x69/0x150 [zfs]
[17400.595010]  [<ffffffff9b5fa282>] ? kmem_cache_free+0x1e2/0x200
[17400.595025]  [<ffffffffc05643e5>] ? spl_kmem_free+0x35/0x40 [spl]
[17400.595108]  [<ffffffffc0cb1abf>] vdev_disk_error.part.15+0x5f/0x70 [zfs]
[17400.595195]  [<ffffffffc0cb1ba0>] vdev_disk_dio_put+0x60/0x80 [zfs]
[17400.595272]  [<ffffffffc0cb1c05>] vdev_disk_physio_completion+0x45/0x50 [zfs]
[17400.595280]  [<ffffffff9b65cbd7>] bio_endio+0x67/0xb0
[17400.595287]  [<ffffffff9b71f8e0>] blk_update_request+0x90/0x360
[17400.595295]  [<ffffffff9b8a98f4>] scsi_end_request+0x34/0x1e0
[17400.595300]  [<ffffffff9b8a9dd7>] scsi_io_completion+0x2d7/0x6a0
[17400.595306]  [<ffffffff9b89f14c>] scsi_finish_command+0xdc/0x140
[17400.595311]  [<ffffffff9b8a91b2>] scsi_softirq_done+0x132/0x160
[17400.595317]  [<ffffffff9b7272a6>] blk_done_softirq+0x96/0xc0
[17400.595322]  [<ffffffff9b49dba5>] __do_softirq+0xf5/0x280
[17400.595328]  [<ffffffff9bb28cec>] call_softirq+0x1c/0x30
[17400.595334]  [<ffffffff9b42e625>] do_softirq+0x65/0xa0
[17400.595339]  [<ffffffff9b49df25>] irq_exit+0x105/0x110
[17400.595344]  [<ffffffff9bb29fa6>] do_IRQ+0x56/0xf0
[17400.595349]  [<ffffffff9bb1c362>] common_interrupt+0x162/0x162
[17400.595352]  <EOI>  [<ffffffff9b96e704>] ? cpuidle_enter_state+0x54/0xd0
[17400.595361]  [<ffffffff9b96e85e>] cpuidle_idle_call+0xde/0x230
[17400.595367]  [<ffffffff9b4366ce>] arch_cpu_idle+0xe/0xb0
[17400.595372]  [<ffffffff9b4f5dea>] cpu_startup_entry+0x14a/0x1e0
[17400.595379]  [<ffffffff9b4571b7>] start_secondary+0x1f7/0x270
[17400.595385]  [<ffffffff9b4000d5>] start_cpu+0x5/0x14
[17400.817333] sd 0:1:2:0: [sdc] tag#2 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
[17400.817350] sd 0:1:2:0: [sdc] tag#2 CDB: Read(16) 88 00 00 00 00 01 5c bf 66 10 00 00 00 10 00 00
[17400.817356] blk_update_request: I/O error, dev sdc, sector 5851014672
[17400.817411] BUG: scheduling while atomic: swapper/4/0/0x10000100
[17400.817451] Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter zfs(POE) zunicode(POE) zlua(POE) intel_powerclamp coretemp kvm_intel ipmi_ssif kvm irqbypass crc32_pclmul ghash_clmulni_intel iTCO_wdt iTCO_vendor_support aesni_intel lrw gf128mul glue_helper ablk_helper zcommon(POE) cryptd znvpair(POE) zavl(POE) icp(POE) spl(OE) pcspkr joydev i2c_i801 lpc_ich ipmi_si sg ipmi_devintf ipmi_msghandler tpm_infineon
[17400.817515]  acpi_power_meter ioatdma i7core_edac shpchp acpi_cpufreq ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper qla2xxx syscopyarea sysfillrect ahci sysimgblt fb_sys_fops ttm libahci crct10dif_pclmul crct10dif_common drm crc32c_intel libata igb aacraid ptp pps_core dca i2c_algo_bit i2c_core scsi_transport_fc scsi_tgt dm_mirror dm_region_hash dm_log dm_mod
[17400.817552] CPU: 4 PID: 0 Comm: swapper/4 Kdump: loaded Tainted: P        W IOE  ------------   3.10.0-862.14.4.el7.x86_64 #1
[17400.817556] Hardware name: FUJITSU                          PRIMERGY RX300 S6             /D2619, BIOS 6.00 Rev. 1.13.2619.N1           01/19/2012
[17400.817560] Call Trace:
[17400.817564]  <IRQ>  [<ffffffff9bb13754>] dump_stack+0x19/0x1b
[17400.817577]  [<ffffffff9bb0d862>] __schedule_bug+0x64/0x72
[17400.817584]  [<ffffffff9bb18eeb>] __schedule+0x9fb/0xa20
[17400.817591]  [<ffffffff9b49defa>] ? irq_exit+0xda/0x110
[17400.817599]  [<ffffffff9b4ceb76>] __cond_resched+0x26/0x30
[17400.817604]  [<ffffffff9bb191da>] _cond_resched+0x3a/0x50
[17400.817610]  [<ffffffff9b5f92fc>] __kmalloc_node+0x5c/0x2b0
[17400.817623]  [<ffffffffc05644cf>] ? spl_kmem_alloc+0xdf/0x140 [spl]
[17400.817634]  [<ffffffffc05644cf>] spl_kmem_alloc+0xdf/0x140 [spl]
[17400.817705]  [<ffffffffc0cdb629>] __dprintf+0x69/0x150 [zfs]
[17400.817712]  [<ffffffff9b5fa282>] ? kmem_cache_free+0x1e2/0x200
[17400.817723]  [<ffffffffc05643e5>] ? spl_kmem_free+0x35/0x40 [spl]
[17400.817786]  [<ffffffffc0cb1abf>] vdev_disk_error.part.15+0x5f/0x70 [zfs]
[17400.817853]  [<ffffffffc0cb1ba0>] vdev_disk_dio_put+0x60/0x80 [zfs]
[17400.817915]  [<ffffffffc0cb1c05>] vdev_disk_physio_completion+0x45/0x50 [zfs]
[17400.817922]  [<ffffffff9b65cbd7>] bio_endio+0x67/0xb0
[17400.817929]  [<ffffffff9b71f8e0>] blk_update_request+0x90/0x360
[17400.817937]  [<ffffffff9b8a98f4>] scsi_end_request+0x34/0x1e0
[17400.817942]  [<ffffffff9b8a9dd7>] scsi_io_completion+0x2d7/0x6a0
[17400.817947]  [<ffffffff9b89f14c>] scsi_finish_command+0xdc/0x140
[17400.817952]  [<ffffffff9b8a91b2>] scsi_softirq_done+0x132/0x160
[17400.817958]  [<ffffffff9b7272a6>] blk_done_softirq+0x96/0xc0
[17400.817963]  [<ffffffff9b49dba5>] __do_softirq+0xf5/0x280
[17400.817969]  [<ffffffff9bb28cec>] call_softirq+0x1c/0x30
[17400.817976]  [<ffffffff9b42e625>] do_softirq+0x65/0xa0
[17400.817980]  [<ffffffff9b49df25>] irq_exit+0x105/0x110
[17400.817985]  [<ffffffff9bb29fa6>] do_IRQ+0x56/0xf0
[17400.817990]  [<ffffffff9bb1c362>] common_interrupt+0x162/0x162
[17400.817994]  <EOI>  [<ffffffff9b96e704>] ? cpuidle_enter_state+0x54/0xd0
[17400.818003]  [<ffffffff9b96e85e>] cpuidle_idle_call+0xde/0x230
[17400.818008]  [<ffffffff9b4366ce>] arch_cpu_idle+0xe/0xb0
[17400.818014]  [<ffffffff9b4f5dea>] cpu_startup_entry+0x14a/0x1e0
[17400.818021]  [<ffffffff9b4571b7>] start_secondary+0x1f7/0x270
[17400.818027]  [<ffffffff9b4000d5>] start_cpu+0x5/0x14
[17401.306204] sd 0:1:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
[17401.306222] sd 0:1:2:0: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 01 5c bf 64 10 00 00 00 10 00 00
[17401.306228] blk_update_request: I/O error, dev sdc, sector 5851014160
[17401.306283] BUG: scheduling while atomic: swapper/4/0/0x10000100
[17401.306323] Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter zfs(POE) zunicode(POE) zlua(POE) intel_powerclamp coretemp kvm_intel ipmi_ssif kvm irqbypass crc32_pclmul ghash_clmulni_intel iTCO_wdt iTCO_vendor_support aesni_intel lrw gf128mul glue_helper ablk_helper zcommon(POE) cryptd znvpair(POE) zavl(POE) icp(POE) spl(OE) pcspkr joydev i2c_i801 lpc_ich ipmi_si sg ipmi_devintf ipmi_msghandler tpm_infineon
[17401.306386]  acpi_power_meter ioatdma i7core_edac shpchp acpi_cpufreq ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper qla2xxx syscopyarea sysfillrect ahci sysimgblt fb_sys_fops ttm libahci crct10dif_pclmul crct10dif_common drm crc32c_intel libata igb aacraid ptp pps_core dca i2c_algo_bit i2c_core scsi_transport_fc scsi_tgt dm_mirror dm_region_hash dm_log dm_mod
[17401.306424] CPU: 4 PID: 0 Comm: swapper/4 Kdump: loaded Tainted: P        W IOE  ------------   3.10.0-862.14.4.el7.x86_64 #1
[17401.306427] Hardware name: FUJITSU                          PRIMERGY RX300 S6             /D2619, BIOS 6.00 Rev. 1.13.2619.N1           01/19/2012
[17401.306431] Call Trace:
[17401.306436]  <IRQ>  [<ffffffff9bb13754>] dump_stack+0x19/0x1b
[17401.306449]  [<ffffffff9bb0d862>] __schedule_bug+0x64/0x72
[17401.306456]  [<ffffffff9bb18eeb>] __schedule+0x9fb/0xa20
[17401.306463]  [<ffffffff9b49defa>] ? irq_exit+0xda/0x110
[17401.306471]  [<ffffffff9b4ceb76>] __cond_resched+0x26/0x30
[17401.306476]  [<ffffffff9bb191da>] _cond_resched+0x3a/0x50
[17401.306482]  [<ffffffff9b5f92fc>] __kmalloc_node+0x5c/0x2b0
[17401.306496]  [<ffffffffc05644cf>] ? spl_kmem_alloc+0xdf/0x140 [spl]
[17401.306506]  [<ffffffffc05644cf>] spl_kmem_alloc+0xdf/0x140 [spl]
[17401.306585]  [<ffffffffc0cdb629>] __dprintf+0x69/0x150 [zfs]
[17401.306592]  [<ffffffff9b5fa282>] ? kmem_cache_free+0x1e2/0x200
[17401.306603]  [<ffffffffc05643e5>] ? spl_kmem_free+0x35/0x40 [spl]
[17401.306662]  [<ffffffffc0cb1abf>] vdev_disk_error.part.15+0x5f/0x70 [zfs]
[17401.306722]  [<ffffffffc0cb1ba0>] vdev_disk_dio_put+0x60/0x80 [zfs]
[17401.306781]  [<ffffffffc0cb1c05>] vdev_disk_physio_completion+0x45/0x50 [zfs]
[17401.306788]  [<ffffffff9b65cbd7>] bio_endio+0x67/0xb0
[17401.306795]  [<ffffffff9b71f8e0>] blk_update_request+0x90/0x360
[17401.306802]  [<ffffffff9b8a98f4>] scsi_end_request+0x34/0x1e0
[17401.306808]  [<ffffffff9b8a9dd7>] scsi_io_completion+0x2d7/0x6a0
[17401.306813]  [<ffffffff9b89f14c>] scsi_finish_command+0xdc/0x140
[17401.306818]  [<ffffffff9b8a91b2>] scsi_softirq_done+0x132/0x160
[17401.306824]  [<ffffffff9b7272a6>] blk_done_softirq+0x96/0xc0
[17401.306829]  [<ffffffff9b49dba5>] __do_softirq+0xf5/0x280
[17401.306835]  [<ffffffff9bb28cec>] call_softirq+0x1c/0x30
[17401.306842]  [<ffffffff9b42e625>] do_softirq+0x65/0xa0
[17401.306846]  [<ffffffff9b49df25>] irq_exit+0x105/0x110
[17401.306851]  [<ffffffff9bb29fa6>] do_IRQ+0x56/0xf0
[17401.306856]  [<ffffffff9bb1c362>] common_interrupt+0x162/0x162
[17401.306859]  <EOI>  [<ffffffff9b96e704>] ? cpuidle_enter_state+0x54/0xd0
[17401.306869]  [<ffffffff9b96e85e>] cpuidle_idle_call+0xde/0x230
[17401.306874]  [<ffffffff9b4366ce>] arch_cpu_idle+0xe/0xb0
[17401.306880]  [<ffffffff9b4f5dea>] cpu_startup_entry+0x14a/0x1e0
[17401.306887]  [<ffffffff9b4571b7>] start_secondary+0x1f7/0x270
[17401.306893]  [<ffffffff9b4000d5>] start_cpu+0x5/0x14
devZer0 commented 5 years ago

it also happens with 0.7.12

loli10K commented 5 years ago

it also happens with 0.7.12

Comparing 0.8.x with 0.7.x seems to suggest that some of the code responsible for the backtrace in your report has changed, and from a cursory reading i would have guessed this oops cannot happen in the 0.7.x release branch.

If you are actually able to reproduce this running 0.7.x please post a new backtrace.

devZer0 commented 5 years ago

yes, you may be right, i will re-check. i found yesterday that when downgrading the 0.8rc2 install to 0.7.12 i was curious, that before reboot i could use the pool but after reboot i could not import it due to "missing features". and that makes me think that perhaps 0.8rc2 zfs modules may not have been unloaded and 0.7.12 loaded before re-testing this issue... i did the downgrade to check if the problem is reproducible there

devZer0 commented 5 years ago

you are right, my last observation must be wrong. i was not able to reproduce the problem with 0.7.12 again.