search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.48k stars 1.74k forks source link

Linux 4.4.0-38 / ZOL v0.6.5.8: NULL pointer dereference on zfs receive #5188

Closed bendikro closed 8 years ago

bendikro commented 8 years ago

I've been trying to copy my zfs file systems over ssh, but it keeps failing. zfs send -R rpool/ROOT@transfer2 | mbuffer -s 128k -m 1G 2>/dev/null | ssh garfield "mbuffer -s 128k -m 1G | zfs recv -vFdu rpool

I first used the version from Ubuntu 16.04 PPA (0.6.5.6), which failed on one file system, so I tried 0.6.5.8 compiled from source, and it successfully transferred the 70GB filesystem. However, it still fails on another file system. This error occurred first after transferring 20GB.

[  111.443150] spl: module verification failed: signature and/or required key missing - tainting kernel
[  111.445932] SPL: Loaded module v0.6.5.8-1
[  111.448718] SPLAT: Loaded module v0.6.5.8-1
[  111.451447] zavl: module license 'CDDL' taints kernel.
[  111.451448] Disabling lock debugging due to kernel taint
[  111.488517] ZFS: Loaded module v0.6.5.8-1, ZFS pool version 5000, ZFS filesystem version 5
[  234.862637] SPL: using hostid 0x00000000
[ 1818.921077] BUG: unable to handle kernel NULL pointer dereference at 0000000000000aa8
[ 1818.923193] IP: [<ffffffffc0a50228>] dbuf_evict_user.isra.7+0x18/0x40 [zfs]
[ 1818.924156] PGD 0 
[ 1818.924816] Oops: 0000 [#1] SMP 
[ 1818.925477] Modules linked in: zfs(POE) zcommon(POE) zunicode(POE) znvpair(POE) zavl(POE) splat(OE) spl(OE) bnep binfmt_misc snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm i915_bpo irqbypass crct10dif_pclmul crc32_pclmul snd_seq_midi snd_seq_midi_event aesni_intel snd_rawmidi intel_ips aes_x86_64 lrw gf128mul glue_helper ablk_helper snd_hda_intel cryptd snd_hda_codec snd_hda_core snd_hwdep snd_pcm serio_raw snd_seq snd_seq_device snd_timer snd mei_me soundcore mei shpchp hci_uart 8250_fintek btbcm btqca btintel bluetooth intel_lpss_acpi intel_lpss acpi_als acpi_pad tpm_infineon kfifo_buf industrialio nls_iso8859_1 input_leds mac_hid parport_pc ppdev lp parport autofs4 hid_generic usbhid nouveau mxm_wmi i2c_algo_bit
[ 1818.929013]  ttm psmouse drm_kms_helper syscopyarea nvme sysfillrect alx sysimgblt mdio fb_sys_fops ahci drm libahci wmi pinctrl_sunrisepoint i2c_hid video pinctrl_intel hid fjes
[ 1818.929927] CPU: 2 PID: 2677 Comm: dbu_evict Tainted: P           OE   4.4.0-38-generic #57-Ubuntu
[ 1818.930389] Hardware name: Gigabyte Technology Co., Ltd. Z170X-Gaming 3/Z170X-Gaming 3, BIOS F2 07/24/2015
[ 1818.930865] task: ffff880fb52b6740 ti: ffff880f74d14000 task.ti: ffff880f74d14000
[ 1818.931346] RIP: 0010:[<ffffffffc0a50228>]  [<ffffffffc0a50228>] dbuf_evict_user.isra.7+0x18/0x40 [zfs]
[ 1818.931858] RSP: 0018:ffff880f74d17d68  EFLAGS: 00010206
[ 1818.932351] RAX: ffff880fb52b6740 RBX: ffff88086ecafd40 RCX: 00000001820000c1
[ 1818.932853] RDX: 0000000000000a00 RSI: ffffea001d56fb80 RDI: ffff88086ecafe50
[ 1818.933359] RBP: ffff880f74d17d98 R08: 0000000000000a00 R09: 00000001820000c1
[ 1818.933869] R10: ffff880755bee9e8 R11: 0000000000000001 R12: ffff88086ecaefc0
[ 1818.934383] R13: 0000000000000000 R14: ffff880fb412bb70 R15: ffff880fb412bb48
[ 1818.934903] FS:  0000000000000000(0000) GS:ffff880fffc80000(0000) knlGS:0000000000000000
[ 1818.935431] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1818.935962] CR2: 0000000000000aa8 CR3: 0000000002e0a000 CR4: 00000000003406e0
[ 1818.936498] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1818.937032] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1818.937561] Stack:
[ 1818.938083]  ffff880f74d17d98 ffffffffc0a50483 ffff88086ecafd40 ffff880fb412b800
[ 1818.938630]  0000000000000000 ffff880fb412bb70 ffff880f74d17db0 ffffffffc0a505c2
[ 1818.939182]  ffff8808734a51c0 ffff880f74d17de8 ffffffffc0a6ab45 ffff8809dc98a630
[ 1818.939816] Call Trace:
[ 1818.940685]  [<ffffffffc0a50483>] ? dbuf_clear+0x23/0x150 [zfs]
[ 1818.941400]  [<ffffffffc0a505c2>] dbuf_evict+0x12/0x20 [zfs]
[ 1818.941973]  [<ffffffffc0a6ab45>] dnode_destroy+0x115/0x1f0 [zfs]
[ 1818.942546]  [<ffffffffc0a6ac64>] dnode_buf_pageout+0x44/0xc0 [zfs]
[ 1818.943114]  [<ffffffffc06110d8>] taskq_thread+0x238/0x420 [spl]
[ 1818.943685]  [<ffffffff810ac190>] ? wake_up_q+0x70/0x70
[ 1818.944249]  [<ffffffffc0610ea0>] ? taskq_cancel_id+0x140/0x140 [spl]
[ 1818.944817]  [<ffffffff810a08d8>] kthread+0xd8/0xf0
[ 1818.945385]  [<ffffffff810a0800>] ? kthread_create_on_node+0x1e0/0x1e0
[ 1818.945960]  [<ffffffff81830a8f>] ret_from_fork+0x3f/0x70
[ 1818.946533]  [<ffffffff810a0800>] ? kthread_create_on_node+0x1e0/0x1e0
[ 1818.947112] Code: ff ff ff 5d f3 c3 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 e8 ab 2b de c0 48 8b 17 48 85 d2 74 24 55 48 c7 07 00 00 00 00 49 89 d0 <48> 8b b2 a8 00 00 00 48 8b 3d fa c6 1c 00 31 c9 48 89 e5 e8 e0 
[ 1818.948945] RIP  [<ffffffffc0a50228>] dbuf_evict_user.isra.7+0x18/0x40 [zfs]
[ 1818.949982]  RSP <ffff880f74d17d68>
[ 1818.950977] CR2: 0000000000000aa8
[ 1818.951952] ---[ end trace 398915886a08e008 ]---
bendikro commented 8 years ago

Rebooted and reran the command gave me this:

[   23.644498] spl: module verification failed: signature and/or required key missing - tainting kernel
[   23.647560] SPL: Loaded module v0.6.5.8-1
[   23.650548] SPLAT: Loaded module v0.6.5.8-1
[   23.653150] zavl: module license 'CDDL' taints kernel.
[   23.653152] Disabling lock debugging due to kernel taint
[   23.690610] ZFS: Loaded module v0.6.5.8-1, ZFS pool version 5000, ZFS filesystem version 5
[   40.212298] SPL: using hostid 0x00000000
[  424.023896] VERIFY3(sa.sa_magic == 0x2F505A) failed (979712098 == 3100762)
[  424.024527] PANIC at zfs_vfsops.c:426:zfs_space_delta_cb()
[  424.025172] Showing stack for process 3157
[  424.025175] CPU: 0 PID: 3157 Comm: txg_sync Tainted: P           OE   4.4.0-38-generic #57-Ubuntu
[  424.025176] Hardware name: Gigabyte Technology Co., Ltd. Z170X-Gaming 3/Z170X-Gaming 3, BIOS F2 07/24/2015
[  424.025178]  ffffffff813f1b73 0000000088fb80dd ffff880f711a3840 ffffffff813f1b73
[  424.025182]  ffffffffc0bb9a11 00000000000001aa ffff880f711a3850 ffffffffc069fd22
[  424.025185]  ffff880f711a39d8 ffffffffc069fdeb ffff880fb4c32eb8 0000000000000030
[  424.025188] Call Trace:
[  424.025193]  [<ffffffff813f1b73>] ? dump_stack+0x63/0x90
[  424.025195]  [<ffffffff813f1b73>] dump_stack+0x63/0x90
[  424.025204]  [<ffffffffc069fd22>] spl_dumpstack+0x42/0x50 [spl]
[  424.025208]  [<ffffffffc069fdeb>] spl_panic+0xbb/0xf0 [spl]
[  424.025212]  [<ffffffff81158dcf>] ? stack_trace_call+0x4f/0x80
[  424.025216]  [<ffffffff81832e37>] ? ftrace_call+0x5/0x34
[  424.025218]  [<ffffffff81158dcf>] ? stack_trace_call+0x4f/0x80
[  424.025224]  [<ffffffffc069fd35>] ? spl_panic+0x5/0xf0 [spl]
[  424.025259]  [<ffffffffc0b6bf43>] zfs_space_delta_cb+0xc3/0x190 [zfs]
[  424.025279]  [<ffffffffc0aefc52>] dmu_objset_userquota_get_ids+0x112/0x3b0 [zfs]
[  424.025298]  [<ffffffffc0afe73b>] dnode_sync+0xcb/0x830 [zfs]
[  424.025302]  [<ffffffff8182c815>] ? _cond_resched+0x5/0x30
[  424.025304]  [<ffffffff8182e5d5>] ? mutex_unlock+0x5/0x20
[  424.025322]  [<ffffffffc0aeea6e>] dmu_objset_sync_dnodes+0xce/0xf0 [zfs]
[  424.025342]  [<ffffffffc0aeec67>] dmu_objset_sync+0x1d7/0x320 [zfs]
[  424.025359]  [<ffffffffc0aed100>] ? recordsize_changed_cb+0x20/0x20 [zfs]
[  424.025376]  [<ffffffffc0aeedb0>] ? dmu_objset_sync+0x320/0x320 [zfs]
[  424.025398]  [<ffffffffc0b06efe>] dsl_dataset_sync+0x4e/0x90 [zfs]
[  424.025420]  [<ffffffffc0b0fdcf>] dsl_pool_sync+0x9f/0x430 [zfs]
[  424.025446]  [<ffffffffc0b2ade6>] spa_sync+0x366/0xb10 [zfs]
[  424.025471]  [<ffffffffc0b3b9b6>] txg_sync_thread+0x3a6/0x600 [zfs]
[  424.025496]  [<ffffffffc0b3b610>] ? txg_delay+0x160/0x160 [zfs]
[  424.025502]  [<ffffffffc069ce31>] thread_generic_wrapper+0x71/0x80 [spl]
[  424.025506]  [<ffffffffc069cdc0>] ? __thread_exit+0x20/0x20 [spl]
[  424.025508]  [<ffffffff810a08d8>] kthread+0xd8/0xf0
[  424.025511]  [<ffffffff810a0800>] ? kthread_create_on_node+0x1e0/0x1e0
[  424.025514]  [<ffffffff81830a8f>] ret_from_fork+0x3f/0x70
[  424.025516]  [<ffffffff810a0800>] ? kthread_create_on_node+0x1e0/0x1e0
bendikro commented 8 years ago

Reran again and got this:

[   25.966812] spl: module verification failed: signature and/or required key missing - tainting kernel
[   25.969755] SPL: Loaded module v0.6.5.8-1
[   25.972947] SPLAT: Loaded module v0.6.5.8-1
[   25.975694] zavl: module license 'CDDL' taints kernel.
[   25.975696] Disabling lock debugging due to kernel taint
[   26.012737] ZFS: Loaded module v0.6.5.8-1, ZFS pool version 5000, ZFS filesystem version 5
[   37.721092] SPL: using hostid 0x00000000
[  375.992402] BUG: unable to handle kernel paging request at 00000000346545f4
[  375.992745] IP: [<ffffffff811ebd27>] kmem_cache_alloc+0x77/0x1f0
[  375.993091] PGD 341bb067 PUD 3413b067 PMD 0 
[  375.993436] Oops: 0000 [#1] SMP 
[  375.993770] Modules linked in: zfs(POE) zcommon(POE) zunicode(POE) znvpair(POE) zavl(POE) splat(OE) spl(OE) bnep snd_hda_codec_hdmi binfmt_misc snd_hda_codec_realtek nls_iso8859_1 snd_hda_codec_generic intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel i915_bpo kvm irqbypass crct10dif_pclmul crc32_pclmul intel_ips aesni_intel aes_x86_64 lrw gf128mul glue_helper snd_hda_intel ablk_helper cryptd snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_seq_midi serio_raw snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer mei_me snd mei soundcore shpchp input_leds hci_uart btbcm btqca btintel bluetooth 8250_fintek intel_lpss_acpi intel_lpss acpi_als tpm_infineon kfifo_buf acpi_pad mac_hid industrialio parport_pc ppdev lp parport autofs4 hid_generic usbhid nouveau mxm_wmi i2c_algo_bit
[  375.996223]  psmouse ttm drm_kms_helper syscopyarea nvme alx sysfillrect sysimgblt mdio fb_sys_fops ahci drm libahci wmi video pinctrl_sunrisepoint i2c_hid pinctrl_intel hid fjes
[  375.997132] CPU: 3 PID: 3172 Comm: lt-zfs Tainted: P           OE   4.4.0-38-generic #57-Ubuntu
[  375.997597] Hardware name: Gigabyte Technology Co., Ltd. Z170X-Gaming 3/Z170X-Gaming 3, BIOS F2 07/24/2015
[  375.998075] task: ffff880fb2fb9d80 ti: ffff880f6f484000 task.ti: ffff880f6f484000
[  375.998554] RIP: 0010:[<ffffffff811ebd27>]  [<ffffffff811ebd27>] kmem_cache_alloc+0x77/0x1f0
[  375.999046] RSP: 0018:ffff880f6f487828  EFLAGS: 00010202
[  375.999537] RAX: 0000000000000000 RBX: 0000000002404200 RCX: 0000000000072a39
[  376.000038] RDX: 0000000000072a38 RSI: 0000000002404200 RDI: 000060f000005e10
[  376.000536] RBP: ffff880f6f487858 R08: ffffe8ffffcc5e10 R09: 00000000346545f4
[  376.001034] R10: ffffea003a740e00 R11: ffff880e86dae900 R12: 0000000002404200
[  376.001530] R13: ffffffffc066ec32 R14: ffff880fb45e8e00 R15: ffff880fb45e8e00
[  376.002028] FS:  00007fe358a6f780(0000) GS:ffff880fffcc0000(0000) knlGS:0000000000000000
[  376.002534] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  376.003039] CR2: 00000000346545f4 CR3: 0000000f6f441000 CR4: 00000000003406e0
[  376.003551] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  376.004064] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  376.004577] Stack:
[  376.005087]  ffff880f6f4878e8 ffff880fb45e8e00 0000000000000000 0000000000000000
[  376.005624]  0000000000000000 ffff880fb3db5400 ffff880f6f4878f8 ffffffffc066ec32
[  376.006165]  000000000000000e 0000000000000000 0000000000000286 ffff8807686fe240
[  376.006709] Call Trace:
[  376.007248]  [<ffffffffc066ec32>] spl_kmem_cache_alloc+0x72/0x7d0 [spl]
[  376.007816]  [<ffffffff81832e37>] ? ftrace_call+0x5/0x34
[  376.008363]  [<ffffffffc066ebc5>] ? spl_kmem_cache_alloc+0x5/0x7d0 [spl]
[  376.008917]  [<ffffffff81832e37>] ? ftrace_call+0x5/0x34
[  376.009490]  [<ffffffffc0b7baec>] dbuf_create+0x3c/0x440 [zfs]
[  376.010061]  [<ffffffffc0b7bab5>] ? dbuf_create+0x5/0x440 [zfs]
[  376.010629]  [<ffffffffc0b7bf12>] dbuf_create_bonus+0x22/0x30 [zfs]
[  376.011200]  [<ffffffffc0b84f70>] dmu_bonus_hold+0x190/0x280 [zfs]
[  376.011777]  [<ffffffffc0b8d0f3>] restore_object+0x323/0x3c0 [zfs]
[  376.012352]  [<ffffffffc0b8cdd5>] ? restore_object+0x5/0x3c0 [zfs]
[  376.012929]  [<ffffffffc0b8f2de>] dmu_recv_stream+0x7ae/0xb50 [zfs]
[  376.013496]  [<ffffffff81158dcf>] ? stack_trace_call+0x4f/0x80
[  376.014067]  [<ffffffff81158dcf>] ? stack_trace_call+0x4f/0x80
[  376.014654]  [<ffffffffc0bfde89>] zfs_ioc_recv+0x209/0xc20 [zfs]
[  376.015224]  [<ffffffffc0008077>] ? 0xffffffffc0008077
[  376.015795]  [<ffffffff81832e37>] ? ftrace_call+0x5/0x34
[  376.016357]  [<ffffffff81158dcf>] ? stack_trace_call+0x4f/0x80
[  376.016910]  [<ffffffff81832e37>] ? ftrace_call+0x5/0x34
[  376.017460]  [<ffffffff8182e5d5>] ? mutex_unlock+0x5/0x20
[  376.018008]  [<ffffffff8182c815>] ? _cond_resched+0x5/0x30
[  376.018564]  [<ffffffffc0bfdc85>] ? zfs_ioc_recv+0x5/0xc20 [zfs]
[  376.019107]  [<ffffffffc0bfcc6b>] zfsdev_ioctl+0x44b/0x4e0 [zfs]
[  376.019617]  [<ffffffff812211ff>] do_vfs_ioctl+0x29f/0x490
[  376.020108]  [<ffffffff81220f65>] ? do_vfs_ioctl+0x5/0x490
[  376.020585]  [<ffffffff8122b385>] ? __fget_light+0x5/0x60
[  376.021045]  [<ffffffff81221469>] SyS_ioctl+0x79/0x90
[  376.021490]  [<ffffffff818306f2>] entry_SYSCALL_64_fastpath+0x16/0x71
[  376.021924] Code: 08 65 4c 03 05 83 e4 e1 7e 49 83 78 10 00 4d 8b 08 0f 84 29 01 00 00 4d 85 c9 0f 84 20 01 00 00 49 63 47 20 48 8d 4a 01 49 8b 3f <49> 8b 1c 01 4c 89 c8 65 48 0f c7 0f 0f 94 c0 84 c0 74 bb 49 63 
[  376.022929] RIP  [<ffffffff811ebd27>] kmem_cache_alloc+0x77/0x1f0
[  376.023371]  RSP <ffff880f6f487828>
[  376.023801] CR2: 00000000346545f4
[  376.024223] ---[ end trace 9b157c6e3eb970bf ]---
tuxoko commented 8 years ago

Please boot with slub_debug and try again.

bendikro commented 8 years ago

@tuxoko I compiled from master before I saw your reply, so this error is with the latest zfs (12fa7f34) and spl (8acfb2bc)

[   26.950393] spl: module verification failed: signature and/or required key missing - tainting kernel
[   26.956240] SPL: Loaded module v0.7.0-rc1_4_g8acfb2b
[   26.969676] SPLAT: Loaded module v0.7.0-rc1_4_g8acfb2b
[   26.980527] zavl: module license 'CDDL' taints kernel.
[   26.980529] Disabling lock debugging due to kernel taint
[   28.867571] ZFS: Loaded module v0.7.0-rc1_45_g12fa7f3, ZFS pool version 5000, ZFS filesystem version 5
[   45.030327] SPL: using hostid 0x00000000
[  762.355996] =============================================================================
[  762.356322] BUG arc_buf_hdr_t_full (Tainted: P           OE  ): Padding overwritten. 0xffff8808022dff41-0xffff8808022dff79
[  762.356662] -----------------------------------------------------------------------------

[  762.357347] INFO: Slab 0xffffea002008b700 objects=24 used=24 fp=0x          (null) flags=0x17fff8000004081
[  762.357714] CPU: 2 PID: 2241 Comm: txg_sync Tainted: P    B      OE   4.4.0-38-generic #57-Ubuntu
[  762.357715] Hardware name: Gigabyte Technology Co., Ltd. Z170X-Gaming 3/Z170X-Gaming 3, BIOS F2 07/24/2015
[  762.357716]  ffffffff813f1b73 0000000026038bac ffff880f6d1f77c0 ffffffff813f1b73
[  762.357718]  ffffea002008b700 ffff880fb5f92d00 ffff880f6d1f7898 ffffffff811e82b0
[  762.357720]  ffff880f00000028 ffff880f6d1f78a8 ffff880f6d1f7858 646461506d1f78e8
[  762.357722] Call Trace:
[  762.357726]  [<ffffffff813f1b73>] ? dump_stack+0x63/0x90
[  762.357727]  [<ffffffff813f1b73>] dump_stack+0x63/0x90
[  762.357730]  [<ffffffff811e82b0>] slab_err+0xb0/0xe0
[  762.357731]  [<ffffffff811e8205>] ? slab_err+0x5/0xe0
[  762.357733]  [<ffffffff811e98f1>] slab_pad_check.part.46+0xf1/0x180
[  762.357735]  [<ffffffff811e9985>] ? check_slab+0x5/0xb0
[  762.357736]  [<ffffffff811e9a22>] check_slab+0xa2/0xb0
[  762.357738]  [<ffffffff811ea885>] free_debug_processing+0x65/0x2f0
[  762.357739]  [<ffffffff811ea825>] ? free_debug_processing+0x5/0x2f0
[  762.357743]  [<ffffffffc07bdad5>] ? spl_kmem_cache_free+0x135/0x1c0 [spl]
[  762.357745]  [<ffffffff811ec745>] __slab_free+0x195/0x2c0
[  762.357747]  [<ffffffff81158dcf>] ? stack_trace_call+0x4f/0x80
[  762.357748]  [<ffffffff811ec5b5>] ? __slab_free+0x5/0x2c0
[  762.357750]  [<ffffffff811ecd64>] kmem_cache_free+0x1d4/0x1e0
[  762.357753]  [<ffffffffc07bdad5>] spl_kmem_cache_free+0x135/0x1c0 [spl]
[  762.357771]  [<ffffffffc0903c38>] arc_hdr_destroy+0x148/0x170 [zfs]
[  762.357781]  [<ffffffffc0907779>] arc_freed+0x69/0xb0 [zfs]
[  762.357803]  [<ffffffffc09b9be5>] zio_free_sync+0x45/0x140 [zfs]
[  762.357821]  [<ffffffffc0956950>] ? spa_avz_build+0x130/0x130 [zfs]
[  762.357842]  [<ffffffffc09b9ba5>] ? zio_free_sync+0x5/0x140 [zfs]
[  762.357860]  [<ffffffffc0956988>] spa_free_sync_cb+0x38/0x50 [zfs]
[  762.357871]  [<ffffffffc090a785>] bpobj_iterate_impl+0x115/0x7a0 [zfs]
[  762.357888]  [<ffffffffc0956950>] ? spa_avz_build+0x130/0x130 [zfs]
[  762.357890]  [<ffffffff81832e37>] ? ftrace_call+0x5/0x34
[  762.357892]  [<ffffffff81832e37>] ? ftrace_call+0x5/0x34
[  762.357913]  [<ffffffffc09b9222>] ? zio_create+0x3d2/0x470 [zfs]
[  762.357930]  [<ffffffffc0956950>] ? spa_avz_build+0x130/0x130 [zfs]
[  762.357947]  [<ffffffffc0956950>] ? spa_avz_build+0x130/0x130 [zfs]
[  762.357957]  [<ffffffffc090a675>] ? bpobj_iterate_impl+0x5/0x7a0 [zfs]
[  762.357967]  [<ffffffffc090ae15>] ? bpobj_iterate+0x5/0x20 [zfs]
[  762.357978]  [<ffffffffc090ae24>] bpobj_iterate+0x14/0x20 [zfs]
[  762.357994]  [<ffffffffc0959361>] spa_sync+0x6d1/0xa50 [zfs]
[  762.357996]  [<ffffffff810c3d92>] ? autoremove_wake_function+0x12/0x40
[  762.358016]  [<ffffffffc0969c96>] txg_sync_thread+0x3a6/0x600 [zfs]
[  762.358035]  [<ffffffffc09698f0>] ? txg_delay+0x160/0x160 [zfs]
[  762.358038]  [<ffffffffc07be441>] thread_generic_wrapper+0x71/0x80 [spl]
[  762.358041]  [<ffffffffc07be3d0>] ? __thread_exit+0x20/0x20 [spl]
[  762.358042]  [<ffffffff810a08d8>] kthread+0xd8/0xf0
[  762.358044]  [<ffffffff810a0800>] ? kthread_create_on_node+0x1e0/0x1e0
[  762.358046]  [<ffffffff81830a8f>] ret_from_fork+0x3f/0x70
[  762.358047]  [<ffffffff810a0800>] ? kthread_create_on_node+0x1e0/0x1e0
[  762.358049] Padding ffff8808022dfcfa: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  762.358432] Padding ffff8808022dfd0a: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  762.358800] Padding ffff8808022dfd1a: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  762.359161] Padding ffff8808022dfd2a: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  762.359515] Padding ffff8808022dfd3a: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  762.359868] Padding ffff8808022dfd4a: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  762.360216] Padding ffff8808022dfd5a: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
[  762.360558] Padding ffff8808022dfd6a: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 5a 5a  ..............ZZ
[  762.360896] Padding ffff8808022dfd7a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.361240] Padding ffff8808022dfd8a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.361581] Padding ffff8808022dfd9a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.361918] Padding ffff8808022dfdaa: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.362253] Padding ffff8808022dfdba: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.362588] Padding ffff8808022dfdca: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.362917] Padding ffff8808022dfdda: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.363245] Padding ffff8808022dfdea: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.363570] Padding ffff8808022dfdfa: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.363893] Padding ffff8808022dfe0a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.364210] Padding ffff8808022dfe1a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.364522] Padding ffff8808022dfe2a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.364834] Padding ffff8808022dfe3a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.365143] Padding ffff8808022dfe4a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.365449] Padding ffff8808022dfe5a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.365750] Padding ffff8808022dfe6a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.366050] Padding ffff8808022dfe7a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.366351] Padding ffff8808022dfe8a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.366647] Padding ffff8808022dfe9a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.366939] Padding ffff8808022dfeaa: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.367227] Padding ffff8808022dfeba: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.367514] Padding ffff8808022dfeca: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.367794] Padding ffff8808022dfeda: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.368068] Padding ffff8808022dfeea: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.368338] Padding ffff8808022dfefa: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.368606] Padding ffff8808022dff0a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.368873] Padding ffff8808022dff1a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.369134] Padding ffff8808022dff2a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.369392] Padding ffff8808022dff3a: 5a 5a 5a 5a 5a 5a 5a da 5a 5a 5a 5a 5a 5a 5a da  ZZZZZZZ.ZZZZZZZ.
[  762.369650] Padding ffff8808022dff4a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.369906] Padding ffff8808022dff5a: 5a 5a 5a 5a 5a 5a 5a 4e 5a 5a 5a 5a 5a 5a 5a 4e  ZZZZZZZNZZZZZZZN
[  762.370161] Padding ffff8808022dff6a: 5a 5a 5a 5a 5a 5a 5a 4e 5a 5a 5a 5a 5a 5a 5a 4e  ZZZZZZZNZZZZZZZN
[  762.370418] FIX arc_buf_hdr_t_full: Restoring 0xffff8808022dfcfa-0xffff8808022dff79=0x5a

[  798.681132] =============================================================================
[  798.681596] BUG zio_buf_8192 (Tainted: P    B      OE  ): Padding overwritten. 0xffff88087b25fe41-0xffff88087b25fe79
[  798.682040] -----------------------------------------------------------------------------

[  798.682916] INFO: Slab 0xffffea0021ec9600 objects=3 used=3 fp=0x          (null) flags=0x17fff8000004080
[  798.683374] CPU: 5 PID: 10761 Comm: z_wr_iss Tainted: P    B      OE   4.4.0-38-generic #57-Ubuntu
[  798.683375] Hardware name: Gigabyte Technology Co., Ltd. Z170X-Gaming 3/Z170X-Gaming 3, BIOS F2 07/24/2015
[  798.683377]  ffffffff813f1b73 00000000cdb7e766 ffff880750c27998 ffffffff813f1b73
[  798.683381]  ffffea0021ec9600 ffff880fb5f90000 ffff880750c27a70 ffffffff811e82b0
[  798.683384]  0000000000000028 ffff880750c27a80 ffff880750c27a30 6464615050c27ac0
[  798.683387] Call Trace:
[  798.683392]  [<ffffffff813f1b73>] ? dump_stack+0x63/0x90
[  798.683394]  [<ffffffff813f1b73>] dump_stack+0x63/0x90
[  798.683397]  [<ffffffff811e82b0>] slab_err+0xb0/0xe0
[  798.683400]  [<ffffffff811e8205>] ? slab_err+0x5/0xe0
[  798.683403]  [<ffffffff811e98f1>] slab_pad_check.part.46+0xf1/0x180
[  798.683405]  [<ffffffff811e9985>] ? check_slab+0x5/0xb0
[  798.683411]  [<ffffffffc07bd222>] ? spl_kmem_cache_alloc+0x72/0x7f0 [spl]
[  798.683413]  [<ffffffff811e9a22>] check_slab+0xa2/0xb0
[  798.683415]  [<ffffffff811e9a55>] alloc_debug_processing+0x25/0x180
[  798.683417]  [<ffffffff811eb891>] ___slab_alloc+0x451/0x470
[  798.683419]  [<ffffffffc0008077>] ? 0xffffffffc0008077
[  798.683424]  [<ffffffffc07bd222>] ? spl_kmem_cache_alloc+0x72/0x7f0 [spl]
[  798.683426]  [<ffffffff811eb445>] ? ___slab_alloc+0x5/0x470
[  798.683428]  [<ffffffff811eb8b5>] ? __slab_alloc+0x5/0x40
[  798.683433]  [<ffffffffc07bd222>] ? spl_kmem_cache_alloc+0x72/0x7f0 [spl]
[  798.683435]  [<ffffffff811eb8d0>] __slab_alloc+0x20/0x40
[  798.683437]  [<ffffffff811ebe4f>] kmem_cache_alloc+0x19f/0x1f0
[  798.683441]  [<ffffffffc07bd222>] spl_kmem_cache_alloc+0x72/0x7f0 [spl]
[  798.683446]  [<ffffffffc07bd1b5>] ? spl_kmem_cache_alloc+0x5/0x7f0 [spl]
[  798.683480]  [<ffffffffc09b67d5>] ? zio_buf_alloc+0x5/0x60 [zfs]
[  798.683510]  [<ffffffffc09b6825>] zio_buf_alloc+0x55/0x60 [zfs]
[  798.683528]  [<ffffffffc09027ef>] arc_get_data_buf.isra.21+0x25f/0x390 [zfs]
[  798.683547]  [<ffffffffc090295e>] arc_hdr_alloc_pdata+0x3e/0xb0 [zfs]
[  798.683566]  [<ffffffffc09038c2>] arc_write_ready+0x1d2/0x200 [zfs]
[  798.683594]  [<ffffffffc09b6a75>] ? zio_wait_for_children+0x65/0x80 [zfs]
[  798.683624]  [<ffffffffc09bbdf6>] zio_ready+0x66/0x3c0 [zfs]
[  798.683629]  [<ffffffffc07be828>] ? taskq_member+0x18/0x30 [spl]
[  798.683658]  [<ffffffffc09b7663>] zio_execute+0x83/0xe0 [zfs]
[  798.683663]  [<ffffffffc07bf835>] taskq_thread+0x255/0x440 [spl]
[  798.683667]  [<ffffffff810ac190>] ? wake_up_q+0x70/0x70
[  798.683671]  [<ffffffffc07bf5e0>] ? taskq_cancel_id+0x130/0x130 [spl]
[  798.683674]  [<ffffffff810a08d8>] kthread+0xd8/0xf0
[  798.683677]  [<ffffffff810a0800>] ? kthread_create_on_node+0x1e0/0x1e0
[  798.683680]  [<ffffffff81830a8f>] ret_from_fork+0x3f/0x70
[  798.683682]  [<ffffffff810a0800>] ? kthread_create_on_node+0x1e0/0x1e0
[  798.683685] Padding ffff88087b25f67a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.684143] Padding ffff88087b25f68a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.684600] Padding ffff88087b25f69a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.685054] Padding ffff88087b25f6aa: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.685513] Padding ffff88087b25f6ba: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.685966] Padding ffff88087b25f6ca: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.686422] Padding ffff88087b25f6da: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.686869] Padding ffff88087b25f6ea: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.687321] Padding ffff88087b25f6fa: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.687760] Padding ffff88087b25f70a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.688197] Padding ffff88087b25f71a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.688635] Padding ffff88087b25f72a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.689073] Padding ffff88087b25f73a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.689494] Padding ffff88087b25f74a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.689924] Padding ffff88087b25f75a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.690355] Padding ffff88087b25f76a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.690784] Padding ffff88087b25f77a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.691199] Padding ffff88087b25f78a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.691632] Padding ffff88087b25f79a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.692060] Padding ffff88087b25f7aa: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.692487] Padding ffff88087b25f7ba: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.692916] Padding ffff88087b25f7ca: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.693350] Padding ffff88087b25f7da: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.693764] Padding ffff88087b25f7ea: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.694199] Padding ffff88087b25f7fa: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.694628] Padding ffff88087b25f80a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.695061] Padding ffff88087b25f81a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.695496] Padding ffff88087b25f82a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.695922] Padding ffff88087b25f83a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.696362] Padding ffff88087b25f84a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.696799] Padding ffff88087b25f85a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.697225] Padding ffff88087b25f86a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.697663] Padding ffff88087b25f87a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.698101] Padding ffff88087b25f88a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.698525] Padding ffff88087b25f89a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.698957] Padding ffff88087b25f8aa: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.699389] Padding ffff88087b25f8ba: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.699824] Padding ffff88087b25f8ca: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.700248] Padding ffff88087b25f8da: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.700678] Padding ffff88087b25f8ea: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.701101] Padding ffff88087b25f8fa: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.701512] Padding ffff88087b25f90a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.701882] Padding ffff88087b25f91a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.702249] Padding ffff88087b25f92a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.702592] Padding ffff88087b25f93a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.702899] Padding ffff88087b25f94a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.703197] Padding ffff88087b25f95a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.703464] Padding ffff88087b25f96a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.703715] Padding ffff88087b25f97a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.703964] Padding ffff88087b25f98a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.704201] Padding ffff88087b25f99a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.704445] Padding ffff88087b25f9aa: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.704695] Padding ffff88087b25f9ba: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.704941] Padding ffff88087b25f9ca: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.705183] Padding ffff88087b25f9da: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.705430] Padding ffff88087b25f9ea: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.705682] Padding ffff88087b25f9fa: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.705920] Padding ffff88087b25fa0a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.706169] Padding ffff88087b25fa1a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.706420] Padding ffff88087b25fa2a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.706672] Padding ffff88087b25fa3a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.706910] Padding ffff88087b25fa4a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.707154] Padding ffff88087b25fa5a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.707400] Padding ffff88087b25fa6a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.707649] Padding ffff88087b25fa7a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.707893] Padding ffff88087b25fa8a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.708137] Padding ffff88087b25fa9a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.708385] Padding ffff88087b25faaa: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.708634] Padding ffff88087b25faba: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.708879] Padding ffff88087b25faca: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.709127] Padding ffff88087b25fada: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.709377] Padding ffff88087b25faea: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.709618] Padding ffff88087b25fafa: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.709868] Padding ffff88087b25fb0a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.710115] Padding ffff88087b25fb1a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.710357] Padding ffff88087b25fb2a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.710598] Padding ffff88087b25fb3a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.710848] Padding ffff88087b25fb4a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.711089] Padding ffff88087b25fb5a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.711325] Padding ffff88087b25fb6a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.711557] Padding ffff88087b25fb7a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.711803] Padding ffff88087b25fb8a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.712042] Padding ffff88087b25fb9a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.712289] Padding ffff88087b25fbaa: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.712534] Padding ffff88087b25fbba: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.712783] Padding ffff88087b25fbca: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.713019] Padding ffff88087b25fbda: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.713251] Padding ffff88087b25fbea: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.713496] Padding ffff88087b25fbfa: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.713729] Padding ffff88087b25fc0a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.713966] Padding ffff88087b25fc1a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.714214] Padding ffff88087b25fc2a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.714461] Padding ffff88087b25fc3a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.714706] Padding ffff88087b25fc4a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.714946] Padding ffff88087b25fc5a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.715190] Padding ffff88087b25fc6a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.715437] Padding ffff88087b25fc7a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.715674] Padding ffff88087b25fc8a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.715915] Padding ffff88087b25fc9a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.716156] Padding ffff88087b25fcaa: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.716405] Padding ffff88087b25fcba: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.716641] Padding ffff88087b25fcca: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.716885] Padding ffff88087b25fcda: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.717130] Padding ffff88087b25fcea: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.717377] Padding ffff88087b25fcfa: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.717621] Padding ffff88087b25fd0a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.717856] Padding ffff88087b25fd1a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.718097] Padding ffff88087b25fd2a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.718347] Padding ffff88087b25fd3a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.718595] Padding ffff88087b25fd4a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.718844] Padding ffff88087b25fd5a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.719093] Padding ffff88087b25fd6a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.719340] Padding ffff88087b25fd7a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.719574] Padding ffff88087b25fd8a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.719820] Padding ffff88087b25fd9a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.720068] Padding ffff88087b25fdaa: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.720304] Padding ffff88087b25fdba: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.720541] Padding ffff88087b25fdca: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.720777] Padding ffff88087b25fdda: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.721025] Padding ffff88087b25fdea: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.721273] Padding ffff88087b25fdfa: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.721522] Padding ffff88087b25fe0a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.721769] Padding ffff88087b25fe1a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.722016] Padding ffff88087b25fe2a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.722265] Padding ffff88087b25fe3a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.722515] Padding ffff88087b25fe4a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.722763] Padding ffff88087b25fe5a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.723010] Padding ffff88087b25fe6a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  798.723261] FIX zio_buf_8192: Restoring 0xffff88087b25f67a-0xffff88087b25fe79=0x5a
kernelOfTruth commented 8 years ago

@bendikro please post Hardware stats of your system,

ECC Ram ? Processor ? Motherboard ?

Disk controller ? harddrives ?

Pool configuration ? Special settings ?

/etc/modprobe.d/zfs.conf , spl.conf etc.

Besides

slub_debug

please also add

slub_nomerge

there's suspected memory corruption happening which might be caused by some driver malfunctioning (or hardware) - thus separating slabs can be somewhat of a countermeasure against that

ironMann commented 8 years ago

@bendikro Many different stack traces, and bit flips in slub padding (see below), point to faulty RAM on your system. Can you run memtest to rule that out first?

[  762.369392] Padding ffff8808022dff3a: 5a 5a 5a 5a 5a 5a 5a da 5a 5a 5a 5a 5a 5a 5a da  ZZZZZZZ.ZZZZZZZ.
[  762.369650] Padding ffff8808022dff4a: 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a 5a  ZZZZZZZZZZZZZZZZ
[  762.369906] Padding ffff8808022dff5a: 5a 5a 5a 5a 5a 5a 5a 4e 5a 5a 5a 5a 5a 5a 5a 4e  ZZZZZZZNZZZZZZZN
[  762.370161] Padding ffff8808022dff6a: 5a 5a 5a 5a 5a 5a 5a 4e 5a 5a 5a 5a 5a 5a 5a 4e  ZZZZZZZNZZZZZZZN
tuxoko commented 8 years ago

@bendikro Do the memtest as suggested by others. If memory is all good, please run slub_debug on original version. There might be different issues on different versions.

bendikro commented 8 years ago

Phew!, it turns out this project isn't as broken as I feared, it was a RAM issue :-)

After removing 3 RAM modules, leaving only 1 x 16GB (running memtest on that one gave no errors), recompiled 0.6.5.8, the file system was transferred successfully. Then recreated the pool and transferred all the the file systems successfully.

Thanks for the help and for the great work!

bendikro commented 8 years ago

@tuxoko Hey. I put all the RAM modules back in and ran memtest86, and after three passes there are no errors, so maybe it's not bad RAM after all, but some other issue (hardware or software) that only occurs with multiple RAM modules installed?

I've encountered issues when having many RAM modules installed before (on other computers), specifically in relation to BIOS settings and graphic drivers. Do you think this could still be a ZOL bug? If so, any suggestions on how to proceed debugging this?

I've a got a brand new computer with the follow hardware:

tuxoko commented 7 years ago

@bendikro Can you try again with all RAM inserted on 0.6.5.8 with slub_debug. My concern is that your first and third dump both have memory corruption inside dbuf. So there could be a legitimate bug inside ZFS given that your memtest result is clean.

bendikro commented 7 years ago

Hi, good news :-P

Since memtest86 didn't find any errors, I simply booted with the different RAM modules inserted and re-ran the zfs send/recv command. It turns out that with one specific RAM module inserted, the zfs file system transfer consistently fails with a kernel oops.

I then re-ran memtest86 with only that RAM module over night, and after 6 passes, it reported errors in a memory area.

Thanks again!