beren12
commented
6 years ago Did you say on IRC you were using encryption on this pool?
Closed morphinz closed 6 years ago
beren12
commented
6 years ago Did you say on IRC you were using encryption on this pool?
morphinz
commented
6 years ago @beren12 No. I don't use encryption on both system.
morphinz
commented
6 years ago Hello again. I upgraded A server with "0.7.6-1" and B server 0.7.6-1 & after that I try zfs-git Nothing changed.
I try to send with resume token and B server got panic again with same log..
After that I rollback the destination and I try to send an incremental with new snapshot on source.
B server got panic again with same log.
After that I run a scrub on destination:
pool: FKM1
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub repaired 0B in 0 days 19:01:45 with 0 errors on Mon Feb 26 14:22:26 2018(The future are encryption which I don't use, nothing important)
As you can see the pool have 0 issue.
Now I'm gonna run a scrub on source but this will take 2 day atleast so I need to wait Friday.
After scrub I will try again but if there is any way to get more log about the problem can you tell me please?
See you soon dear friends.
tcaputi
commented
6 years ago @kpande
fwiw I'm hitting a similar problem on master with encrypted datasets but it deadlocks so hard the audio playing (when it happens) skips forever and I do not get any stack traces.
Usually crashing like that is caused when the code hits a call to panic(). If you are using a GUI or ssh you can probably get a stack trace from the raw tty. Let me know if you can get a stack trace from this.
@morphinz and @kpande
This looks like either a double free or a buffer overrun, which are usually pretty hard to debug without being able to reproduce the problem. Fortunately, the spl has 2 build options (--enable-debug-kmem-tracking and --enable-debug-kmem, which can be provided to ./configure) which can help debug double frees. If one occurs, it will ASSERT right at the problem and provide useful info for debugging. Could you guys give that option a try (with the latest code from master if possible) and see what happens? If it is a buffer overrun then I will need to see if I can reproduce it myself.
tcaputi
commented
6 years ago @kpande. With the configure options there is a good chance it might not hard panic like that and you won't need the cable. You can also get to a raw terminal (not blocked by X) with Ctrl + alt + f5 (or any other f-key).
tcaputi
commented
6 years ago @kpande Was this comment meant to be made here? If so, that was not the kind of double free I was talking about. That one is an on-disk spacemap double free, while this one might be a memory double free.
morphinz
commented
6 years ago Sorry for the late answer.
I ran Scrub on both pool "repaired 0B - with 0 errors" After Scrub I try to send an incremental "I don't use resume token" and B server got panic again... Nothing changed!
After that I tried few weird things; Information: 1- The dataset named "xXx" 27T. 2- xXx has 1 snapshot on source and destination. This was the first Send-Revc snaphot. 3- If I take a new snapshot and try to send Incremental to B server, B server panics.
On A server "which is the source" I decide to try a FULL send-recv "In same pool".
zfs send -Rc aserver\xXx@FirstSnapshot | zfs recv aserver\xXx-new
Everything was Fine.
After that I tried to send Incremental, "aserver\xXx@SecondSnapshot | zfs recv aserver\xXx-new" A server got panic exactly like B server.
I tried to send Full "-Rc aserver\xXx@SecondSnapshot | zfs recv aserver\xXx-BRANDNEW" nothing changed, A server got panic when FirstSnapshot over and Second begin.
I'm confused. What is going on???
zfs rollback aserver\xXx-new@FirstSnapshot
dd if=/dev/zero of=/xXx-new/test count=100 bs=1G
zfs send -cI aserver\xXx-new@NewSnapshot | ssh Bserver zfs recv aserver\xXx
Gues what? Everything was JUST FINE... I didnt get panic.
This reminds me a simple question: What the heck is that?
IF I clone first snapshot and write some file on it I was able to RESUME. BUT when I try to take a snapshot on active dataset I cant resume it.. How is that possible? The change was only 970G on first snapshot to Second..
@kpande Yes I don't use encryption. I was trying zfs-git and the feature was encryption.
@tcaputi I have bigger dataset than the "xXx" but I only have the problem on 2 dataset. 1 of them 27t and the other was 80T.
But like I said I have different datasets too like "40T, 60T, 80T" and they don't have the problem.
tcaputi
commented
6 years ago @morphinz I'm a little confused at the moment. Does this issue only happen when you use a resume token? In your comment from yesterday you don't mention the resume token in your commands. When the kernel panics is the stack trace the same as the one you posted, or are you unable to get a stack trace at all?
@kpande any updates here?
morphinz
commented
6 years ago @tcaputi When I open the issue I was thinking this is all about with resume token but now %100 I'm sure its not. Also I don't use resume token anymore. I have the issue with or without resume token.
Also today I create a bookmark from that snapshot and I delete the snapshot because I was thinking the snapshot will cause the problem and deleting will be solve my problem but it did not.. I got same result and same stack trace.
As you can see at below my source is "Apool/xXx" and destination "Apool/xXx-testing" in same pool.
PS: "I changed pool and dataset name for easy understanding."
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD USERREFS CREATION
Apool/xXx 42.8T 28.2T 47.1M 28.2T 0B 0B - Fri Nov 3 0:48 2017
Apool/xXx#1516182586920REPL-bookmark - - - - - - - Wed Jan 17 12:49 2018
Apool/xXx@1520955317597REPL - 47.1M - - - - 1 Tue Mar 13 18:35 2018NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD USERREFS CREATION
Apool/xXx-testing 43.8T 27.2T 0B 27.2T 0B 0B - Thu Mar 8 21:33 2018
Apool/xXx-testing@1516182586920REPL - 0B - - - - 0 Wed Jan 17 12:49 2018zfs send -i Apool/xXx#1516182586920REPL-bookmark Apool/xXx@NewlyCreatedSNAP| zfs receive -sv Apool/xXx
And I got panic. If I got any free time I will compile zfs for debuging the issue and I will share my result.
morphinz
commented
6 years ago @tcaputi sorry for the outstanding delay. I was busy with other stuff. Also I thought I found a workaround for this problem and that bought me some time. However my workaround failed and this is a top priority for me again :(
Since it's been a while I've setup a fresh new platform in order to make the problem perfectly reproduciable. I have tried a couple of distributions, kernel and zfs versions. Finally I've compiled the latest spl bits with DEBUG parameters (--enable-debug-kmem-tracking and --enable-debug-kmem).
First let me share my various distro and version results.
zfs receive -v fkmmedium/images < /fkmmedium/aai-8mart2130.zvolreceiving incremental stream of clspool/images@8mart2130 into fkmmedium/images@8mart2130Mar 24 21:09:20 FKM2 zed[19821]: eid=12 class=history_event pool_guid=0x2F6EE146278F0CC1
Mar 24 21:09:22 FKM2 kernel: general protection fault: 0000 [#1] SMP NOPTI
Mar 24 21:09:22 FKM2 kernel: Modules linked in: msr dm_service_time dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua iTCO_wdt iTCO_vendor_support intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr mxm_wmi ttm drm_kms_helper ipmi_ssif drm 8021q evdev mrp joydev agpgart syscopyarea mousedev input_leds sysfillrect igb led_class mei_me sysimgblt mac_hid fb_sys_fops i2c_i801 mei lpc_ich i2c_algo_bit ioatdma ixgbe mdio ptp pps_core dca shpchp ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter button acpi_pad bonding sch_fq_codel ip_tables x_tables mptsas mptscsih mptbase hpsa hid_generic usbhid zfs(PO) zunicode(PO) zavl(PO)
Mar 24 21:09:22 FKM2 kernel: icp(PO) ses enclosure zcommon(PO) znvpair(PO) spl(O) sd_mod mpt3sas raid_class scsi_transport_sas megaraid_sas ahci libahci libata virtio_scsi virtio_ring virtio vmw_vsock_vmci_transport vsock vmw_pvscsi scsi_mod vmw_vmci hid xhci_pci ehci_pci ehci_hcd xhci_hcd usbcore usb_common
Mar 24 21:09:22 FKM2 kernel: CPU: 7 PID: 19823 Comm: receive_writer Tainted: P O 4.14.29-1-lts #1
Mar 24 21:09:22 FKM2 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Mar 24 21:09:22 FKM2 kernel: task: ffff9b1cb7256580 task.stack: ffffb2fa6a9ec000
Mar 24 21:09:22 FKM2 kernel: RIP: 0010:__kmalloc_node+0x15e/0x2a0
Mar 24 21:09:22 FKM2 kernel: RSP: 0018:ffffb2fa6a9efce0 EFLAGS: 00010282
Mar 24 21:09:22 FKM2 kernel: RAX: 0000000000000000 RBX: 92558eb84083930e RCX: 0000000000604cd4
Mar 24 21:09:22 FKM2 kernel: RDX: 0000000000604cd3 RSI: 0000000000000000 RDI: 00000000000226a0
Mar 24 21:09:22 FKM2 kernel: RBP: ffff9adc88b9dc00 R08: ffff9adc88b9dc00 R09: ffff9adcbf407000
Mar 24 21:09:22 FKM2 kernel: R10: 6daa1464c83a4f0e R11: ffffffffc02b4097 R12: 000000000140c200
Mar 24 21:09:22 FKM2 kernel: R13: 0000000000000158 R14: 00000000ffffffff R15: ffff9adcbf407000
Mar 24 21:09:22 FKM2 kernel: FS: 0000000000000000(0000) GS:ffff9adcbfbc0000(0000) knlGS:0000000000000000
Mar 24 21:09:22 FKM2 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 24 21:09:22 FKM2 kernel: CR2: 00007fd448a31eb0 CR3: 00000014a700a002 CR4: 00000000003606e0
Mar 24 21:09:22 FKM2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 24 21:09:22 FKM2 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Mar 24 21:09:22 FKM2 kernel: Call Trace:
Mar 24 21:09:22 FKM2 kernel: ? spl_kmem_alloc+0xd0/0x180 [spl]
Mar 24 21:09:22 FKM2 kernel: spl_kmem_zalloc+0xc7/0x180 [spl]
Mar 24 21:09:22 FKM2 kernel: dbuf_dirty+0x1ff/0x830 [zfs]
Mar 24 21:09:22 FKM2 kernel: receive_object+0x248/0x460 [zfs]
Mar 24 21:09:22 FKM2 kernel: ? dmu_free_long_range+0x35b/0x440 [zfs]
Mar 24 21:09:22 FKM2 kernel: receive_writer_thread+0x580/0x9c0 [zfs]
Mar 24 21:09:22 FKM2 kernel: ? set_curr_task_fair+0x29/0x60
Mar 24 21:09:22 FKM2 kernel: ? set_user_nice.part.71+0x14e/0x240
Mar 24 21:09:22 FKM2 kernel: ? receive_free.isra.13+0xb0/0xb0 [zfs]
Mar 24 21:09:22 FKM2 kernel: ? __thread_exit+0x20/0x20 [spl]
Mar 24 21:09:22 FKM2 kernel: thread_generic_wrapper+0x6f/0x80 [spl]
Mar 24 21:09:22 FKM2 kernel: kthread+0x11a/0x130
Mar 24 21:09:22 FKM2 kernel: ? kthread_create_on_node+0x70/0x70
Mar 24 21:09:22 FKM2 kernel: ret_from_fork+0x35/0x40
Mar 24 21:09:22 FKM2 kernel: Code: 8b 10 4d 33 97 40 01 00 00 4c 89 c3 4c 31 d3 65 48 0f c7 0f 0f 94 c0 84 c0 0f 84 ff fe ff ff 4d 39 d0 74 14 49 63 47 20 48 01 c3 <48> 33 1b 49 33 9f 40 01 00 00 0f 18 0b 41 f7 c4 00 80 00 00 0f
Mar 24 21:09:22 FKM2 kernel: RIP: __kmalloc_node+0x15e/0x2a0 RSP: ffffb2fa6a9efce0
Mar 24 21:09:22 FKM2 kernel: ---[ end trace 5ee89471431f04fb ]---
Mar 24 21:09:27 FKM2 kernel: general protection fault: 0000 [#2] SMP NOPTI
Mar 24 21:09:27 FKM2 kernel: Modules linked in: msr dm_service_time dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua iTCO_wdt iTCO_vendor_support intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr mxm_wmi ttm drm_kms_helper ipmi_ssif drm 8021q evdev mrp joydev agpgart syscopyarea mousedev input_leds sysfillrect igb led_class mei_me sysimgblt mac_hid fb_sys_fops i2c_i801 mei lpc_ich i2c_algo_bit ioatdma ixgbe mdio ptp pps_core dca shpchp ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter button acpi_pad bonding sch_fq_codel ip_tables x_tables mptsas mptscsih mptbase hpsa hid_generic usbhid zfs(PO) zunicode(PO) zavl(PO)
Mar 24 21:09:27 FKM2 kernel: icp(PO) ses enclosure zcommon(PO) znvpair(PO) spl(O) sd_mod mpt3sas raid_class scsi_transport_sas megaraid_sas ahci libahci libata virtio_scsi virtio_ring virtio vmw_vsock_vmci_transport vsock vmw_pvscsi scsi_mod vmw_vmci hid xhci_pci ehci_pci ehci_hcd xhci_hcd usbcore usb_common
Mar 24 21:09:27 FKM2 kernel: CPU: 7 PID: 3504 Comm: txg_sync Tainted: P D O 4.14.29-1-lts #1
Mar 24 21:09:27 FKM2 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Mar 24 21:09:27 FKM2 kernel: task: ffff9adcadb52b80 task.stack: ffffb2fa21520000
Mar 24 21:09:27 FKM2 kernel: RIP: 0010:__kmalloc_node+0x132/0x2a0
Mar 24 21:09:27 FKM2 kernel: RSP: 0018:ffffb2fa21523bf8 EFLAGS: 00010282
Mar 24 21:09:27 FKM2 kernel: RAX: 92558eb84083930e RBX: 000000000140c200 RCX: 0000000000604cd5
Mar 24 21:09:27 FKM2 kernel: RDX: 0000000000604cd4 RSI: 0000000000000000 RDI: 00000000000226a0
Mar 24 21:09:27 FKM2 kernel: RBP: 92558eb84083930e R08: 92558eb84083930e R09: ffff9adcbf407000
Mar 24 21:09:27 FKM2 kernel: R10: 0000000000001800 R11: ffffffffc02b4097 R12: 000000000140c200
Mar 24 21:09:27 FKM2 kernel: R13: 0000000000000158 R14: 00000000ffffffff R15: ffff9adcbf407000
Mar 24 21:09:27 FKM2 kernel: FS: 0000000000000000(0000) GS:ffff9adcbfbc0000(0000) knlGS:0000000000000000
Mar 24 21:09:27 FKM2 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 24 21:09:27 FKM2 kernel: CR2: 00007fd448a31eb0 CR3: 00000014a700a002 CR4: 00000000003606e0
Mar 24 21:09:27 FKM2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 24 21:09:27 FKM2 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Mar 24 21:09:27 FKM2 kernel: Call Trace:
Mar 24 21:09:27 FKM2 kernel: ? update_load_avg+0x41d/0x580
Mar 24 21:09:27 FKM2 kernel: spl_kmem_zalloc+0xc7/0x180 [spl]
Mar 24 21:09:27 FKM2 kernel: dbuf_dirty+0x1ff/0x830 [zfs]
Mar 24 21:09:27 FKM2 kernel: dsl_dataset_sync+0x21/0x220 [zfs]
Mar 24 21:09:27 FKM2 kernel: dsl_pool_sync+0xa1/0x420 [zfs]
Mar 24 21:09:27 FKM2 kernel: spa_sync+0x424/0xcf0 [zfs]
Mar 24 21:09:27 FKM2 kernel: txg_sync_thread+0x2cb/0x490 [zfs]
Mar 24 21:09:27 FKM2 kernel: ? txg_delay+0x1b0/0x1b0 [zfs]
Mar 24 21:09:27 FKM2 kernel: ? __thread_exit+0x20/0x20 [spl]
Mar 24 21:09:27 FKM2 kernel: thread_generic_wrapper+0x6f/0x80 [spl]
Mar 24 21:09:27 FKM2 kernel: kthread+0x11a/0x130
Mar 24 21:09:27 FKM2 kernel: ? kthread_create_on_node+0x70/0x70
Mar 24 21:09:27 FKM2 kernel: ret_from_fork+0x35/0x40
Mar 24 21:09:27 FKM2 kernel: Code: 00 49 89 c7 4c 8b 5c 24 08 4c 8b 0c 24 4d 85 ff 0f 85 34 ff ff ff 31 ed eb 5c 4d 63 47 20 49 8b 3f 48 8d 4a 01 48 89 e8 49 01 e8 <4d> 8b 10 4d 33 97 40 01 00 00 4c 89 c3 4c 31 d3 65 48 0f c7 0f
Mar 24 21:09:27 FKM2 kernel: RIP: __kmalloc_node+0x132/0x2a0 RSP: ffffb2fa21523bf8
Mar 24 21:09:27 FKM2 kernel: ---[ end trace 5ee89471431f04fc ]---zfs receive -v fkmmedium/images < /fkmmedium/aai-8mart2130.zvolreceiving incremental stream of clspool/images@8mart2130 into fkmmedium/images@8mart2130 **Segmentation fault**Mar 24 21:40:56 FKM2 zed[29809]: eid=12 class=history_event pool_guid=0x2F6EE146278F0CC1
Mar 24 21:40:57 FKM2 kernel: general protection fault: 0000 [#1] SMP NOPTI
Mar 24 21:40:57 FKM2 kernel: Modules linked in: msr dm_service_time dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua iTCO_wdt iTCO_vendor_support intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr mxm_wmi 8021q mrp ipmi_ssif ttm drm_kms_helper drm evdev joydev input_leds agpgart syscopyarea mousedev mei_me igb sysfillrect led_class sysimgblt fb_sys_fops mac_hid mei lpc_ich i2c_i801 i2c_algo_bit ioatdma ixgbe mdio ptp pps_core dca shpchp ipmi_si ipmi_devintf ipmi_msghandler wmi bonding acpi_power_meter button acpi_pad sch_fq_codel ip_tables x_tables mptsas mptscsih mptbase hpsa hid_generic usbhid zfs(PO) zunicode(PO) zlua(PO)
Mar 24 21:40:57 FKM2 kernel: ses enclosure zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) sd_mod mpt3sas raid_class scsi_transport_sas megaraid_sas ahci libahci libata virtio_scsi virtio_ring virtio vmw_vsock_vmci_transport vsock vmw_pvscsi scsi_mod vmw_vmci hid xhci_pci ehci_pci ehci_hcd xhci_hcd usbcore usb_common
Mar 24 21:40:57 FKM2 kernel: CPU: 21 PID: 29811 Comm: receive_writer Tainted: P O 4.14.29-1-lts #1
Mar 24 21:40:57 FKM2 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Mar 24 21:40:57 FKM2 kernel: task: ffff8b727dfd0e80 task.stack: ffffa9603e510000
Mar 24 21:40:57 FKM2 kernel: RIP: 0010:__kmalloc_node+0x15e/0x2a0
Mar 24 21:40:57 FKM2 kernel: RSP: 0018:ffffa9603e513c78 EFLAGS: 00010202
Mar 24 21:40:57 FKM2 kernel: RAX: 0000000000000000 RBX: 402d0c9ea1f45245 RCX: 00000000000038fd
Mar 24 21:40:57 FKM2 kernel: RDX: 00000000000038fc RSI: 0000000000000000 RDI: 00000000000226a0
Mar 24 21:40:57 FKM2 kernel: RBP: ffff8b7278567e00 R08: ffff8b7278567e00 R09: ffff8b72ff407000
Mar 24 21:40:57 FKM2 kernel: R10: bfd287ecd9a22c45 R11: ffffffffc03d8097 R12: 000000000140c200
Mar 24 21:40:57 FKM2 kernel: R13: 0000000000000158 R14: 00000000ffffffff R15: ffff8b72ff407000
Mar 24 21:40:57 FKM2 kernel: FS: 0000000000000000(0000) GS:ffff8b72ffd40000(0000) knlGS:0000000000000000
Mar 24 21:40:57 FKM2 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 24 21:40:57 FKM2 kernel: CR2: 00007f2c12bbe020 CR3: 0000005eac00a005 CR4: 00000000003606e0
Mar 24 21:40:57 FKM2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 24 21:40:57 FKM2 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Mar 24 21:40:57 FKM2 kernel: Call Trace:
Mar 24 21:40:57 FKM2 kernel: ? spl_kmem_alloc+0xd0/0x180 [spl]
Mar 24 21:40:57 FKM2 kernel: spl_kmem_zalloc+0xc7/0x180 [spl]
Mar 24 21:40:57 FKM2 kernel: dbuf_dirty+0x1ff/0x840 [zfs]
Mar 24 21:40:57 FKM2 kernel: receive_object+0x576/0x920 [zfs]
Mar 24 21:40:57 FKM2 kernel: ? dnode_rele_and_unlock+0x4e/0x80 [zfs]
Mar 24 21:40:57 FKM2 kernel: receive_writer_thread+0x610/0xb60 [zfs]
Mar 24 21:40:57 FKM2 kernel: ? set_curr_task_fair+0x29/0x60
Mar 24 21:40:57 FKM2 kernel: ? receive_freeobjects.isra.14+0x110/0x110 [zfs]
Mar 24 21:40:57 FKM2 kernel: ? __thread_exit+0x20/0x20 [spl]
Mar 24 21:40:57 FKM2 kernel: ? thread_generic_wrapper+0x6f/0x80 [spl]
Mar 24 21:40:57 FKM2 kernel: ? receive_freeobjects.isra.14+0x110/0x110 [zfs]
Mar 24 21:40:57 FKM2 kernel: thread_generic_wrapper+0x6f/0x80 [spl]
Mar 24 21:40:57 FKM2 kernel: kthread+0x11a/0x130
Mar 24 21:40:57 FKM2 kernel: ? kthread_create_on_node+0x70/0x70
Mar 24 21:40:57 FKM2 kernel: ret_from_fork+0x35/0x40
Mar 24 21:40:57 FKM2 kernel: Code: 8b 10 4d 33 97 40 01 00 00 4c 89 c3 4c 31 d3 65 48 0f c7 0f 0f 94 c0 84 c0 0f 84 ff fe ff ff 4d 39 d0 74 14 49 63 47 20 48 01 c3 <48> 33 1b 49 33 9f 40 01 00 00 0f 18 0b 41 f7 c4 00 80 00 00 0f
Mar 24 21:40:57 FKM2 kernel: RIP: __kmalloc_node+0x15e/0x2a0 RSP: ffffa9603e513c78
Mar 24 21:40:57 FKM2 kernel: ---[ end trace a3b314cdabd6aff2 ]---
Mar 24 21:41:01 FKM2 kernel: general protection fault: 0000 [#2] SMP NOPTI
Mar 24 21:41:01 FKM2 kernel: Modules linked in: msr dm_service_time dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua iTCO_wdt iTCO_vendor_support intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr mxm_wmi 8021q mrp ipmi_ssif ttm drm_kms_helper drm evdev joydev input_leds agpgart syscopyarea mousedev mei_me igb sysfillrect led_class sysimgblt fb_sys_fops mac_hid mei lpc_ich i2c_i801 i2c_algo_bit ioatdma ixgbe mdio ptp pps_core dca shpchp ipmi_si ipmi_devintf ipmi_msghandler wmi bonding acpi_power_meter button acpi_pad sch_fq_codel ip_tables x_tables mptsas mptscsih mptbase hpsa hid_generic usbhid zfs(PO) zunicode(PO) zlua(PO)
Mar 24 21:41:01 FKM2 kernel: ses enclosure zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) sd_mod mpt3sas raid_class scsi_transport_sas megaraid_sas ahci libahci libata virtio_scsi virtio_ring virtio vmw_vsock_vmci_transport vsock vmw_pvscsi scsi_mod vmw_vmci hid xhci_pci ehci_pci ehci_hcd xhci_hcd usbcore usb_common
Mar 24 21:41:01 FKM2 kernel: CPU: 21 PID: 29643 Comm: zfs Tainted: P D O 4.14.29-1-lts #1
Mar 24 21:41:01 FKM2 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Mar 24 21:41:01 FKM2 kernel: task: ffff8b72d7c2d700 task.stack: ffffa960311a0000
Mar 24 21:41:01 FKM2 kernel: RIP: 0010:__kmalloc_node+0x132/0x2a0
Mar 24 21:41:01 FKM2 kernel: RSP: 0018:ffffa960311a3140 EFLAGS: 00010202
Mar 24 21:41:01 FKM2 kernel: RAX: 402d0c9ea1f45245 RBX: ffff8bb2ec444b60 RCX: 00000000000038fe
Mar 24 21:41:01 FKM2 kernel: RDX: 00000000000038fd RSI: 0000000000000000 RDI: 00000000000226a0
Mar 24 21:41:01 FKM2 kernel: RBP: 402d0c9ea1f45245 R08: 402d0c9ea1f45245 R09: ffff8b72ff407000
Mar 24 21:41:01 FKM2 kernel: R10: ffffa960311a3188 R11: ffffffff8b3011ab R12: 0000000001011200
Mar 24 21:41:01 FKM2 kernel: R13: 00000000000001d0 R14: 00000000ffffffff R15: ffff8b72ff407000
Mar 24 21:41:01 FKM2 kernel: FS: 00007f89dd8dc7c0(0000) GS:ffff8b72ffd40000(0000) knlGS:0000000000000000
Mar 24 21:41:01 FKM2 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar 24 21:41:01 FKM2 kernel: CR2: 00007f2c12bbe020 CR3: 0000007f3fe56004 CR4: 00000000003606e0
Mar 24 21:41:01 FKM2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Mar 24 21:41:01 FKM2 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Mar 24 21:41:01 FKM2 kernel: Call Trace:
Mar 24 21:41:01 FKM2 kernel: ? wait_woken+0x80/0x80
Mar 24 21:41:01 FKM2 kernel: ? wait_woken+0x80/0x80
Mar 24 21:41:01 FKM2 kernel: alloc_request_size+0x2b/0x70
Mar 24 21:41:01 FKM2 kernel: mempool_alloc+0x65/0x190
Mar 24 21:41:01 FKM2 kernel: ? dm_any_congested+0x60/0x60 [dm_mod]
Mar 24 21:41:01 FKM2 kernel: get_request+0x3b6/0x770
Mar 24 21:41:01 FKM2 kernel: ? wait_woken+0x80/0x80
Mar 24 21:41:01 FKM2 kernel: blk_queue_bio+0x114/0x400
Mar 24 21:41:01 FKM2 kernel: generic_make_request+0x123/0x300
Mar 24 21:41:01 FKM2 kernel: ? submit_bio+0x6c/0x140
Mar 24 21:41:01 FKM2 kernel: ? bio_alloc_bioset+0xc7/0x1e0
Mar 24 21:41:01 FKM2 kernel: submit_bio+0x6c/0x140
Mar 24 21:41:01 FKM2 kernel: vdev_disk_io_start+0x4f5/0x7e0 [zfs]
Mar 24 21:41:01 FKM2 kernel: ? zio_vdev_io_start+0x8d/0x2d0 [zfs]
Mar 24 21:41:01 FKM2 kernel: zio_vdev_io_start+0x8d/0x2d0 [zfs]
Mar 24 21:41:01 FKM2 kernel: zio_nowait+0xa3/0x140 [zfs]
Mar 24 21:41:01 FKM2 kernel: vdev_raidz_io_start+0x145/0x2c0 [zfs]
Mar 24 21:41:01 FKM2 kernel: zio_vdev_io_start+0x8d/0x2d0 [zfs]
Mar 24 21:41:01 FKM2 kernel: ? vdev_config_sync+0x190/0x190 [zfs]
Mar 24 21:41:01 FKM2 kernel: zio_nowait+0xa3/0x140 [zfs]
Mar 24 21:41:01 FKM2 kernel: vdev_mirror_io_start+0x92/0x180 [zfs]
Mar 24 21:41:01 FKM2 kernel: ? spa_config_enter+0xb2/0x100 [zfs]
Mar 24 21:41:01 FKM2 kernel: zio_vdev_io_start+0x17c/0x2d0 [zfs]
Mar 24 21:41:01 FKM2 kernel: ? tsd_get_by_thread+0x2a/0x40 [spl]
Mar 24 21:41:01 FKM2 kernel: ? taskq_member+0x14/0x20 [spl]
Mar 24 21:41:01 FKM2 kernel: zio_nowait+0xa3/0x140 [zfs]
Mar 24 21:41:01 FKM2 kernel: arc_read+0x8a2/0xf20 [zfs]
Mar 24 21:41:01 FKM2 kernel: ? dbuf_rele_and_unlock+0x510/0x510 [zfs]
Mar 24 21:41:01 FKM2 kernel: ? zio_null+0x2b/0x30 [zfs]
Mar 24 21:41:01 FKM2 kernel: dbuf_read+0x26b/0xb10 [zfs]
Mar 24 21:41:01 FKM2 kernel: dnode_hold_impl+0x140/0xc40 [zfs]
Mar 24 21:41:01 FKM2 kernel: dmu_object_info+0x2d/0x80 [zfs]
Mar 24 21:41:01 FKM2 kernel: dmu_recv_stream+0x8da/0xfb0 [zfs]
Mar 24 21:41:01 FKM2 kernel: ? nvlist_free+0x7b/0xb0 [znvpair]
Mar 24 21:41:01 FKM2 kernel: zfs_ioc_recv_impl+0x576/0x1010 [zfs]
Mar 24 21:41:01 FKM2 kernel: ? nvs_native_nvp_op+0x1d0/0x1d0 [znvpair]
Mar 24 21:41:01 FKM2 kernel: zfs_ioc_recv+0x1d3/0x300 [zfs]
Mar 24 21:41:01 FKM2 kernel: ? spa_name_compare+0xa/0x20 [zfs]
Mar 24 21:41:01 FKM2 kernel: ? avl_find+0x58/0x90 [zavl]
Mar 24 21:41:01 FKM2 kernel: ? spa_lookup+0xad/0xd0 [zfs]
Mar 24 21:41:01 FKM2 kernel: ? cap_inode_getsecurity+0x230/0x230
Mar 24 21:41:01 FKM2 kernel: ? security_capable+0x47/0x60
Mar 24 21:41:01 FKM2 kernel: ? cap_inode_getsecurity+0x230/0x230
Mar 24 21:41:01 FKM2 kernel: ? security_capable+0x47/0x60
Mar 24 21:41:01 FKM2 kernel: ? __kmalloc+0x17e/0x200
Mar 24 21:41:01 FKM2 kernel: ? priv_policy.isra.3.part.4+0xd/0x20 [zfs]
Mar 24 21:41:01 FKM2 kernel: zfsdev_ioctl+0x1e1/0x650 [zfs]
Mar 24 21:41:01 FKM2 kernel: do_vfs_ioctl+0xa4/0x630
Mar 24 21:41:01 FKM2 kernel: ? handle_mm_fault+0xaa/0x1e0
Mar 24 21:41:01 FKM2 kernel: SyS_ioctl+0x74/0x80
Mar 24 21:41:01 FKM2 kernel: do_syscall_64+0x67/0x120
Mar 24 21:41:01 FKM2 kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2
Mar 24 21:41:01 FKM2 kernel: RIP: 0033:0x7f89dc960d87
Mar 24 21:41:01 FKM2 kernel: RSP: 002b:00007ffcbdc751e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Mar 24 21:41:01 FKM2 kernel: RAX: ffffffffffffffda RBX: 00005606678d02d0 RCX: 00007f89dc960d87
Mar 24 21:41:01 FKM2 kernel: RDX: 00007ffcbdc75390 RSI: 0000000000005a1b RDI: 0000000000000005
Mar 24 21:41:01 FKM2 kernel: RBP: 00007ffcbdc75390 R08: 0000000000000000 R09: 00007f89dcc25b20
Mar 24 21:41:01 FKM2 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 00007ffcbdc83c88
Mar 24 21:41:01 FKM2 kernel: R13: 00007ffcbdc7f790 R14: 00007ffcbdc7d090 R15: 0000000000000000
Mar 24 21:41:01 FKM2 kernel: Code: 00 49 89 c7 4c 8b 5c 24 08 4c 8b 0c 24 4d 85 ff 0f 85 34 ff ff ff 31 ed eb 5c 4d 63 47 20 49 8b 3f 48 8d 4a 01 48 89 e8 49 01 e8 <4d> 8b 10 4d 33 97 40 01 00 00 4c 89 c3 4c 31 d3 65 48 0f c7 0f
Mar 24 21:41:01 FKM2 kernel: RIP: __kmalloc_node+0x132/0x2a0 RSP: ffffa960311a3140
Mar 24 21:41:01 FKM2 kernel: ---[ end trace a3b314cdabd6aff3 ]---zfs receive -v fkmmedium/images < /fkmmedium/aai-8mart2130.zvol receiving incremental stream of clspool/images@8mart2130 into fkmmedium/images@8mart2130
zfs receive -v fkmmedium/images < /fkmmedium/aai-8mart2130.zvol receiving incremental stream of clspool/images@8mart2130 into fkmmedium/images@8mart2130Nis 04 13:43:47 FKM2 kernel: general protection fault: 0000 [#1] SMP PTI
Nis 04 13:43:47 FKM2 kernel: Modules linked in: zfs(PO) zunicode(PO) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) fuse btrfs xor zstd_decompress zstd_compress xxhash raid6_pq ufs hfsplus hfs vfat msdos fat ext4 mbcache jbd2 dm_service_time ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack bonding ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support mxm_wmi sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
Nis 04 13:43:47 FKM2 kernel: pcbc aesni_intel crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr joydev input_leds ses enclosure sg i2c_i801 lpc_ich mei_me mei mfd_core ioatdma wmi shpchp ipmi_si ipmi_devintf acpi_power_meter ipmi_msghandler acpi_pad dm_multipath ip_tables xfs libcrc32c sd_mod crc32c_intel ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci drm ixgbe libata igb mpt3sas mdio ptp raid_class i2c_algo_bit pps_core dca scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod dax
Nis 04 13:43:48 FKM2 kernel: CPU: 4 PID: 10367 Comm: receive_writer Tainted: P O 4.16.0-1.el7.elrepo.x86_64 #1
Nis 04 13:43:48 FKM2 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Nis 04 13:43:48 FKM2 kernel: RIP: 0010:prefetch_freepointer.isra.65+0x11/0x20
Nis 04 13:43:48 FKM2 kernel: RSP: 0018:ffffc9000e0b7c30 EFLAGS: 00010286
Nis 04 13:43:48 FKM2 kernel: RAX: 0000000000000000 RBX: 94d28d9066885195 RCX: 0000000000014e19
Nis 04 13:43:48 FKM2 kernel: RDX: 0000000000014e18 RSI: 94d28d9066885195 RDI: ffff883f7f407020
Nis 04 13:43:48 FKM2 kernel: RBP: ffffc9000e0b7c30 R08: ffff883f7fb25f60 R09: ffff883f6e2d0200
Nis 04 13:43:48 FKM2 kernel: R10: ffff883f7f407000 R11: ffffffffa07ff1bb R12: 000000000140c200
Nis 04 13:43:48 FKM2 kernel: R13: 0000000000000158 R14: 00000000ffffffff R15: ffff883f7f407000
Nis 04 13:43:48 FKM2 kernel: FS: 0000000000000000(0000) GS:ffff883f7fb00000(0000) knlGS:0000000000000000
Nis 04 13:43:48 FKM2 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nis 04 13:43:48 FKM2 kernel: CR2: 00000000025eaf8c CR3: 000000000220a002 CR4: 00000000003606e0
Nis 04 13:43:48 FKM2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nis 04 13:43:48 FKM2 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nis 04 13:43:48 FKM2 kernel: Call Trace:
Nis 04 13:43:48 FKM2 kernel: __kmalloc_node+0x1d0/0x280
Nis 04 13:43:48 FKM2 kernel: spl_kmem_zalloc+0xbb/0x160 [spl]
Nis 04 13:43:48 FKM2 kernel: dbuf_dirty+0x1ea/0x860 [zfs]
Nis 04 13:43:48 FKM2 kernel: ? dnode_rele_and_unlock+0x53/0x90 [zfs]
Nis 04 13:43:48 FKM2 kernel: dmu_buf_will_dirty+0x11c/0x130 [zfs]
Nis 04 13:43:48 FKM2 kernel: receive_object+0x379/0x430 [zfs]
Nis 04 13:43:48 FKM2 kernel: receive_writer_thread+0x289/0xa30 [zfs]
Nis 04 13:43:48 FKM2 kernel: ? set_next_entity+0x93/0xe0
Nis 04 13:43:48 FKM2 kernel: ? do_dump+0x980/0x980 [zfs]
Nis 04 13:43:48 FKM2 kernel: ? spl_kmem_free+0x2a/0x40 [spl]
Nis 04 13:43:48 FKM2 kernel: ? do_dump+0x980/0x980 [zfs]
Nis 04 13:43:48 FKM2 kernel: thread_generic_wrapper+0x74/0x90 [spl]
Nis 04 13:43:48 FKM2 kernel: kthread+0x105/0x140
Nis 04 13:43:48 FKM2 kernel: ? __thread_exit+0x20/0x20 [spl]
Nis 04 13:43:48 FKM2 kernel: ? kthread_bind+0x20/0x20
Nis 04 13:43:48 FKM2 kernel: ret_from_fork+0x35/0x40
Nis 04 13:43:48 FKM2 kernel: Code: 31 d2 e8 a3 ea ff ff 5b 41 5c 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 85 f6 48 89 e5 74 0a 48 63 07 <48> 8b 04 06 0f 18 08 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 48
Nis 04 13:43:48 FKM2 kernel: RIP: prefetch_freepointer.isra.65+0x11/0x20 RSP: ffffc9000e0b7c30
Nis 04 13:43:48 FKM2 kernel: ---[ end trace d9616e2c3c7ce6f2 ]---And finally here is the results of the latest bits of spl/zfs with SPL DEBUG flags.
zfs receive -v fkmmedium/images < /fkmmedium/aai-8mart2130.zvolreceiving incremental stream of clspool/images@8mart2130 into fkmmedium/images@8mart2130Nis 06 02:29:34 FKM2 kernel: general protection fault: 0000 [#1] SMP PTI
Nis 06 02:29:34 FKM2 kernel: Modules linked in: zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) ip6t_rpfilter bonding ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack dm_service_time ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support mxm_wmi sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr joydev input_leds
Nis 06 02:29:34 FKM2 kernel: mei_me lpc_ich i2c_i801 mei ses mfd_core enclosure sg ioatdma ipmi_si ipmi_devintf wmi acpi_power_meter ipmi_msghandler shpchp acpi_pad dm_multipath binfmt_misc ip_tables xfs libcrc32c sd_mod crc32c_intel ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci ttm libahci drm ixgbe igb libata mdio mpt3sas ptp raid_class pps_core i2c_algo_bit dca scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod dax
Nis 06 02:29:34 FKM2 kernel: CPU: 11 PID: 24929 Comm: receive_writer Tainted: P O 4.16.0-1.el7.elrepo.x86_64 #1
Nis 06 02:29:34 FKM2 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Nis 06 02:29:34 FKM2 kernel: RIP: 0010:prefetch_freepointer.isra.65+0x11/0x20
Nis 06 02:29:34 FKM2 kernel: RSP: 0018:ffffc9000f1b7be8 EFLAGS: 00010286
Nis 06 02:29:34 FKM2 kernel: RAX: 0000000000000000 RBX: 94d28d9066885195 RCX: 0000000000006ec7
Nis 06 02:29:34 FKM2 kernel: RDX: 0000000000006ec6 RSI: 94d28d9066885195 RDI: ffff883f7f407020
Nis 06 02:29:34 FKM2 kernel: RBP: ffffc9000f1b7be8 R08: ffff887f7f2e5f60 R09: ffff887eaf5c0400
Nis 06 02:29:34 FKM2 kernel: R10: ffff883f7f407000 R11: ffffffffa068570f R12: 000000000140c200
Nis 06 02:29:34 FKM2 kernel: R13: 0000000000000158 R14: 00000000ffffffff R15: ffff883f7f407000
Nis 06 02:29:34 FKM2 kernel: FS: 0000000000000000(0000) GS:ffff887f7f2c0000(0000) knlGS:0000000000000000
Nis 06 02:29:34 FKM2 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nis 06 02:29:34 FKM2 kernel: CR2: 00007f3520024048 CR3: 000000000220a004 CR4: 00000000003606e0
Nis 06 02:29:34 FKM2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Nis 06 02:29:34 FKM2 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Nis 06 02:29:34 FKM2 kernel: Call Trace:
Nis 06 02:29:34 FKM2 kernel: __kmalloc_node+0x1d0/0x280
Nis 06 02:29:34 FKM2 kernel: spl_kmem_zalloc+0x22f/0x320 [spl]
Nis 06 02:29:34 FKM2 kernel: dbuf_dirty+0x1ea/0x860 [zfs]
Nis 06 02:29:34 FKM2 kernel: ? dnode_rele_and_unlock+0x53/0x90 [zfs]
Nis 06 02:29:34 FKM2 kernel: dmu_buf_will_dirty_impl+0x11d/0x130 [zfs]
Nis 06 02:29:34 FKM2 kernel: dmu_buf_will_dirty+0x16/0x20 [zfs]
Nis 06 02:29:34 FKM2 kernel: receive_object+0x4f9/0x900 [zfs]
Nis 06 02:29:34 FKM2 kernel: ? kfree+0x125/0x170
Nis 06 02:29:34 FKM2 kernel: ? _cond_resched+0x19/0x30
Nis 06 02:29:34 FKM2 kernel: ? mutex_lock+0x12/0x40
Nis 06 02:29:34 FKM2 kernel: receive_writer_thread+0x40d/0xb60 [zfs]
Nis 06 02:29:34 FKM2 kernel: ? set_next_entity+0x93/0xe0
Nis 06 02:29:34 FKM2 kernel: ? do_dump+0xd70/0xd70 [zfs]
Nis 06 02:29:34 FKM2 kernel: thread_generic_wrapper+0x74/0x90 [spl]
Nis 06 02:29:34 FKM2 kernel: kthread+0x105/0x140
Nis 06 02:29:34 FKM2 kernel: ? __thread_exit+0x20/0x20 [spl]
Nis 06 02:29:34 FKM2 kernel: ? kthread_bind+0x20/0x20
Nis 06 02:29:34 FKM2 kernel: ret_from_fork+0x35/0x40
Nis 06 02:29:34 FKM2 kernel: Code: 31 d2 e8 a3 ea ff ff 5b 41 5c 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 85 f6 48 89 e5 74 0a 48 63 07 <48> 8b 04 06 0f 18 08 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 48
Nis 06 02:29:34 FKM2 kernel: RIP: prefetch_freepointer.isra.65+0x11/0x20 RSP: ffffc9000f1b7be8
Nis 06 02:29:34 FKM2 kernel: ---[ end trace f99762de42dd24be ]---As supplementary information here is my pool: (encryption & project_quota features are not enabled)
zpool status
pool: fkmmedium
id: 3417916858648431809
state: ONLINE
status: Some supported features are not enabled on the pool.
action: The pool can be imported using its name or numeric identifier, though
some features will not be available without an explicit 'zpool upgrade'.
config:
NAME STATE READ WRITE CKSUM
fkmmedium ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
35000c500844bbd4b1 ONLINE 0 0 0
35000c50095f3fa87 ONLINE 0 0 0
35000c500844b8017 ONLINE 0 0 0
35000c5009604e493 ONLINE 0 0 0
35000c500844b9e1f1 ONLINE 0 0 0
35000c50095f9e53f1 ONLINE 0 0 0
35000c50095ff4abf1 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
35000c50095bf136b1 ONLINE 0 0 0
35000c5009604c2ab1 ONLINE 0 0 0
35000c500844bd82f1 ONLINE 0 0 0
35000c50095f852b7 ONLINE 0 0 0
35000c50095f4e74f1 ONLINE 0 0 0
35000c5009604c7af1 ONLINE 0 0 0
35000c50095fe3b53 ONLINE 0 0 0
raidz2-2 ONLINE 0 0 0
35000c50095f38377 ONLINE 0 0 0
35000c50096071d7f1 ONLINE 0 0 0
35000c500960f1e23 ONLINE 0 0 0
35000c500844c14bf1 ONLINE 0 0 0
35000c5009609d11f1 ONLINE 0 0 0
35000c500960c0f4b1 ONLINE 0 0 0
35000c500844bae13 ONLINE 0 0 0
raidz2-3 ONLINE 0 0 0
35000c500960f3507 ONLINE 0 0 0
35000c50095f5e8c7 ONLINE 0 0 0
35000c50095f60dd3 ONLINE 0 0 0
35000c500844a7353 ONLINE 0 0 0
35000c50095f7deb3 ONLINE 0 0 0
35000c500960c67e7 ONLINE 0 0 0
35000c500960c4a3f1 ONLINE 0 0 0
raidz2-4 ONLINE 0 0 0
35000c5009603592b1 ONLINE 0 0 0
35000c500844ad437 ONLINE 0 0 0
35000c50095f33f07 ONLINE 0 0 0
35000c50096034fcb1 ONLINE 0 0 0
35000c500960925ff1 ONLINE 0 0 0
35000c500844be673 ONLINE 0 0 0
35000c50096125517 ONLINE 0 0 0
raidz2-5 ONLINE 0 0 0
35000c50095f92327 ONLINE 0 0 0
35000c500844bca53 ONLINE 0 0 0
35000c50095f4b277 ONLINE 0 0 0
35000c500844bd9d3 ONLINE 0 0 0
35000c50095f7df27 ONLINE 0 0 0
35000c500844bd4bb1 ONLINE 0 0 0
35000c5009604ee5f1 ONLINE 0 0 0
raidz2-6 ONLINE 0 0 0
35000c50095ddc787 ONLINE 0 0 0
35000c50095ff45df1 ONLINE 0 0 0
35000c50096071ebb1 ONLINE 0 0 0
35000c5009609dc8b1 ONLINE 0 0 0
35000c50095f3e27b1 ONLINE 0 0 0
35000c500844bc42f1 ONLINE 0 0 0
35000c50095fa4327 ONLINE 0 0 0
raidz2-7 ONLINE 0 0 0
35000c500960eb903 ONLINE 0 0 0
35000c500844b114b1 ONLINE 0 0 0
35000c50095e30d63 ONLINE 0 0 0
35000c500960e81e3 ONLINE 0 0 0
35000c500844ba6ff1 ONLINE 0 0 0
35000c500844ad27f1 ONLINE 0 0 0
35000c50095f5afef1 ONLINE 0 0 0
raidz2-8 ONLINE 0 0 0
35000c50095f9b4f7 ONLINE 0 0 0
35000c500844b127f1 ONLINE 0 0 0
35000c50095f7d71f1 ONLINE 0 0 0
35000c50095e27c17 ONLINE 0 0 0
35000c500844bdc1f1 ONLINE 0 0 0
35000c500844b5fab1 ONLINE 0 0 0
35000c50095e1a507 ONLINE 0 0 0
raidz2-9 ONLINE 0 0 0
35000c500844bef67 ONLINE 0 0 0
35000c50095ede39f1 ONLINE 0 0 0
35000c50095f843af1 ONLINE 0 0 0
35000c500844bc45f1 ONLINE 0 0 0
35000c50095f4b45b1 ONLINE 0 0 0
35000c500844b81b7 ONLINE 0 0 0
35000c500844bc053 ONLINE 0 0 0
raidz2-10 ONLINE 0 0 0
35000c50095f92a8f1 ONLINE 0 0 0
35000c50095f8ba23 ONLINE 0 0 0
35000c500844be667 ONLINE 0 0 0
35000c5009606ed9b1 ONLINE 0 0 0
35000c500844bf083 ONLINE 0 0 0
35000c50095f56c9f1 ONLINE 0 0 0
35000c500844ad1bb1 ONLINE 0 0 0
raidz2-11 ONLINE 0 0 0
35000c50095e1a287 ONLINE 0 0 0
35000c50095f8f597 ONLINE 0 0 0
35000c500960f3df3 ONLINE 0 0 0
35000c50095fb12b7 ONLINE 0 0 0
35000c50095febecf1 ONLINE 0 0 0
35000c500844b839f1 ONLINE 0 0 0
35000c5009609287f1 ONLINE 0 0 0
logs
35000c5003023fe73 ONLINE 0 0 0
35000c5003022e38f1 ONLINE 0 0 0
cache
35000c5003023feeb ONLINE 0 0 0
35000c5003023feef1 ONLINE 0 0 0
35000c5003022e327 ONLINE 0 0 0
35000c5003022e2f3 ONLINE 0 0 0
35000c5003022e2c3 ONLINE 0 0 0
spares
35000c50095f7b9c3 AVAIL
35000c50086636db3 AVAIL
errors: No known data errorsHere is my dataset:
[root@FKM2 ~]# zfs get all fkmmedium/images
NAME PROPERTY VALUE SOURCE
fkmmedium/images type filesystem -
fkmmedium/images creation Çrş Mar 7 18:55 2018 -
fkmmedium/images used 27,0T -
fkmmedium/images available 44,0T -
fkmmedium/images referenced 27,0T -
fkmmedium/images compressratio 1.06x -
fkmmedium/images mounted yes -
fkmmedium/images quota 71T received
fkmmedium/images reservation none received
fkmmedium/images recordsize 128K received
fkmmedium/images mountpoint /fkmmedium/images local
fkmmedium/images sharenfs off default
fkmmedium/images checksum on default
fkmmedium/images compression lz4 received
fkmmedium/images atime off received
fkmmedium/images devices on default
fkmmedium/images exec on default
fkmmedium/images setuid on default
fkmmedium/images readonly off received
fkmmedium/images zoned off default
fkmmedium/images snapdir hidden default
fkmmedium/images aclinherit passthrough received
fkmmedium/images createtxg 3253379 -
fkmmedium/images canmount on default
fkmmedium/images xattr sa received
fkmmedium/images copies 1 default
fkmmedium/images version 5 -
fkmmedium/images utf8only off -
fkmmedium/images normalization none -
fkmmedium/images casesensitivity mixed -
fkmmedium/images vscan off default
fkmmedium/images nbmand off default
fkmmedium/images sharesmb off default
fkmmedium/images refquota none default
fkmmedium/images refreservation none default
fkmmedium/images guid 8661600337962025303 -
fkmmedium/images primarycache all received
fkmmedium/images secondarycache all received
fkmmedium/images usedbysnapshots 0B -
fkmmedium/images usedbydataset 27,0T -
fkmmedium/images usedbychildren 0B -
fkmmedium/images usedbyrefreservation 0B -
fkmmedium/images logbias latency received
fkmmedium/images dedup off received
fkmmedium/images mlslabel none default
fkmmedium/images sync standard default
fkmmedium/images dnodesize legacy local
fkmmedium/images refcompressratio 1.06x -
fkmmedium/images written 0 -
fkmmedium/images logicalused 28,4T -
fkmmedium/images logicalreferenced 28,4T -
fkmmedium/images volmode default default
fkmmedium/images filesystem_limit none default
fkmmedium/images snapshot_limit none default
fkmmedium/images filesystem_count none default
fkmmedium/images snapshot_count none default
fkmmedium/images snapdev hidden default
fkmmedium/images acltype posixacl received
fkmmedium/images context none default
fkmmedium/images fscontext none default
fkmmedium/images defcontext none default
fkmmedium/images rootcontext none default
fkmmedium/images relatime off received
fkmmedium/images redundant_metadata all default
fkmmedium/images overlay off default
fkmmedium/images encryption off default
fkmmedium/images keylocation none default
fkmmedium/images keyformat none default
fkmmedium/images pbkdf2iters 0 defaultHere is more info about the dataset zfs list -t all -o name,used,available,usedsnap,usedds,usedrefreserv,usedbychildren,userrefs,creation -s creation -r fkmmedium/images
NAME USED AVAIL USEDSNAP USEDDS USEDREFRESERV USEDCHILD USERREFS CREATION
fkmmedium/images@1516182586920REPL 0B - - - - - 0 Çrş Oca 17 12:49 2018
fkmmedium/images 27,0T 44,0T 0B 27,0T 0B 0B - Çrş Mar 7 18:55 2018zfs get receive_resume_token fkmmedium/images
Resume Token
NAME PROPERTY VALUE SOURCE
fkmmedium/images receive_resume_token - -Finally I have a couple of things with DEBUG enabled latest bits:
Enabled encryption & project_quota features
Crash log:
Same as features disabled.Disabled prefetch:
Crash log:
Nis 06 03:12:52 FKM2 kernel: general protection fault: 0000 [#1] SMP PTI
Nis 06 03:12:52 FKM2 kernel: Modules linked in: zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) dm_service_time ip6t_rpfilter bonding ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support mxm_wmi sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr joydev input_leds
Nis 06 03:12:52 FKM2 kernel: lpc_ich i2c_i801 mfd_core mei_me mei ioatdma ses enclosure sg shpchp ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter wmi acpi_pad dm_multipath binfmt_misc ip_tables xfs libcrc32c sd_mod crc32c_intel ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci drm ixgbe libata igb mdio mpt3sas ptp raid_class pps_core i2c_algo_bit dca scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod dax
Nis 06 03:12:52 FKM2 kernel: CPU: 13 PID: 2517 Comm: receive_writer Tainted: P O 4.16.0-1.el7.elrepo.x86_64 #1
Nis 06 03:12:52 FKM2 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Nis 06 03:12:52 FKM2 kernel: RIP: 0010:prefetch_freepointer.isra.65+0x11/0x20
Nis 06 03:12:52 FKM2 kernel: RSP: 0018:ffffc9002e84fbe8 EFLAGS: 00010286
Nis 06 03:12:52 FKM2 kernel: RAX: 0000000000000000 RBX: 94d28d9066885195 RCX: 000000000000edb4
Nis 06 03:12:52 FKM2 kernel: RDX: 000000000000edb3 RSI: 94d28d9066885195 RDI: ffff883f7f407020
Nis 06 03:12:52 FKM2 kernel: RBP: ffffc9002e84fbe8 R08: ffff887f7f365f60 R09: ffff887f61ceb400
Nis 06 03:12:52 FKM2 kernel: R10: ffff883f7f407000 R11: ffffffffa069a70f R12: 000000000140c200
Nis 06 03:12:52 FKM2 kernel: R13: 0000000000000158 R14: 00000000ffffffff R15: ffff883f7f407000
Nis 06 03:12:52 FKM2 kernel: FS: 0000000000000000(0000) GS:ffff887f7f340000(0000) knlGS:0000000000000000
Nis 06 03:12:52 FKM2 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Nis 06 03:12:52 FKM2 kernel: CR2: 00007f38b4514000 CR3: 000000000220a003 CR4: 00000000003606e0
Nis 06 03:12:52 FKM2 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000Disabled prefetch and removed all logs & l2arc disks from pool.
Crash log:
Nis 06 03:41:36 FKM2 kernel: general protection fault: 0000 [#1] SMP PTI
Nis 06 03:41:36 FKM2 kernel: Modules linked in: zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) dm_service_time bonding ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support mxm_wmi sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel crypto_simd glue_helper cryptd intel_cstate intel_rapl_perf pcspkr joydev input_leds
Nis 06 03:41:36 FKM2 kernel: i2c_i801 ses enclosure mei_me sg lpc_ich mfd_core mei ioatdma ipmi_si ipmi_devintf acpi_power_meter ipmi_msghandler shpchp wmi acpi_pad dm_multipath binfmt_misc ip_tables xfs libcrc32c sd_mod crc32c_intel ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci drm ixgbe libata igb mdio mpt3sas ptp i2c_algo_bit pps_core raid_class dca scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod dax
Nis 06 03:41:36 FKM2 kernel: CPU: 27 PID: 22463 Comm: receive_writer Tainted: P O 4.16.0-1.el7.elrepo.x86_64 #1
Nis 06 03:41:36 FKM2 kernel: Hardware name: Supermicro X10DRH LN4/X10DRH-CLN4, BIOS 2.0 01/30/2016
Nis 06 03:41:36 FKM2 kernel: RIP: 0010:prefetch_freepointer.isra.65+0x11/0x20
Nis 06 03:41:36 FKM2 kernel: RSP: 0018:ffffc9003411bbe8 EFLAGS: 00010286
Nis 06 03:41:36 FKM2 kernel: RAX: 0000000000000000 RBX: 94d28d9066885195 RCX: 00000000000272aa
Nis 06 03:41:36 FKM2 kernel: RDX: 00000000000272a9 RSI: 94d28d9066885195 RDI: ffff883f7f407020
Nis 06 03:41:36 FKM2 kernel: RBP: ffffc9003411bbe8 R08: ffff887f7f4e5f60 R09: ffff887f5fbeec00This issue is perfectly reproduciable. Please let me know if I can provide more information. Also as this is an isoloted test environment I can provide private access if needed.
tcaputi
commented
6 years ago @morphinz thank you very much for the detailed report. If access to the machine is available, that would probably be the most convenient thing for me to work with, since this bug is probably related to something about the send files you already have. You can email credentials to me at tcaputi@datto.com and I will look into it today. Otherwise I can reproduce this myself tomorrow.
System information
Describe the problem you're observing
When i start zfs send -recv with resume token kernel panics!.. I have 2 pool on 2 server and running replication on them via WAN. On "A" pool to "B" pool i have 10++ dataset replication but only 1 of them has this problem. When i start send-recv on this dataset with resume token to B, "B" node goes panic everytime. Other datasets are just fine. I don't see any log when i start them.
I tried with older kernel and newer kernel or "pti=off" nothing changed. And i have this problem only on 1 dataset. For this reason I think the problem is related to zfs.
Include any warning/errors/backtraces from the system logs
When i start zfs send with resume token kernel gives this log and panics: