openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.45k stars 1.73k forks source link

RHEL Kernel Panic on "zfs recv" #7733

Closed ericdaltman closed 1 year ago

ericdaltman commented 6 years ago

System information

Type Version/Name
Distribution Name RHEL
Distribution Version 7.5
Linux Kernel 3.10.0-862.6.3.el7.x86_64
Architecture x86_64
ZFS Version 0.7.9-1
SPL Version 0.7.9-1

Describe the problem you're observing

Receiving a zfs stream randomly leads to a kernel panic and forced reboot. This has happened with the same pool (with same JBOD chassis) with a totally different head unit (with the same installed software). The pool has been destroyed and recreated with no success in the process. This same head unit with a different pool attached (and receiving from a different pool's "zfs send") has succeeded. Two totally different types of HBAs have been tried as well. The sending machine is unaffected. No networking issues have been detected. vmcore-dump included below.

Describe how to reproduce the problem

Set up a "zfs recv" and wait. Sometimes we have a successful process and other attempts, even on the same datasets, leads to a kernel panic.

Include any warning/errors/backtraces from the system logs

[root@REDACTED Primary]# cat /var/crash/127.0.0.1-2018-07-20-13\:36\:54/vmcore-dmesg.txt 
[    0.000000] Linux version 3.10.0-862.6.3.el7.x86_64 (mockbuild@x86-017.build.eng.bos.redhat.com) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-28) (GCC) ) #1 SMP Fri Jun 15 17:57:37 EDT 2018
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x0000000000099fff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009a000-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000078f0ffff] usable
[    0.000000] BIOS-e820: [mem 0x0000000078f10000-0x0000000079818fff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000079819000-0x0000000079d49fff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x0000000079d4a000-0x000000008fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed1c000-0x00000000fed44fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000c07fffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 3.0 present.
[    0.000000] DMI: Supermicro Super Server/X10DRC-T4+, BIOS 2.0a 09/16/2016
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] e820: last_pfn = 0xc080000 max_arch_pfn = 0x400000000
[    0.000000] Reserving 208MB of memory at 624MB for crashkernel (System RAM: 786318MB)
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x00001000-0x00ffffff]
[    0.000000]   DMA32    [mem 0x01000000-0xffffffff]
[    0.000000]   Normal   [mem 0x100000000-0xc07fffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x00001000-0x00099fff]
[    0.000000]   node   0: [mem 0x00100000-0x78f0ffff]
[    0.000000]   node   0: [mem 0x100000000-0x607fffffff]
[    0.000000]   node   1: [mem 0x6080000000-0xc07fffffff]
[    0.000000] Initmem setup node 0 [mem 0x00001000-0x607fffffff]
[    0.000000] On node 0 totalpages: 100634281
[    0.000000]   DMA zone: 64 pages used for memmap
[    0.000000]   DMA zone: 21 pages reserved
[    0.000000]   DMA zone: 3993 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 7677 pages used for memmap
[    0.000000]   DMA32 zone: 491280 pages, LIFO batch:31
[    0.000000]   Normal zone: 1564672 pages used for memmap
[    0.000000]   Normal zone: 100139008 pages, LIFO batch:31
[    0.000000] Initmem setup node 1 [mem 0x6080000000-0xc07fffffff]
[    0.000000] On node 1 totalpages: 100663296
[    0.000000]   Normal zone: 1572864 pages used for memmap
[    0.000000]   Normal zone: 100663296 pages, LIFO batch:31
...
[    0.000000] [Firmware Bug]: TSC_DEADLINE disabled due to Errata; please update microcode to version: 0xb000020 (or later)
[    0.000000] smpboot: Allowing 48 CPUs, 0 hotplug CPUs
...
[    0.000000] e820: [mem 0x90000000-0xfed1bfff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on bare hardware
[    0.000000] setup_percpu: NR_CPUS:5120 nr_cpumask_bits:48 nr_cpu_ids:48 nr_node_ids:2
[    0.000000] PERCPU: Embedded 35 pages/cpu @ffff948f3f800000 s104920 r8192 d30248 u262144
...
[    0.000000] x86/fpu: xstate_offset[2]: 0240, xstate_sizes[2]: 0100
[    0.000000] xsave: enabled xstate_bv 0x7, cntxt size 0x340 using standard form
[    0.000000] Memory: 7962216k/807403520k available (7328k kernel code, 2213212k absent, 12920208k reserved, 6301k data, 1832k init)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=48, Nodes=2
[    0.000000] x86/pti: Unmapping kernel while in userspace
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  RCU restricting CPUs from NR_CPUS=5120 to nr_cpu_ids=48.
[    0.000000] NR_IRQS:327936 nr_irqs:1624 0
...
[    0.000000] allocated 3221225472 bytes of page_cgroup
[    0.000000] please try 'cgroup_disable=memory' option if you don't want memory cgroups
[    0.000000] Enabling automatic NUMA balancing. Configure with numa_balancing= or the kernel.numa_balancing sysctl
[    0.000000] hpet clockevent registered
[    0.000000] tsc: Fast TSC calibration failed
[    0.000000] tsc: Unable to calibrate against PIT
[    0.000000] tsc: using HPET reference calibration
[    0.000000] tsc: Detected 2999.985 MHz processor
[    0.000038] Calibrating delay loop (skipped), value calculated using timer frequency.. 5999.97 BogoMIPS (lpj=2999985)
[    0.000040] pid_max: default: 49152 minimum: 384
...
[    0.059458] Dentry cache hash table entries: 134217728 (order: 18, 1073741824 bytes)
...
[    0.345968] mce: CPU supports 22 MCE banks
...
[    0.407025] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2687W v4 @ 3.00GHz (fam: 06, model: 4f, stepping: 01)
...
[    2.262033] Brought up 48 CPUs
...
[    3.960332] RTC time: 18:48:56, date: 07/20/18
...
[    3.960853] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0x80000000-0x8fffffff] (base 0x80000000)
[    3.960855] PCI: MMCONFIG at [mem 0x80000000-0x8fffffff] reserved in E820
...
[    3.996627] ACPI: [Firmware Bug]: BIOS _OSI(Linux) query ignored
...
[    4.104485] PCI host bridge to bus 0000:ff
[    4.104487] pci_bus 0000:ff: root bus resource [bus ff]
...
[    4.592525] zpool: loaded
[    4.592528] zbud: loaded
...
[    6.302628] megasas: 07.702.06.00-rh2
...
[    6.313542] libata version 3.00 loaded.
[    6.323288] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.1.0-k-rh7.5
...
[    6.323634] pmd_set_huge: Cannot satisfy [mem 0xc4800000-0xc4a00000] with a huge-page mapping due to MTRR override.
[    6.324740] mpt3sas version 16.100.00.00 loaded
...
[    6.328116] mlx5_core 0000:02:00.0: firmware version: 12.18.1000
...
[    6.544727] mpt3sas_cm0: Allocated physical memory: size(21632 kB)
...
[   17.896753] spl: loading out-of-tree module taints kernel.
[   17.899878] spl: module verification failed: signature and/or required key missing - tainting kernel
[   17.905071] SPL: Loaded module v0.7.9-1
[   17.906088] znvpair: module license 'CDDL' taints kernel.
[   17.906090] Disabling lock debugging due to kernel taint
[   17.908314] alg: No test for __gcm-aes-aesni (__driver-gcm-aes-aesni)
[   17.908374] alg: No test for __generic-gcm-aes-aesni (__driver-generic-gcm-aes-aesni)
...
[   21.111668] ZFS: Loaded module v0.7.9-1, ZFS pool version 5000, ZFS filesystem version 5
[   21.215016] mlx5_ib: Mellanox Connect-IB Infiniband driver v5.0-0
...
[ 6428.851941] ------------[ cut here ]------------
[ 6428.851952] WARNING: CPU: 25 PID: 3725 at lib/list_debug.c:33 __list_add+0xac/0xc0
[ 6428.851954] list_add corruption. prev->next should be next (ffff94ad11bb5d08), but was           (null). (prev=ffff94ed40ea4400).
[ 6428.851955] Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bonding rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx5_ib ib_core zfs(POE) zunicode(POE) zavl(POE) icp(POE) iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul zcommon(POE) ghash_clmulni_intel znvpair(POE) aesni_intel spl(OE) lrw gf128mul glue_helper dm_multipath ablk_helper cryptd pcspkr
[ 6428.851997]  ses enclosure joydev i2c_i801 sg lpc_ich mei_me mei ioatdma shpchp wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter nfsd nfs_acl lockd auth_rpcgss grace sunrpc binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci mlx5_core mpt3sas drm ixgbe libahci libata crct10dif_pclmul crct10dif_common crc32c_intel megaraid_sas raid_class scsi_transport_sas mlxfw i2c_core mdio devlink ptp dca pps_core dm_mirror dm_region_hash dm_log dm_mod
[ 6428.852030] CPU: 25 PID: 3725 Comm: receive_writer Kdump: loaded Tainted: P           OE  ------------   3.10.0-862.6.3.el7.x86_64 #1
[ 6428.852048] Hardware name: Supermicro Super Server/X10DRC-T4+, BIOS 2.0a 09/16/2016
[ 6428.852049] Call Trace:
[ 6428.852056]  [<ffffffffb1d0e80e>] dump_stack+0x19/0x1b
[ 6428.852061]  [<ffffffffb1691e18>] __warn+0xd8/0x100
[ 6428.852063]  [<ffffffffb1691e9f>] warn_slowpath_fmt+0x5f/0x80
[ 6428.852065]  [<ffffffffb1968dac>] __list_add+0xac/0xc0
[ 6428.852103]  [<ffffffffc0f02290>] dbuf_dirty+0x4e0/0x810 [zfs]
[ 6428.852121]  [<ffffffffc0f1fab8>] dnode_setdirty+0xc8/0x110 [zfs]
[ 6428.852134]  [<ffffffffc0f096e1>] dmu_object_set_checksum+0x71/0xa0 [zfs]
[ 6428.852149]  [<ffffffffc0f151de>] receive_object+0x31e/0x440 [zfs]
[ 6428.852163]  [<ffffffffc0f17a33>] receive_writer_thread+0x2a3/0xa80 [zfs]
[ 6428.852168]  [<ffffffffb16db5e8>] ? enqueue_task_fair+0x208/0x6c0
[ 6428.852170]  [<ffffffffb16d2765>] ? sched_clock_cpu+0x85/0xc0
[ 6428.852185]  [<ffffffffc0f17790>] ? receive_free.isra.11+0xd0/0xd0 [zfs]
[ 6428.852192]  [<ffffffffc08eef75>] ? spl_kmem_free+0x35/0x40 [spl]
[ 6428.852206]  [<ffffffffc0f17790>] ? receive_free.isra.11+0xd0/0xd0 [zfs]
[ 6428.852211]  [<ffffffffc08f1013>] thread_generic_wrapper+0x73/0x80 [spl]
[ 6428.852215]  [<ffffffffc08f0fa0>] ? __thread_exit+0x20/0x20 [spl]
[ 6428.852218]  [<ffffffffb16bb621>] kthread+0xd1/0xe0
[ 6428.852220]  [<ffffffffb16bb550>] ? insert_kthread_work+0x40/0x40
[ 6428.852226]  [<ffffffffb1d205f7>] ret_from_fork_nospec_begin+0x21/0x21
[ 6428.852228]  [<ffffffffb16bb550>] ? insert_kthread_work+0x40/0x40
[ 6428.852229] ---[ end trace 6f13a567edc68112 ]---
[ 6430.560456] ------------[ cut here ]------------
[ 6430.560468] WARNING: CPU: 5 PID: 24239 at lib/list_debug.c:62 __list_del_entry+0x82/0xd0
[ 6430.560470] list_del corruption. next->prev should be ffff94ed40ea6a00, but was           (null)
[ 6430.560471] Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bonding rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx5_ib ib_core zfs(POE) zunicode(POE) zavl(POE) icp(POE) iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul zcommon(POE) ghash_clmulni_intel znvpair(POE) aesni_intel spl(OE) lrw gf128mul glue_helper dm_multipath ablk_helper cryptd pcspkr
[ 6430.560528]  ses enclosure joydev i2c_i801 sg lpc_ich mei_me mei ioatdma shpchp wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter nfsd nfs_acl lockd auth_rpcgss grace sunrpc binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci mlx5_core mpt3sas drm ixgbe libahci libata crct10dif_pclmul crct10dif_common crc32c_intel megaraid_sas raid_class scsi_transport_sas mlxfw i2c_core mdio devlink ptp dca pps_core dm_mirror dm_region_hash dm_log dm_mod
[ 6430.560576] CPU: 5 PID: 24239 Comm: txg_sync Kdump: loaded Tainted: P        W  OE  ------------   3.10.0-862.6.3.el7.x86_64 #1
[ 6430.560578] Hardware name: Supermicro Super Server/X10DRC-T4+, BIOS 2.0a 09/16/2016
[ 6430.560579] Call Trace:
[ 6430.560587]  [<ffffffffb1d0e80e>] dump_stack+0x19/0x1b
[ 6430.560594]  [<ffffffffb1691e18>] __warn+0xd8/0x100
[ 6430.560596]  [<ffffffffb1691e9f>] warn_slowpath_fmt+0x5f/0x80
[ 6430.560637]  [<ffffffffc0f03010>] ? dbuf_sync_leaf+0x250/0x450 [zfs]
[ 6430.560646]  [<ffffffffc08f0379>] ? spl_kmem_cache_alloc+0x99/0x150 [spl]
[ 6430.560648]  [<ffffffffb1968e42>] __list_del_entry+0x82/0xd0
[ 6430.560650]  [<ffffffffb1968e9d>] list_del+0xd/0x30
[ 6430.560662]  [<ffffffffc0f03298>] dbuf_sync_list+0x88/0xd0 [zfs]
[ 6430.560673]  [<ffffffffc0f033df>] dbuf_sync_indirect+0xff/0x1b0 [zfs]
[ 6430.560684]  [<ffffffffc0f032ad>] dbuf_sync_list+0x9d/0xd0 [zfs]
[ 6430.560694]  [<ffffffffc0f033df>] dbuf_sync_indirect+0xff/0x1b0 [zfs]
[ 6430.560705]  [<ffffffffc0f032ad>] dbuf_sync_list+0x9d/0xd0 [zfs]
[ 6430.560715]  [<ffffffffc0f033df>] dbuf_sync_indirect+0xff/0x1b0 [zfs]
[ 6430.560726]  [<ffffffffc0f032ad>] dbuf_sync_list+0x9d/0xd0 [zfs]
[ 6430.560736]  [<ffffffffc0f033df>] dbuf_sync_indirect+0xff/0x1b0 [zfs]
[ 6430.560747]  [<ffffffffc0f032ad>] dbuf_sync_list+0x9d/0xd0 [zfs]
[ 6430.560757]  [<ffffffffc0f033df>] dbuf_sync_indirect+0xff/0x1b0 [zfs]
[ 6430.560793]  [<ffffffffc0f032ad>] dbuf_sync_list+0x9d/0xd0 [zfs]
[ 6430.560815]  [<ffffffffc0f220b0>] dnode_sync+0x370/0x880 [zfs]
[ 6430.560832]  [<ffffffffc0f0e1b0>] ? sync_dnodes_task+0xc0/0xc0 [zfs]
[ 6430.560848]  [<ffffffffc0f0d550>] ? checksum_changed_cb+0x30/0x30 [zfs]
[ 6430.560864]  [<ffffffffc0f100f8>] dmu_objset_sync+0x1c8/0x420 [zfs]
[ 6430.560881]  [<ffffffffc0f0d550>] ? checksum_changed_cb+0x30/0x30 [zfs]
[ 6430.560896]  [<ffffffffc0f0e1b0>] ? sync_dnodes_task+0xc0/0xc0 [zfs]
[ 6430.560915]  [<ffffffffc0f235a1>] dsl_dataset_sync+0x71/0x250 [zfs]
[ 6430.560938]  [<ffffffffc0f353f3>] dsl_pool_sync+0xa3/0x440 [zfs]
[ 6430.560961]  [<ffffffffc0f534b7>] spa_sync+0x437/0xd90 [zfs]
[ 6430.560964]  [<ffffffffb16cf682>] ? default_wake_function+0x12/0x20
[ 6430.560967]  [<ffffffffb16c8204>] ? __wake_up+0x44/0x50
[ 6430.560991]  [<ffffffffc0f67c41>] txg_sync_thread+0x301/0x510 [zfs]
[ 6430.561014]  [<ffffffffc0f67940>] ? txg_fini+0x2a0/0x2a0 [zfs]
[ 6430.561021]  [<ffffffffc08f1013>] thread_generic_wrapper+0x73/0x80 [spl]
[ 6430.561026]  [<ffffffffc08f0fa0>] ? __thread_exit+0x20/0x20 [spl]
[ 6430.561029]  [<ffffffffb16bb621>] kthread+0xd1/0xe0
[ 6430.561031]  [<ffffffffb16bb550>] ? insert_kthread_work+0x40/0x40
[ 6430.561037]  [<ffffffffb1d205f7>] ret_from_fork_nospec_begin+0x21/0x21
[ 6430.561039]  [<ffffffffb16bb550>] ? insert_kthread_work+0x40/0x40
[ 6430.561041] ---[ end trace 6f13a567edc68113 ]---
[ 6430.561074] BUG: unable to handle kernel NULL pointer dereference at 0000000000000018
[ 6430.563248] IP: [<ffffffffc0f0e131>] sync_dnodes_task+0x41/0xc0 [zfs]
[ 6430.565486] PGD 0 
[ 6430.567651] Oops: 0000 [#1] SMP 
[ 6430.569778] Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bonding rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx5_ib ib_core zfs(POE) zunicode(POE) zavl(POE) icp(POE) iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul zcommon(POE) ghash_clmulni_intel znvpair(POE) aesni_intel spl(OE) lrw gf128mul glue_helper dm_multipath ablk_helper cryptd pcspkr
[ 6430.582967]  ses enclosure joydev i2c_i801 sg lpc_ich mei_me mei ioatdma shpchp wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter nfsd nfs_acl lockd auth_rpcgss grace sunrpc binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci mlx5_core mpt3sas drm ixgbe libahci libata crct10dif_pclmul crct10dif_common crc32c_intel megaraid_sas raid_class scsi_transport_sas mlxfw i2c_core mdio devlink ptp dca pps_core dm_mirror dm_region_hash dm_log dm_mod
[ 6430.594158] CPU: 6 PID: 22020 Comm: dp_sync_taskq Kdump: loaded Tainted: P        W  OE  ------------   3.10.0-862.6.3.el7.x86_64 #1
[ 6430.598547] Hardware name: Supermicro Super Server/X10DRC-T4+, BIOS 2.0a 09/16/2016
[ 6430.600739] task: ffff94eea1619fa0 ti: ffff94ab67940000 task.ti: ffff94ab67940000
[ 6430.602909] RIP: 0010:[<ffffffffc0f0e131>]  [<ffffffffc0f0e131>] sync_dnodes_task+0x41/0xc0 [zfs]
[ 6430.605091] RSP: 0018:ffff94ab67943d40  EFLAGS: 00010286
[ 6430.607214] RAX: 0000000000000000 RBX: ffff94cfc300f1c0 RCX: ffff94ab67943fd8
[ 6430.609321] RDX: ffff94bbc5e86048 RSI: ffff94cfc300f1c0 RDI: ffff94bbc5e86000
[ 6430.611395] RBP: ffff94ab67943d68 R08: ffff94ae9cf59a28 R09: 0000000000000001
[ 6430.613441] R10: ffff9430b40e7d38 R11: 7fffffffffffffff R12: ffff948de7aafba0
[ 6430.615492] R13: ffff94bbc5e86000 R14: ffff948eb42e58c0 R15: ffff94ac598cd0c0
[ 6430.617509] FS:  0000000000000000(0000) GS:ffff948f3f980000(0000) knlGS:0000000000000000
[ 6430.619516] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6430.621489] CR2: 0000000000000018 CR3: 0000009801a0e000 CR4: 00000000003607e0
[ 6430.623436] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6430.625367] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6430.627279] Call Trace:
[ 6430.629155]  [<ffffffffc08f1d2c>] taskq_thread+0x2ac/0x4f0 [spl]
[ 6430.631012]  [<ffffffffb16cf670>] ? wake_up_state+0x20/0x20
[ 6430.632865]  [<ffffffffc08f1a80>] ? taskq_thread_spawn+0x60/0x60 [spl]
[ 6430.634707]  [<ffffffffb16bb621>] kthread+0xd1/0xe0
[ 6430.636512]  [<ffffffffb16bb550>] ? insert_kthread_work+0x40/0x40
[ 6430.638284]  [<ffffffffb1d205f7>] ret_from_fork_nospec_begin+0x21/0x21
[ 6430.640051]  [<ffffffffb16bb550>] ? insert_kthread_work+0x40/0x40
[ 6430.641808] Code: 77 08 48 8b 3f e8 60 9e 03 00 4d 8b 77 18 49 89 c5 eb 59 0f 1f 80 00 00 00 00 48 8b 43 40 48 89 de 4c 89 ef 48 8b 80 f0 00 00 00 <48> 8b 40 18 48 89 83 b8 02 00 00 e8 9f 9f 03 00 48 8b 43 30 4c 
[ 6430.645484] RIP  [<ffffffffc0f0e131>] sync_dnodes_task+0x41/0xc0 [zfs]
[ 6430.647259]  RSP <ffff94ab67943d40>
[ 6430.648996] CR2: 0000000000000018
ericdaltman commented 6 years ago

Not really sure what will help here so I'll add what I can think of. First of all, not using dedup or any such silliness.

Some messages from dbgmsg (many of which repeat heavily):

1532140180   dnode.c:1314:dnode_hold_impl(): error 2
1532140180   dnode.c:576:dnode_allocate(): os=ffffa04afee16800 obj=345529 txg=360 blocksize=131072 ibs=17 dn_slots=1
1532140179   dnode_sync.c:61:dnode_increase_indirection(): os=ffffa04afee16800 obj=344691, increase to 2
1532140188 dbuf.c:2269:dbuf_findbp(): error 2

These appear once starting a zfs recv:

1532140545   dsl_prop.c:147:dsl_prop_get_dd(): error 2
1532140545   zap_micro.c:628:mzap_upgrade(): upgrading obj=1544 with 7 chunks
1532140545   zap_micro.c:640:mzap_upgrade(): adding org.zfsonlinux:userobj_accounting=0
1532140545   zap_leaf.c:441:zap_leaf_lookup(): error 2
1532140545   zap_micro.c:640:mzap_upgrade(): adding com.delphix:resume_fromguid=3794124285519673276
1532140545   zap_leaf.c:441:zap_leaf_lookup(): error 2
1532140545   zap_micro.c:640:mzap_upgrade(): adding com.delphix:resume_toguid=2494501697666804117
1532140545   dbuf.c:2269:dbuf_findbp(): error 2
1532140545   zap_leaf.c:441:zap_leaf_lookup(): error 2
1532140545   spa_history.c:317:spa_history_log_sync(): txg 425 receive REDACTED/REDACTED/%recv (id 1544) 
1532140545   zap.c:766:fzap_checksize(): error 22
ericdaltman commented 6 years ago

This issue continues to plague us and could cost a whole lot of well.. my job.

Continuing to seek advice and help. Happy to provide ANY required additional data.

At this point, out of 11 datasets on the Pool, only two (one 131TB and the other 11TB) cause a hard panic and reboot on the receiving side while not affecting the sending side.

I tried to send to a stream file, transfer that file, checksum it to verify transfer, and receive it. Only difference is that it crashes quicker.

Attempted downgrade to 0.7.4. Tried dkms instead of kmod.

zstreamdump did not seem to have any issue with processing the transferred zfs stream file.

Turning on zfs_dbgmsg_enable with as many flags as possible and I see these:

zfs_ioctl.c:1388:put_nvlist(): error 12
zfs_ioctl.c:1601:zfs_ioc_pool_configs(): error 17

The last reported txg was txg.c:509:txg_sync_thread(): waiting; tx_synced=135774 waiting=135772 dp=ffff93f46ddb6800

I'm a little at the end of my rope here.

One strange behavior I see... the drives were used for another pool before (that also crashed on this process) and I rebuilt the pool. Now when I do a 'zpool import -d /dev/disk/by-id Primary' it still sees remnants of the old pool and makes me have to type in the id to import the correct pool. If I don't use '-d', it imports the proper one with no problem.

rincebrain commented 6 years ago

Yikes, that's nasty.

What's zpool get all/zfs get all say on the (source and destination) pool/datasets involved, respectively?

What's the hardware/software configuration of the source and destination of these sends?

You could try rolling a kernel+modules with kASAN enabled to try and tell you what corrupted the in-memory data structure, though that is only a single level of indirection resolved (e.g. it tells you where the wild memory write happened, but not, on its own, how the pointer math ended up eating paste).

GregorKopka commented 6 years ago

One strange behavior I see... the drives were used for another pool before (that also crashed on this process) and I rebuilt the pool. Now when I do a 'zpool import -d /dev/disk/by-id Primary' it still sees remnants of the old pool and makes me have to type in the id to import the correct pool. If I don't use '-d', it imports the proper one with no problem.

Possibly the partition table is a bit different now. You could dd if=/dev/zero the parts of the drive that are not used by the partition holding the new pool, to get rid of stale uberblocks from the old pool.

See https://github.com/zfsonlinux/zfs/issues/7656 on why import -d behaves differently.

ericdaltman commented 6 years ago

Ah! Replies! I think I had given up on help. Thank you.

I'm remote at the moment and unable to get the details requested. However, as soon as I'm back in country I'll get you details you're asking for.

And thank you GregorKopka. That is both interesting and troubling, I'll have to figure out the specific methodology for cleaning up the dead pool (heh) information without killing my perfectly healthy pool..

stale[bot] commented 4 years ago

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

Quoor8ee commented 4 years ago

I hit into something similar, maybe the same issue (I'll gather kernel panic logs on next time I have access to the machine in question)

Type Version/Name
Distribution Name Debian
Distribution Version 10.4 (stable)
Linux Kernel Debian 4.19.132-1 (2020-07-24)
Architecture x86_64
ZFS Version 0.7.12-2+deb10u2
SPL Version 0.7.12-2+deb10u1

zfs recv seemingly received the whole snapshot (from what the network transfer statistics say), but at the very end the kernel panics. Repeated occurrence with various snapshots of one remote dataset (unfortunately one of unhandy multi-TB size), but a small one (a few hundred KB) does get received without issues. I paid attention to receive the snapshot into a non-existing dataset.

I suspect my HBA of being too impatient with my HDDs, but I don't have a replacement ATM.

I'll give the buster-backports version of zfs-dkms (0.8.4-2~bpo10+1) a run.

stale[bot] commented 3 years ago

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

GregorKopka commented 3 years ago

Another defect where I have to stop the stale bot.

behlendorf commented 3 years ago

While I can't say for certain the issue described here is consistent with what's described in this https://github.com/openzfs/zfs/issues/7735#issuecomment-416628385. The root cause of those issues were resolved in the 0.7.12 tag. There was a report here of a similar issue with 0.7.12-2+deb10u2, but it would be helpful to know if this has ever been reproduced with something more current.

stale[bot] commented 2 years ago

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.