search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.45k stars 1.73k forks source link

Panic while receiving and encrypting dataset #13491

Open zrav opened 2 years ago

zrav commented 2 years ago

System information

Type Version/Name
Distribution Name Ubuntu
Distribution Version 22.04
Kernel Version 5.15.0-30-generic
Architecture x86_64
OpenZFS Version zfs-2.1.2-1ubuntu3

Describe the problem you're observing

While receiving an unencrypted dataset that was then compressed and encrypted, the system paniced.

Describe how to reproduce the problem

I don't have a surefire way to reproduce. But it did happen twice in a matter of minutes. Send options -L -p, receive options -x encryption -x compression.

Include any warning/errors/backtraces from the system logs

[18664.867056] ------------[ cut here ]------------
[18664.867059] virt_to_cache: Object is not a Slab page!
[18664.867077] WARNING: CPU: 61 PID: 2590174 at mm/slab.h:413 kmem_cache_free+0x22d/0x290
[18664.867085] Modules linked in: tls unix_diag cdc_ether usbnet joydev mii br_netfilter xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat bpfilter overlay binfmt_misc veth nvme_fabrics vhost_vsock vmw_vsock_virtio_transport_common vsock vhost_net vhost vhost_iotlb tap xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_counter nf_tables nfnetlink bridge stp llc intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd kvm_amd kvm ipmi_ssif rapl efi_pstore zfs(PO) zunicode(PO) zzstd(O) zlua(O) nls_iso8859_1 zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) input_leds ccp k10temp ptdma acpi_ipmi ipmi_si mac_hid dm_multipath sch_fq_codel scsi_dh_rdac scsi_dh_emc scsi_dh_alua jc42 ipmi_devintf ipmi_msghandler msr ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath
[18664.867145]  linear raid1 hid_generic ast i2c_algo_bit drm_vram_helper drm_ttm_helper ttm usbhid drm_kms_helper hid syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd cec rc_core nvme ahci xhci_pci drm tg3 i2c_piix4 xhci_pci_renesas nvme_core libahci
[18664.867166] CPU: 61 PID: 2590174 Comm: receive_writer Tainted: P           O      5.15.0-30-generic #31-Ubuntu
[18664.867168] Hardware name: TYAN B8036T65AV12E16HR/S8036GM2NE, BIOS V1.00.B20 10/31/2019
[18664.867170] RIP: 0010:kmem_cache_free+0x22d/0x290
[18664.867173] Code: 01 0f 87 00 6e 98 00 83 e3 01 0f 85 64 ff ff ff 48 c7 c6 60 11 04 b0 48 c7 c7 88 a8 3f b0 c6 05 19 4b eb 01 01 e8 83 8c 96 00 <0f> 0b e9 43 ff ff ff ba 01 00 00 00 48 8d 75 c8 4c 89 ff e8 4b 92
[18664.867174] RSP: 0018:ffffbb4a5aedf990 EFLAGS: 00010286
[18664.867176] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000027
[18664.867177] RDX: ffff9c9f4bd60988 RSI: 0000000000000001 RDI: ffff9c9f4bd60980
[18664.867178] RBP: ffffbb4a5aedf9d8 R08: 0000000000000003 R09: ffffffffffe386f0
[18664.867179] R10: ffffffffffffffff R11: 0000000000000001 R12: ffff9c508d650000
[18664.867180] R13: ffff9c510d650000 R14: ffff9c508d650000 R15: 0000000000004000
[18664.867181] FS:  0000000000000000(0000) GS:ffff9c9f4bd40000(0000) knlGS:0000000000000000
[18664.867183] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[18664.867184] CR2: 00007fb775966010 CR3: 0000001cfd9a8000 CR4: 0000000000350ee0
[18664.867185] Call Trace:
[18664.867187]  <TASK>
[18664.867190]  spl_kmem_cache_free+0xc9/0x130 [spl]
[18664.867201]  zio_buf_free+0x33/0x80 [zfs]
[18664.867388]  abd_return_buf+0x2d/0x40 [zfs]
[18664.867470]  zio_crypt_copy_dnode_bonus+0x11d/0x170 [zfs]
[18664.867566]  arc_buf_untransform_in_place.constprop.0+0x2a/0x40 [zfs]
[18664.867648]  arc_buf_fill+0x2c0/0x4d0 [zfs]
[18664.867730]  arc_untransform+0x23/0x80 [zfs]
[18664.867812]  dbuf_read_verify_dnode_crypt+0xe7/0x180 [zfs]
[18664.867892]  dbuf_read_bonus+0x23/0x110 [zfs]
[18664.867971]  dbuf_read_impl.constprop.0+0x303/0x490 [zfs]
[18664.868050]  ? dbuf_rele_and_unlock+0x12c/0x530 [zfs]
[18664.868129]  ? wmsum_add+0xe/0x10 [zfs]
[18664.868208]  ? arc_buf_access+0x229/0x240 [zfs]
[18664.868286]  ? __cond_resched+0x1a/0x50
[18664.868291]  ? do_raw_spin_unlock+0x9/0x10 [zfs]
[18664.868369]  ? __raw_spin_unlock+0x9/0x10 [zfs]
[18664.868447]  ? __cond_resched+0x1a/0x50
[18664.868450]  ? down_read+0x13/0x90
[18664.868452]  dbuf_read+0x1b6/0x5a0 [zfs]
[18664.868531]  ? __cond_resched+0x1a/0x50
[18664.868533]  ? do_raw_spin_unlock+0x9/0x10 [zfs]
[18664.868616]  dmu_bonus_hold_by_dnode+0x8d/0x180 [zfs]
[18664.868696]  receive_object+0x48e/0xa60 [zfs]
[18664.868777]  ? dnode_rele_and_unlock+0x69/0xe0 [zfs]
[18664.868858]  ? spl_kmem_free_impl+0x25/0x30 [spl]
[18664.868866]  ? __list_del_entry+0x1d/0x30 [zfs]
[18664.868939]  ? list_del+0xd/0x30 [zfs]
[18664.869012]  receive_process_record+0xbf/0x230 [zfs]
[18664.869087]  receive_writer_thread+0xb3/0x180 [zfs]
[18664.869161]  ? receive_process_record+0x230/0x230 [zfs]
[18664.869236]  thread_generic_wrapper+0x64/0x70 [spl]
[18664.869243]  ? __thread_exit+0x20/0x20 [spl]
[18664.869250]  kthread+0x12a/0x150
[18664.869254]  ? set_kthread_struct+0x50/0x50
[18664.869256]  ret_from_fork+0x22/0x30
[18664.869260]  </TASK>
[18664.869261] ---[ end trace 50b300a59d683227 ]---
[19408.896773] BUG: unable to handle page fault for address: 0000000100000000
[19408.896807] #PF: supervisor write access in kernel mode
[19408.896824] #PF: error_code(0x0002) - not-present page
[19408.896840] PGD 0 P4D 0
[19408.896854] Oops: 0002 [#1] SMP NOPTI
[19408.896873] CPU: 98 PID: 1011197 Comm: receive_writer Tainted: P        W  O      5.15.0-30-generic #31-Ubuntu
[19408.896906] Hardware name: TYAN B8036T65AV12E16HR/S8036GM2NE, BIOS V1.00.B20 10/31/2019
[19408.896928] RIP: 0010:__memcpy+0x12/0x20
[19408.896945] Code: cc cc cc cc cc cc cc cc 48 8b 05 a1 23 8e 01 c3 cc cc cc cc cc cc cc cc 0f 1f 44 00 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48 89 d1 f3 a4
[19408.896995] RSP: 0018:ffffbb491f1bb878 EFLAGS: 00010246
[19408.897011] RAX: 0000000100000000 RBX: ffffbb491f1bb928 RCX: 0000000000000800
[19408.897032] RDX: 0000000000000000 RSI: ffff9c50dc884000 RDI: 0000000100000000
[19408.897053] RBP: ffffbb491f1bb890 R08: ffff9c50dc884000 R09: 0000000000000000
[19408.897072] R10: ffff9c51bb1e7500 R11: 000000000000000c R12: 0000000000004000
[19408.897093] R13: 0000000000000000 R14: ffffbb491f1bb928 R15: 0000000000000000
[19408.897114] FS:  0000000000000000(0000) GS:ffff9c9f4c680000(0000) knlGS:0000000000000000
[19408.897137] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[19408.897153] CR2: 0000000100000000 CR3: 00000028c25c8000 CR4: 0000000000350ee0
[19408.897174] Call Trace:
[19408.897184]  <TASK>
[19408.897194]  ? abd_copy_to_buf_off_cb+0x23/0x30 [zfs]
[19408.897282]  abd_iterate_func+0xbb/0x100 [zfs]
[19408.897360]  ? abd_advance_abd_iter+0x80/0x80 [zfs]
[19408.897440]  abd_copy_to_buf_off+0x3c/0x60 [zfs]
[19408.897518]  abd_copy_to_buf+0xe/0x10 [zfs]
[19408.897596]  abd_borrow_buf_copy+0x42/0x50 [zfs]
[19408.897675]  zio_crypt_copy_dnode_bonus+0x33/0x170 [zfs]
[19408.897783]  ? do_raw_spin_unlock+0x9/0x10 [zfs]
[19408.897862]  arc_buf_untransform_in_place.constprop.0+0x2a/0x40 [zfs]
[19408.897947]  arc_buf_fill+0x2c0/0x4d0 [zfs]
[19408.898026]  arc_untransform+0x23/0x80 [zfs]
[19408.898104]  dbuf_read_verify_dnode_crypt+0xe7/0x180 [zfs]
[19408.898189]  dbuf_read_impl.constprop.0+0xf0/0x490 [zfs]
[19408.898280]  ? do_raw_spin_unlock+0x9/0x10 [zfs]
[19408.898360]  ? __cond_resched+0x1a/0x50
[19408.898374]  ? down_read+0x13/0x90
[19408.898386]  dbuf_read+0x1b6/0x5a0 [zfs]
[19408.898466]  dmu_tx_check_ioerr+0x6e/0xd0 [zfs]
[19408.898551]  dmu_tx_hold_free_impl+0x151/0x290 [zfs]
[19408.898634]  dmu_tx_hold_free+0x40/0x50 [zfs]
[19408.898717]  dmu_free_long_range_impl+0x120/0x350 [zfs]
[19408.898803]  dmu_free_long_range+0x74/0xc0 [zfs]
[19408.898886]  dmu_free_long_object+0x27/0xc0 [zfs]
[19408.898968]  receive_freeobjects+0x85/0x100 [zfs]
[19408.899052]  receive_process_record+0x159/0x230 [zfs]
[19408.899136]  receive_writer_thread+0xb3/0x180 [zfs]
[19408.899219]  ? receive_process_record+0x230/0x230 [zfs]
[19408.899273]  thread_generic_wrapper+0x64/0x70 [spl]
[19408.899273]  ? __thread_exit+0x20/0x20 [spl]
[19408.899273]  kthread+0x12a/0x150
[19408.899273]  ? set_kthread_struct+0x50/0x50
[19408.899273]  ret_from_fork+0x22/0x30
[19408.899273]  </TASK>
[19408.899273] Modules linked in: tls unix_diag cdc_ether usbnet joydev mii br_netfilter xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat bpfilter overlay binfmt_misc veth nvme_fabrics vhost_vsock vmw_vsock_virtio_transport_common vsock vhost_net vhost vhost_iotlb tap xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nft_counter nf_tables nfnetlink bridge stp llc intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd kvm_amd kvm ipmi_ssif rapl efi_pstore zfs(PO) zunicode(PO) zzstd(O) zlua(O) nls_iso8859_1 zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) input_leds ccp k10temp ptdma acpi_ipmi ipmi_si mac_hid dm_multipath sch_fq_codel scsi_dh_rdac scsi_dh_emc scsi_dh_alua jc42 ipmi_devintf ipmi_msghandler msr ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath
[19408.899273]  linear raid1 hid_generic ast i2c_algo_bit drm_vram_helper drm_ttm_helper ttm usbhid drm_kms_helper hid syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd cec rc_core nvme ahci xhci_pci drm tg3 i2c_piix4 xhci_pci_renesas nvme_core libahci
[19408.903187] CR2: 0000000100000000
[19408.903574] ---[ end trace 50b300a59d683228 ]---
[19408.983169] RIP: 0010:__memcpy+0x12/0x20
[19409.013186] Code: cc cc cc cc cc cc cc cc 48 8b 05 a1 23 8e 01 c3 cc cc cc cc cc cc cc cc 0f 1f 44 00 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48 89 d1 f3 a4
[19409.014940] RSP: 0018:ffffbb491f1bb878 EFLAGS: 00010246
[19409.016205] RAX: 0000000100000000 RBX: ffffbb491f1bb928 RCX: 0000000000000800
[19409.017341] RDX: 0000000000000000 RSI: ffff9c50dc884000 RDI: 0000000100000000
[19409.018089] RBP: ffffbb491f1bb890 R08: ffff9c50dc884000 R09: 0000000000000000
[19409.019396] R10: ffff9c51bb1e7500 R11: 000000000000000c R12: 0000000000004000
[19409.019728] R13: 0000000000000000 R14: ffffbb491f1bb928 R15: 0000000000000000
[19409.020799] FS:  0000000000000000(0000) GS:ffff9c9f4c680000(0000) knlGS:0000000000000000
[19409.021727] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[19409.022719] CR2: 0000000100000000 CR3: 00000028c25c8000 CR4: 0000000000350ee0

The pool was hard-locked after that but I was able to do a regular reboot. Shortly after I got the next panic while receiving another dataset:

[  157.268524] ------------[ cut here ]------------
[  157.268528] virt_to_cache: Object is not a Slab page!
[  157.268536] WARNING: CPU: 121 PID: 198913 at mm/slab.h:413 kmem_cache_free+0x22d/0x290
[  157.268549] Modules linked in: br_netfilter unix_diag xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat bpfilter overlay binfmt_misc veth vhost_vsock vmw_vsock_virtio_transport_com                    mon vsock nvme_fabrics vhost_net vhost vhost_iotlb tap xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp nft_compat nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf                    _defrag_ipv4 nft_counter nf_tables nfnetlink bridge stp llc intel_rapl_msr intel_rapl_common amd64_edac edac_mce_amd kvm_amd ipmi_ssif kvm rapl efi_pstore zfs(PO) zunicode(PO) zzstd(O) zlua(O)                     nls_iso8859_1 zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) joydev input_leds ccp k10temp ptdma acpi_ipmi ipmi_si mac_hid sch_fq_codel jc42 ipmi_devintf dm_multipath ipmi_msghandler scsi_dh                    _rdac scsi_dh_emc scsi_dh_alua msr ip_tables x_tables autofs4 btrfs blake2b_generic zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc3                    2c raid0 multipath linear raid1 hid_generic
[  157.268647]  ast i2c_algo_bit drm_vram_helper drm_ttm_helper ttm cdc_ether drm_kms_helper usbnet mii syscopyarea sysfillrect crct10dif_pclmul sysimgblt crc32_pclmul fb_sys_fops usbhid ghash                    _clmulni_intel hid aesni_intel crypto_simd cryptd cec rc_core nvme ahci xhci_pci drm i2c_piix4 tg3 xhci_pci_renesas nvme_core libahci
[  157.268683] CPU: 121 PID: 198913 Comm: receive_writer Tainted: P           O      5.15.0-30-generic #31-Ubuntu
[  157.268688] Hardware name: TYAN B8036T65AV12E16HR/S8036GM2NE, BIOS V1.00.B20 10/31/2019
[  157.268691] RIP: 0010:kmem_cache_free+0x22d/0x290
[  157.268697] Code: 01 0f 87 00 6e 98 00 83 e3 01 0f 85 64 ff ff ff 48 c7 c6 60 11 44 bd 48 c7 c7 88 a8 7f bd c6 05 19 4b eb 01 01 e8 83 8c 96 00 <0f> 0b e9 43 ff ff ff ba 01 00 00 00 48 8d 7                    5 c8 4c 89 ff e8 4b 92
[  157.268700] RSP: 0018:ffffa564732138b8 EFLAGS: 00010286
[  157.268703] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000027
[  157.268706] RDX: ffff96454cc60988 RSI: 0000000000000001 RDI: ffff96454cc60980
[  157.268708] RBP: ffffa56473213900 R08: 0000000000000003 R09: ffffffffffe382f8
[  157.268710] R10: ffffffffffffffff R11: 0000000000000001 R12: ffff95f423424000
[  157.268712] R13: ffff95f4a3424000 R14: ffff95f423424000 R15: 0000000000004000
[  157.268715] FS:  0000000000000000(0000) GS:ffff96454cc40000(0000) knlGS:0000000000000000
[  157.268718] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  157.268720] CR2: 00001a49cdb42010 CR3: 00000028b04b4000 CR4: 0000000000350ee0
[  157.268723] Call Trace:
[  157.268726]  <TASK>
[  157.268732]  spl_kmem_cache_free+0xc9/0x130 [spl]
[  157.268751]  zio_buf_free+0x33/0x80 [zfs]
[  157.268972]  abd_return_buf+0x2d/0x40 [zfs]
[  157.269121]  zio_crypt_copy_dnode_bonus+0x11d/0x170 [zfs]
[  157.269319]  arc_buf_untransform_in_place.constprop.0+0x2a/0x40 [zfs]
[  157.269470]  arc_buf_fill+0x2c0/0x4d0 [zfs]
[  157.269620]  ? psi_task_switch+0xc6/0x220
[  157.269626]  arc_untransform+0x23/0x80 [zfs]
[  157.269776]  dbuf_read_verify_dnode_crypt+0xe7/0x180 [zfs]
[  157.269931]  dbuf_read_impl.constprop.0+0xf0/0x490 [zfs]
[  157.270085]  ? do_raw_spin_unlock+0x9/0x10 [zfs]
[  157.270235]  ? __cond_resched+0x1a/0x50
[  157.270241]  ? down_read+0x13/0x90
[  157.270244]  dbuf_read+0x1b6/0x5a0 [zfs]
[  157.270398]  dmu_tx_check_ioerr+0x6e/0xd0 [zfs]
[  157.270561]  dmu_tx_hold_free_impl+0x151/0x290 [zfs]
[  157.270729]  dmu_tx_hold_free+0x40/0x50 [zfs]
[  157.270888]  dmu_free_long_range_impl+0x120/0x350 [zfs]
[  157.271048]  dmu_free_long_range+0x74/0xc0 [zfs]
[  157.271204]  dmu_free_long_object+0x27/0xc0 [zfs]
[  157.271359]  receive_freeobjects+0x85/0x100 [zfs]
[  157.271518]  receive_process_record+0x159/0x230 [zfs]
[  157.271675]  receive_writer_thread+0xb3/0x180 [zfs]
[  157.271832]  ? receive_process_record+0x230/0x230 [zfs]
[  157.271908]  thread_generic_wrapper+0x64/0x70 [spl]
[  157.271916]  ? __thread_exit+0x20/0x20 [spl]
[  157.271923]  kthread+0x12a/0x150
[  157.271926]  ? set_kthread_struct+0x50/0x50
[  157.271928]  ret_from_fork+0x22/0x30
[  157.271932]  </TASK>
[  157.271933] ---[ end trace 0bd55b9717fb7a97 ]---

This time the pool was not frozen, but I rebooted just in case.

Receiving Pool stats:

  pool: tank2
 state: ONLINE
config:

        NAME                                             STATE     READ WRITE CKSUM
        tank2                                            ONLINE       0     0     0
          mirror-0                                       ONLINE       0     0     0
            nvme-Micron_9200_MTFDHAL3T8TCT_17451CC81F6C  ONLINE       0     0     0
            nvme-Micron_9200_MTFDHAL3T8TCT_17451CC81EBE  ONLINE       0     0     0

errors: No known data errors

NAME   PROPERTY                       VALUE                          SOURCE
tank2  size                           3.48T                          -
tank2  capacity                       18%                            -
tank2  altroot                        -                              default
tank2  health                         ONLINE                         -
tank2  guid                           4747071815621839097            -
tank2  version                        -                              default
tank2  bootfs                         -                              default
tank2  delegation                     on                             default
tank2  autoreplace                    off                            default
tank2  cachefile                      -                              default
tank2  failmode                       wait                           default
tank2  listsnapshots                  off                            default
tank2  autoexpand                     off                            default
tank2  dedupratio                     1.00x                          -
tank2  free                           2.84T                          -
tank2  allocated                      664G                           -
tank2  readonly                       off                            -
tank2  ashift                         12                             local
tank2  comment                        -                              default
tank2  expandsize                     -                              -
tank2  freeing                        0                              -
tank2  fragmentation                  0%                             -
tank2  leaked                         0                              -
tank2  multihost                      off                            default
tank2  checkpoint                     -                              -
tank2  load_guid                      13208566083222792801           -
tank2  autotrim                       on                             local
tank2  compatibility                  off                            default
tank2  feature@async_destroy          enabled                        local
tank2  feature@empty_bpobj            active                         local
tank2  feature@lz4_compress           active                         local
tank2  feature@multi_vdev_crash_dump  enabled                        local
tank2  feature@spacemap_histogram     active                         local
tank2  feature@enabled_txg            active                         local
tank2  feature@hole_birth             active                         local
tank2  feature@extensible_dataset     active                         local
tank2  feature@embedded_data          active                         local
tank2  feature@bookmarks              enabled                        local
tank2  feature@filesystem_limits      enabled                        local
tank2  feature@large_blocks           enabled                        local
tank2  feature@large_dnode            active                         local
tank2  feature@sha512                 active                         local
tank2  feature@skein                  enabled                        local
tank2  feature@edonr                  enabled                        local
tank2  feature@userobj_accounting     active                         local
tank2  feature@encryption             active                         local
tank2  feature@project_quota          active                         local
tank2  feature@device_removal         enabled                        local
tank2  feature@obsolete_counts        enabled                        local
tank2  feature@zpool_checkpoint       enabled                        local
tank2  feature@spacemap_v2            active                         local
tank2  feature@allocation_classes     enabled                        local
tank2  feature@resilver_defer         enabled                        local
tank2  feature@bookmark_v2            enabled                        local
tank2  feature@redaction_bookmarks    enabled                        local
tank2  feature@redacted_datasets      enabled                        local
tank2  feature@bookmark_written       enabled                        local
tank2  feature@log_spacemap           active                         local
tank2  feature@livelist               active                         local
tank2  feature@device_rebuild         enabled                        local
tank2  feature@zstd_compress          active                         local
tank2  feature@draid                  enabled                        local

NAME   PROPERTY              VALUE                  SOURCE
tank2  type                  filesystem             -
tank2  creation              Sat May 21 17:03 2022  -
tank2  used                  682G                   -
tank2  available             2.71T                  -
tank2  referenced            200K                   -
tank2  compressratio         1.73x                  -
tank2  mounted               yes                    -
tank2  quota                 none                   default
tank2  reservation           none                   default
tank2  recordsize            128K                   default
tank2  mountpoint            /tank2                 default
tank2  sharenfs              off                    default
tank2  checksum              sha512                 local
tank2  compression           zstd-1                 local
tank2  atime                 off                    local
tank2  devices               on                     default
tank2  exec                  on                     default
tank2  setuid                on                     default
tank2  readonly              off                    default
tank2  zoned                 off                    default
tank2  snapdir               hidden                 default
tank2  aclmode               passthrough            local
tank2  aclinherit            passthrough            local
tank2  createtxg             1                      -
tank2  canmount              on                     default
tank2  xattr                 sa                     local
tank2  copies                1                      default
tank2  version               5                      -
tank2  utf8only              on                     -
tank2  normalization         formD                  -
tank2  casesensitivity       mixed                  -
tank2  vscan                 off                    default
tank2  nbmand                off                    default
tank2  sharesmb              off                    default
tank2  refquota              none                   default
tank2  refreservation        none                   default
tank2  guid                  3540662221705091815    -
tank2  primarycache          all                    default
tank2  secondarycache        all                    default
tank2  usedbysnapshots       0B                     -
tank2  usedbydataset         200K                   -
tank2  usedbychildren        682G                   -
tank2  usedbyrefreservation  0B                     -
tank2  logbias               latency                default
tank2  objsetid              54                     -
tank2  dedup                 off                    default
tank2  mlslabel              none                   default
tank2  sync                  standard               default
tank2  dnodesize             auto                   local
tank2  refcompressratio      1.00x                  -
tank2  written               200K                   -
tank2  logicalused           1.05T                  -
tank2  logicalreferenced     69.5K                  -
tank2  volmode               default                default
tank2  filesystem_limit      none                   default
tank2  snapshot_limit        none                   default
tank2  filesystem_count      none                   default
tank2  snapshot_count        none                   default
tank2  snapdev               hidden                 default
tank2  acltype               posix                  local
tank2  context               none                   default
tank2  fscontext             none                   default
tank2  defcontext            none                   default
tank2  rootcontext           none                   default
tank2  relatime              off                    default
tank2  redundant_metadata    all                    default
tank2  overlay               on                     default
tank2  encryption            aes-256-gcm            -
tank2  keylocation           prompt                 local
tank2  keyformat             passphrase             -
tank2  pbkdf2iters           350000                 -
tank2  encryptionroot        tank2                  -
tank2  keystatus             available              -
tank2  special_small_blocks  0                      default
rincebrain commented 2 years ago

Hi #11679, nice to see you again.

zrav commented 2 years ago

@rincebrain if this is a duplicate, feel free to close. In the meantime, I had the last stack trace again...

rincebrain commented 2 years ago

It may be the most duplicated bug in recent memory. I don't think they've been being closed so far, though, since some of them are more ambiguously duplicates than others.

siilike commented 1 year ago

Another one here with 2.1.9-1~bpo11+1.

[3514154.883087] ------------[ cut here ]------------
[3514154.888230] virt_to_cache: Object is not a Slab page!
[3514154.893808] WARNING: CPU: 7 PID: 3387058 at mm/slab.h:620 kmem_cache_free+0x35d/0x410
[3514154.902133] Modules linked in: xt_nat nf_conntrack_netlink xfrm_user binfmt_misc veth tun xt_CHECKSUM xt_MASQUERADE xt_tcpudp nft_compat overlay macvlan bridge stp llc nls_ascii nls_cp437 vfat fat nft_nat nft_masq nft_fib_ipv4 nft_fib nft_chain_nat nf_nat nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_limit intel_rapl_msr intel_rapl_common ipmi_ssif x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm nft_ct irqbypass nf_conntrack ast nf_defrag_ipv6 ghash_clmulni_intel nf_defrag_ipv4 drm_vram_helper acpi_ipmi aesni_intel drm_ttm_helper cdc_ether iTCO_wdt crypto_simd ttm cryptd intel_pmc_bxt usbnet ipmi_si mei_me intel_cstate iTCO_vendor_support drm_kms_helper joydev pcspkr evdev ipmi_devintf intel_uncore efi_pstore mii i2c_algo_bit mei watchdog sg ipmi_msghandler acpi_pad intel_pmc_core acpi_power_meter button acpi_tad nf_tables crc32c_generic nfnetlink fuse drm configfs ip_tables x_tables autofs4 zfs(POE) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) icp(POE)
[3514154.902177]  zcommon(POE) znvpair(POE) spl(OE) efivarfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear md_mod sr_mod cdrom hid_generic usbhid hid uas usb_storage sd_mod ixgbe xfrm_algo ahci xhci_pci dca libahci nvme mdio_devres xhci_hcd libata nvme_core libphy t10_pi intel_ish_ipc intel_lpss_pci i2c_i801 crc64_rocksoft crc32_pclmul ptp scsi_mod crc64 usbcore intel_lpss crc32c_intel i2c_smbus intel_ishtp pps_core crc_t10dif idma64 mdio usb_common crct10dif_generic scsi_common crct10dif_pclmul crct10dif_common fan video
[3514155.046131] CPU: 7 PID: 3387058 Comm: receive_writer Tainted: P           OE      6.0.0-0.deb11.6-amd64 #1  Debian 6.0.12-1~bpo11+1
[3514155.058687] Hardware name: GIGABYTE MX33-BS1-V1/MX33-BS1-V1, BIOS F04a 10/01/2021
[3514155.066642] RIP: 0010:kmem_cache_free+0x35d/0x410
[3514155.071823] Code: fd ff ff 80 3d d0 4f 6f 01 00 0f 85 d7 fe ff ff 48 c7 c6 00 de a3 ba 48 c7 c7 48 30 d6 ba c6 05 b5 4f 6f 01 01 e8 29 c9 67 00 <0f> 0b 5b 5d 41 5c 41 5d 41 5e c3 cc cc cc cc 48 8b 57 60 48 8b 4d
[3514155.091324] RSP: 0018:ffffac7894ee3a30 EFLAGS: 00010286
[3514155.097026] RAX: 0000000000000000 RBX: ffff93be21f74000 RCX: 0000000000000000
[3514155.104658] RDX: 0000000000000001 RSI: ffffffffbad7c802 RDI: 00000000ffffffff
[3514155.112293] RBP: ffff93baf07a6400 R08: 0000000000000000 R09: 00000001000005a4
[3514155.119896] R10: ffffac7894ee38d0 R11: ffffffffbb4c2a20 R12: ffff93bea1f74000
[3514155.127489] R13: ffffffffc0d838a0 R14: 0000000000000000 R15: ffffffffc0d838c0
[3514155.135082] FS:  0000000000000000(0000) GS:ffff93c9dfbc0000(0000) knlGS:0000000000000000
[3514155.143617] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[3514155.149814] CR2: 00005595f4b1eac0 CR3: 00000001ac810001 CR4: 0000000000772ee0
[3514155.157405] PKRU: 55555554
[3514155.160553] Call Trace:
[3514155.163446]  <TASK>
[3514155.165983]  spl_kmem_cache_free+0x12e/0x1d0 [spl]
[3514155.171212]  arc_buf_fill+0x3f5/0xce0 [zfs]
[3514155.175867]  ? dbuf_dirty+0x498/0x9c0 [zfs]
[3514155.180520]  arc_untransform+0x1d/0x80 [zfs]
[3514155.185261]  dbuf_read_verify_dnode_crypt+0xf2/0x160 [zfs]
[3514155.191191]  dbuf_read_impl.constprop.0+0x52a/0x6d0 [zfs]
[3514155.197027]  ? percpu_counter_add_batch+0x58/0xc0
[3514155.202150]  ? dbuf_cons+0xa7/0xc0 [zfs]
[3514155.206513]  ? spl_kmem_cache_alloc+0x9f/0x7d0 [spl]
[3514155.211887]  ? dbuf_rele_and_unlock+0x132/0x690 [zfs]
[3514155.217361]  ? preempt_count_add+0x70/0xa0
[3514155.221853]  ? preempt_count_add+0x70/0xa0
[3514155.226345]  ? _raw_spin_lock+0x13/0x40
[3514155.230570]  ? aggsum_add+0x178/0x190 [zfs]
[3514155.235160]  dbuf_read+0xe5/0x5f0 [zfs]
[3514155.239402]  ? dnode_hold_impl+0x4eb/0xee0 [zfs]
[3514155.244439]  ? preempt_count_add+0x70/0xa0
[3514155.248921]  dmu_bonus_hold_by_dnode+0x86/0x1a0 [zfs]
[3514155.254376]  receive_object+0x410/0xca0 [zfs]
[3514155.259146]  ? dnode_rele_and_unlock+0x59/0xf0 [zfs]
[3514155.264508]  ? preempt_count_add+0x70/0xa0
[3514155.268976]  ? __thread_exit+0x20/0x20 [spl]
[3514155.273616]  receive_writer_thread+0x1cc/0xad0 [zfs]
[3514155.278972]  ? __slab_free+0xa1/0x2d0
[3514155.283000]  ? set_next_task_fair+0x2d/0xd0
[3514155.287533]  ? _raw_spin_unlock_irqrestore+0x23/0x40
[3514155.292849]  ? receive_process_write_record+0x1a0/0x1a0 [zfs]
[3514155.298972]  ? __thread_exit+0x20/0x20 [spl]
[3514155.303669]  ? thread_generic_wrapper+0x56/0x70 [spl]
[3514155.309066]  thread_generic_wrapper+0x56/0x70 [spl]
[3514155.314273]  kthread+0xe5/0x110
[3514155.317761]  ? kthread_complete_and_exit+0x20/0x20
[3514155.322885]  ret_from_fork+0x1f/0x30
[3514155.326789]  </TASK>
[3514155.329331] ---[ end trace 0000000000000000 ]---