search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.6k stars 1.75k forks source link

ZFS Receive of encrypted incremental data stream causes a PANIC #13445

Open cyberpower678 opened 2 years ago

cyberpower678 commented 2 years ago

System information

Type Version/Name
Distribution Name Debian
Distribution Version Bullseye
Kernel Version 5.10.109+truenas
Architecture amd64
OpenZFS Version zfs-2.1.2-95_g1d2cdd23b zfs-kmod-2.1.2-95_g1d2cdd23b

Describe the problem you're observing

During an incremental receive, ZFS caused a panic and a system hangup.

Describe how to reproduce the problem

It happens randomly.

Include any warning/errors/backtraces from the system logs

[456678.240841] VERIFY3(0 == dmu_object_set_blocksize(rwa->os, drro->drr_object, drro->drr_blksz, drro->drr_indblkshift, tx)) failed (0 == 95)
[456678.243815] PANIC at dmu_recv.c:1776:receive_object()
[456678.245141] Showing stack for process 2936808
[456678.246532] CPU: 10 PID: 2936808 Comm: receive_writer Tainted: P           OE     5.10.109+truenas #1
[456678.247840] Hardware name: Supermicro X9QR7-TF+/X9QRi-F+/X9QR7-TF+/X9QRi-F+, BIOS 3.0b 05/20/2015
[456678.249138] Call Trace:
[456678.250421]  dump_stack+0x6b/0x83
[456678.251676]  spl_panic+0xd4/0xfc [spl]
[456678.253038]  ? arc_buf_access+0x14c/0x250 [zfs]
[456678.254276]  ? dnode_hold_impl+0x4e9/0xef0 [zfs]
[456678.255493]  ? dnode_set_blksz+0x13b/0x300 [zfs]
[456678.256677]  ? dnode_rele_and_unlock+0x5c/0xc0 [zfs]
[456678.257846]  receive_object+0xc2c/0xca0 [zfs]
[456678.258984]  ? dmu_object_next+0xd6/0x120 [zfs]
[456678.260098]  ? receive_writer_thread+0xbd/0xad0 [zfs]
[456678.261160]  ? kfree+0x40c/0x480
[456678.262202]  ? _cond_resched+0x16/0x40
[456678.263244]  receive_writer_thread+0x1cc/0xad0 [zfs]
[456678.264280]  ? thread_generic_wrapper+0x62/0x80 [spl]
[456678.265252]  ? kfree+0x40c/0x480
[456678.266242]  ? receive_process_write_record+0x190/0x190 [zfs]
[456678.267177]  ? thread_generic_wrapper+0x6f/0x80 [spl]
[456678.268092]  thread_generic_wrapper+0x6f/0x80 [spl]
[456678.268988]  ? __thread_exit+0x20/0x20 [spl]
[456678.269864]  kthread+0x11b/0x140
[456678.270706]  ? __kthread_bind_mask+0x60/0x60
[456678.271538]  ret_from_fork+0x22/0x30
amotin commented 1 year ago

@sfatula Yes, that is the plan: https://ixsystems.atlassian.net/browse/NAS-122583 .

sfatula commented 1 year ago

Am I to understand this issue only presents when using raw sends? i.e., if I decrypt and re-encrypt on a send/recv, the issue should not manifest?

amotin commented 1 year ago

My PR fixes only raw send issues. Non-raw does not care about indirect block size and should not be affected. I am not sure "re-encrypt" part actually works (while in theory it I think it could), but if yes, it should work.