amotin
commented
1 day ago @scratchings Have you checked to what replication process 594865 triggering the panic belongs? Is it remote raw/encrypted or local unencrypted one? The error 5 returned by dmu_bonus_hold_by_dnode() is EIO. But bonus buffers reads do not need actual read disk that could fail. And for encrypted receive it should not decrypt, which could produce the EIO error. I have difficulty to understand what may produce the EIO error there.
Is the target pool your local replication receives to also encrypted or plaintext? I wonder if the panic may be happening on that plain-text receive, since one unlike raw receive does decrypt on receive, and it may produce EIO there as part of decryption or authentication.
If those guesses give nothing, I wonder if it is possible to try bpftrace SET_ERROR() calls inside dmu_bonus_hold_by_dnode() with 5 as argument. Stack trace from that, if we could get it could show where the error is coming from.
PS: Just as a random thought: haven't you observed random disconnects and replication restarts just before panics by chance? I suppose for remote system it may happen more often than for local.
scratchings
System information
Describe the problem you're observing
Panic reported for pool receiving snapshots into encrypted pool
Describe how to reproduce the problem
Use Syncoid/Sanoid to receive (and prune) snapshots from remote servers on a cron schedule. Wait some time, crash will occur.
As requested, this is a new ticket reporting a related issue to the closed ticket #11679.
As described in #11679, this WAS using my exclusive lock python script to prevent multiple receives to the same pool, but as I am transferring data out of this pool to a LUKS encrypted pool on a different JBOD, the effected pool is being used as the source of another internal send/receive via syncoid (and several rsync sources which are also transferring data out) (source dataset is not the same as the dataset being received into). These other egress syncoids are continuing to run and the syncoid that most likely caused the panic remains in the process table, but the dataset is not increasing in size:
Include any warning/errors/backtraces from the system logs