Possible regression when receiving crypted datasets/snapshots

[olidal]

System information

Type	Version/Name
Distribution Name	debian / proxmox
Distribution Version	10.8 / pve 6.3-1
Linux Kernel	5.4.103-1-pve
Architecture	amd64
ZFS Version	2.0.3-pve2
SPL Version	2.0.3-pve2

Describe the problem you're observing

• When receiveing a first crypted snapshot (to create the receiving dataset), the receive fails with error cannot receive: failed to read from stream, even though the sanpshot was successfully created

• When trying to receive subsequent snapshots (20+ total) of previous dataset using a recursive replication stream, the receive fails with error cannot receive incremental stream: most recent snapshot of <receiving dataset name> does not match incremental source. However one snapshot is received (next one in chromological order)

• When trying again the same receive with same parameters, I am able to receive all snapshots, one at a time, each time with the same error message (except at the end)

Describe how to reproduce the problem

# ssh user@example.com 'zfs send -Rw zpool/vol@2020-11-09-215243' | zfs recv -F -u zpool/priv/test3
cannot receive: failed to read from stream
# zfs list -r -t all zpool/priv/test3
NAME                                 USED  AVAIL     REFER  MOUNTPOINT
zpool/priv/test3                    25.2M  5.08T     25.2M  /zpool/priv/test3
zpool/priv/test3@2020-11-09-215243     0B      -     25.2M  -

# ssh user@example.com 'zfs send -Rw -I zpool/vol@2020-11-09-215243 zpool/vol@2021-03-14-215000' | zfs recv -F -u zpool/priv/test3
cannot receive incremental stream: most recent snapshot of zpool/priv/test3 does not
match incremental source
# zfs list -r -t all zpool/priv/test3
NAME                                 USED  AVAIL     REFER  MOUNTPOINT
zpool/priv/test3                    28.0M  5.08T     25.2M  /zpool/priv/test3
zpool/priv/test3@2020-11-09-215243  2.88M      -     25.2M  -
zpool/priv/test3@2020-12-06-004500     0B      -     25.2M  -

The exact same command execute without error on another machine with kernel 5.4.78-2-pve and zfs/spl 0.8.5-pve1: with the same source dataset, first receive works as exepcted, and the second is succesful receiving all the snapshots in one run.

Binary file of zfs command are identical in both systems:

# file /usr/sbin/zfs
/usr/sbin/zfs: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=7d039fd5a13b51e63b63f957e55345781abd1213, stripped

Following the properties of the receiving dataset:

# zfs get all zpool/priv/test3
NAME              PROPERTY              VALUE                  SOURCE
zpool/priv/test3  type                  filesystem             -
zpool/priv/test3  creation              Sun Mar 14 21:45 2021  -
zpool/priv/test3  used                  28.0M                  -
zpool/priv/test3  available             5.08T                  -
zpool/priv/test3  referenced            25.2M                  -
zpool/priv/test3  compressratio         3.28x                  -
zpool/priv/test3  mounted               no                     -
zpool/priv/test3  quota                 none                   default
zpool/priv/test3  reservation           none                   default
zpool/priv/test3  recordsize            128K                   received
zpool/priv/test3  mountpoint            /zpool/priv/test3      default
zpool/priv/test3  sharenfs              off                    default
zpool/priv/test3  checksum              on                     default
zpool/priv/test3  compression           lz4                    inherited from zpool
zpool/priv/test3  atime                 on                     default
zpool/priv/test3  devices               on                     default
zpool/priv/test3  exec                  on                     default
zpool/priv/test3  setuid                on                     default
zpool/priv/test3  readonly              off                    default
zpool/priv/test3  zoned                 off                    default
zpool/priv/test3  snapdir               hidden                 default
zpool/priv/test3  aclinherit            restricted             default
zpool/priv/test3  createtxg             7291                   -
zpool/priv/test3  canmount              on                     default
zpool/priv/test3  xattr                 sa                     inherited from zpool
zpool/priv/test3  copies                1                      default
zpool/priv/test3  version               5                      -
zpool/priv/test3  utf8only              off                    -
zpool/priv/test3  normalization         none                   -
zpool/priv/test3  casesensitivity       sensitive              -
zpool/priv/test3  vscan                 off                    default
zpool/priv/test3  nbmand                off                    default
zpool/priv/test3  sharesmb              off                    default
zpool/priv/test3  refquota              none                   default
zpool/priv/test3  refreservation        none                   default
zpool/priv/test3  guid                  14678652092881917073   -
zpool/priv/test3  primarycache          all                    default
zpool/priv/test3  secondarycache        all                    default
zpool/priv/test3  usedbysnapshots       2.88M                  -
zpool/priv/test3  usedbydataset         25.2M                  -
zpool/priv/test3  usedbychildren        0B                     -
zpool/priv/test3  usedbyrefreservation  0B                     -
zpool/priv/test3  logbias               latency                default
zpool/priv/test3  objsetid              4261                   -
zpool/priv/test3  dedup                 off                    default
zpool/priv/test3  mlslabel              none                   default
zpool/priv/test3  sync                  standard               default
zpool/priv/test3  dnodesize             legacy                 default
zpool/priv/test3  refcompressratio      3.38x                  -
zpool/priv/test3  written               0                      -
zpool/priv/test3  logicalused           69.0M                  -
zpool/priv/test3  logicalreferenced     66.4M                  -
zpool/priv/test3  volmode               default                default
zpool/priv/test3  filesystem_limit      none                   default
zpool/priv/test3  snapshot_limit        none                   default
zpool/priv/test3  filesystem_count      none                   default
zpool/priv/test3  snapshot_count        none                   default
zpool/priv/test3  snapdev               hidden                 default
zpool/priv/test3  acltype               posixacl               inherited from zpool/priv
zpool/priv/test3  context               none                   default
zpool/priv/test3  fscontext             none                   default
zpool/priv/test3  defcontext            none                   default
zpool/priv/test3  rootcontext           none                   default
zpool/priv/test3  relatime              on                     inherited from zpool
zpool/priv/test3  redundant_metadata    all                    default
zpool/priv/test3  overlay               off                    default
zpool/priv/test3  encryption            aes-256-gcm            -
zpool/priv/test3  keylocation           prompt                 local
zpool/priv/test3  keyformat             passphrase             -
zpool/priv/test3  pbkdf2iters           342K                   -
zpool/priv/test3  encryptionroot        zpool/priv/test3       -
zpool/priv/test3  keystatus             unavailable            -
zpool/priv/test3  special_small_blocks  0                      default

And the pool:

# zpool get all zpool
NAME   PROPERTY                       VALUE                          SOURCE
zpool  size                           7.23T                          -
zpool  capacity                       0%                             -
zpool  altroot                        -                              default
zpool  health                         ONLINE                         -
zpool  guid                           815103344673130543             -
zpool  version                        -                              default
zpool  bootfs                         -                              default
zpool  delegation                     on                             default
zpool  autoreplace                    off                            default
zpool  cachefile                      -                              default
zpool  failmode                       wait                           default
zpool  listsnapshots                  off                            default
zpool  autoexpand                     off                            default
zpool  dedupditto                     0                              default
zpool  dedupratio                     1.00x                          -
zpool  free                           7.22T                          -
zpool  allocated                      14.9G                          -
zpool  readonly                       off                            -
zpool  ashift                         12                             local
zpool  comment                        -                              default
zpool  expandsize                     -                              -
zpool  freeing                        0                              -
zpool  fragmentation                  0%                             -
zpool  leaked                         0                              -
zpool  multihost                      off                            default
zpool  checkpoint                     -                              -
zpool  load_guid                      8252958448737607247            -
zpool  autotrim                       off                            default
zpool  feature@async_destroy          enabled                        local
zpool  feature@empty_bpobj            active                         local
zpool  feature@lz4_compress           active                         local
zpool  feature@multi_vdev_crash_dump  enabled                        local
zpool  feature@spacemap_histogram     active                         local
zpool  feature@enabled_txg            active                         local
zpool  feature@hole_birth             active                         local
zpool  feature@extensible_dataset     active                         local
zpool  feature@embedded_data          active                         local
zpool  feature@bookmarks              enabled                        local
zpool  feature@filesystem_limits      enabled                        local
zpool  feature@large_blocks           enabled                        local
zpool  feature@large_dnode            enabled                        local
zpool  feature@sha512                 enabled                        local
zpool  feature@skein                  enabled                        local
zpool  feature@edonr                  enabled                        local
zpool  feature@userobj_accounting     active                         local
zpool  feature@encryption             active                         local
zpool  feature@project_quota          active                         local
zpool  feature@device_removal         enabled                        local
zpool  feature@obsolete_counts        enabled                        local
zpool  feature@zpool_checkpoint       enabled                        local
zpool  feature@spacemap_v2            active                         local
zpool  feature@allocation_classes     enabled                        local
zpool  feature@resilver_defer         enabled                        local
zpool  feature@bookmark_v2            enabled                        local

(Features are identical in the two systems.)

Include any warning/errors/backtraces from the system logs

No error in dmesg output, nor /var/log/syslog nor /var/log/kern.log.

[stale[bot]]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.