Misleading error "kernel modules must be upgraded to receive this stream" when send and receive are on the same kernel

[rptb1]

System information

Type	Version/Name
Distribution Name	Ubuntu
Distribution Version	22.04
Kernel Version	5.15.0-86-generic
Architecture	x86_64
OpenZFS Version	zfs-2.1.5-1ubuntu6~22.04.1

zfs-kmod-2.1.5-1ubuntu6~22.04.1

Describe the problem you're observing

When sending an encrypted replication stream from one pool to another on the same machine, zfs receive exits with a message "cannot receive incremental stream: kernel modules must be upgraded to receive this stream".

This error is both false and misleading.

This has come up several times before in #6706 , #7024 , #6547 , #6574 but those are all closed and it still seems to happen.

I'm raising it again because today the message caused a delay in restoring a server, and it'd be good if that didn't happen to anyone else.

Describe how to reproduce the problem

Send and receive an encrypted replication stream, then create a snapshot in the destination while the receive is in progress.

Workaround

For my situation, I worked around this by adding -o com.sun:auto-snapshot=false on the receive, but other people will have other snapshot-creating systems that have to be suppressed.

Suggestions

1. Change the message to describe the actual problem(s), and not make false suggestions. At the very least, delete the second part of the message "kernel modules must be upgraded to receive this stream". Obviously something more descriptive would be better though.

2. Lock the receiving filesystem, making it immutable until the receive has completed (or aborted).

Thanks all!

[joshenders]

Seeing this myself on Debian Linux: 6.1.0-15-amd64:

$ zfs version
zfs-2.2.2-1~exp1
zfs-kmod-2.2.2-1~exp1

Repro steps:

# zfs send -wRpV private/self@zfs-auto-snap_daily-2023-12-25-0148 | pv | zfs recv -s backup/private/self
# zfs send -t '<token>' | zfs recv backup/private/self
cannot receive resume stream: invalid backup stream
# zfs send -t '<token>' | zfs recv -s backup/private/self
cannot receive resume stream: kernel modules must be upgraded to receive this stream.

[1dolla]

Is anyone looking at this?

I set up a pool solely for backing up another pool. Then created a snapshot on the main pool, and replicated it:

zfs send -R --verbose --raw decky/Bridge@replication-2024-07-30-07.04.42 | ssh backup.local zfs recv -Fsdu bridge

This went fine. Both are running macOS, both up-to-date with current stable binaries.

Now, trying to do an incremental update fails,

zfs send --raw -vRi "$latestSentSnapshot" "$snap" | ssh backup.local zfs recv -Fsv bridge/Bridge

with exit code 1. Doing it manually gives the error mentioned here, no useful info given. I have no clue what's wrong, and my boss is pissed. I advocated for zfs.

Tried the workaround here, zfs send --raw -vRi "$latestSentSnapshot" "$snap" | ssh backup.local zfs recv -o com.sun:auto-snapshot=false -Fsv bridge/Bridge

Same deal. I don't know what's the actual problem, and I don't know how to fix it. The error message is unhelpful.

[IvanVolosyuk]

    err = ioctl_err = lzc_receive_with_cmdprops(destsnap, rcvprops,
        oxprops, wkeydata, wkeylen, origin, flags->force, flags->resumable,
        raw, infd, drr_noswap, -1, &read_bytes, &errflags,
        NULL, &prop_errors);
.......
    if (err && (ioctl_errno == ENOENT || ioctl_errno == EEXIST)) {
........
        if (gather_nvlist(hdl, destsnap, NULL, NULL, B_FALSE, B_TRUE,
            B_FALSE, B_FALSE, B_FALSE, B_FALSE, B_FALSE, B_FALSE,
            B_TRUE, &local_nv, &local_avl) == 0) {
............
            if (fs != NULL) {
                if (flags->verbose) {
                    (void) printf("snap %s already exists; "
                        "ignoring\n", destsnap);
                }
                err = ioctl_err = recv_skip(hdl, infd,
                    flags->byteswap);
............

    if (ioctl_err != 0) {
        switch (ioctl_errno) {
           ...........
        case EINVAL:
            if (flags->resumable) {
                zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
                    "kernel modules must be upgraded to "
                    "receive this stream."));
            } else if (embedded && !raw) {
                zfs_error_aux(hdl, dgettext(TEXT_DOMAIN,
                    "incompatible embedded data stream "
                    "feature with encrypted receive."));
            }
            (void) zfs_error(hdl, EZFS_BADSTREAM, errbuf);
            break;

Not sure what is going on here as I'm not expert. I would try receive without '-s' option, and send without '-R' as well. Can you also tell more if source or destination or both are encrypted?