zfs umount fails to unmount in the correct order and zfs diff fails

[siilike]

System information

Type	Version/Name
Distribution Name	Debian
Distribution Version	bullseye
Kernel Version	5.18.0-0.bpo.1-amd64
Architecture	amd64
OpenZFS Version	zfs-2.1.5-1~bpo11+1

Describe the problem you're observing

Firstly, there is an issue regarding mount order and zfs umount:

# zfs list data2/test -r -o name,mountpoint,mounted
NAME                 MOUNTPOINT            MOUNTED
data2/test           /data2/test           yes
data2/test/a         /data2/test/a         no
data2/test/a/b       /data2/test/a/b       no
data2/test/a/b/c     /data2/test/a/b/c     no
data2/test/a/b/c/d   /data2/test/a/b/c/d   no
data2/test/a/b/c/d2  /data2/test/a/b/c/d2  no
data2/test/a/b/c2    /data2/test/a/b/c2    no
data2/test/a/b2      /data2/test/a/b2      no
data2/test/a2        /data2/test/a2        yes
# zfs mount data2/test/a/b/c/d
# zfs mount data2/test/a/b/c
# zfs umount data2/test/a/b/c/d
cannot unmount '/data2/test/a/b/c/d': no such pool or dataset
# zfs umount data2/test/a/b/c
cannot unmount '/data2/test/a/b/c/d': no such pool or dataset
# umount /data2/test/a/b/c/d
umount: /data2/test/a/b/c/d: no mount point specified.
# umount /data2/test/a/b/c
# umount /data2/test/a/b/c/d

When mounting a parent dataset after the child dataset zfs umount fails to unmount either of them properly, so they have to be unmounted manually in the correct order with umount.

Secondly, I have also seen:

zfs umount parent
cannot open 'parent/child@backup_2022-07-17_10:35:01': snapshot delimiter '@' is not expected here
cannot open 'parent@backup_2022-07-17_10:35:01': snapshot delimiter '@' is not expected here
cannot unmount '/parent/child2': no such pool or dataset

but could not reproduce at this time. Likely some snapshots were mounted while running zfs umount.

Thirdly, there is an issue regarding zfs diff:

zfs diff -H parent/child@backup_2022-07-17_10:20:01 parent/child@backup_2022-07-17_10:35:01
Cannot stat /parent/child/.zfs/shares/: unable to generate diffs

When all datasets are mounted using zfs mount -a running zfs mount parent/child temporarily resolves the issue.

When only parent/child is mounted it does not help.

The issue appears on a server that is concurrently receiving snapshots from several other servers using syncoid.

The latest received snapshots are held, mounted, diffed with the held snapshots from the previous run, changed files copied, unmounted, the held snapshots from the previous run released.

The pool is encrypted.

Also present on the same system: #13598

[rincebrain]

The way this is written sounds like you're mounting /parent after /parent/child, which is, because that's how UNIX mountpoints work, going to hide /parent/child from being visible for all intents and purposes.

If you don't want to allow it to hide mounts like that, you can set overlay=off, but then the mount will just fail.

[siilike]

That makes sense for the 2nd one. But it should still be possible to unmount them.

[rincebrain]

Usually you pass in a reference to the mountpoint you want to remove.

Since you can't "see" the mountpoint to get a pointer to it, you can't do that, and if you ask it to umount the currently visible thing at /foo/bar, there's nothing mounted there.

This isn't ZFS-specific, this is a problem if you mount over any existing FS mounts.

[siilike]

I was referring to:

Thirdly, zfs umount fails:

zfs umount parent
cannot unmount 'parent/child': no such pool or dataset

zfs umount parent/child
cannot unmount 'parent/child': no such pool or dataset

I never use mountpoints for unmounting, so I haven't tried whether umount /parent worked.

[rincebrain]

Sorry, are you objecting to the fact that umount parent fails after you mounted parent over parent/child, or that umount parent/child fails?

I could see implementing a behavior change to allow it to try the former anyway even if it thinks it'll fail based on all the information it has, but the latter doesn't seem easily resolvable, as I understand how this works. I suppose that could be messy in practice, though...as mountpoint interactions tend to be.

[siilike]

The issue is that after mounting parent/child and then parent it is impossible to unmount either of them using zfs umount. The error message that parent/child does not exist is also very confusing, even if this is somehow the expected behaviour.

It actually is possible to resolve this with umount:

# zfs list data2/test -r -o name,mountpoint,mounted
NAME                 MOUNTPOINT            MOUNTED
data2/test           /data2/test           yes
data2/test/a         /data2/test/a         no
data2/test/a/b       /data2/test/a/b       no
data2/test/a/b/c     /data2/test/a/b/c     no
data2/test/a/b/c/d   /data2/test/a/b/c/d   no
data2/test/a/b/c/d2  /data2/test/a/b/c/d2  no
data2/test/a/b/c2    /data2/test/a/b/c2    no
data2/test/a/b2      /data2/test/a/b2      no
data2/test/a2        /data2/test/a2        yes
# zfs mount data2/test/a/b/c/d
# zfs mount data2/test/a/b/c
# zfs umount data2/test/a/b/c/d
cannot unmount '/data2/test/a/b/c/d': no such pool or dataset
# zfs umount data2/test/a/b/c
cannot unmount '/data2/test/a/b/c/d': no such pool or dataset
# umount /data2/test/a/b/c/d
umount: /data2/test/a/b/c/d: no mount point specified.
# umount /data2/test/a/b/c
# umount /data2/test/a/b/c/d

[ErikBjare]

I just stumbled into this and got stuck (zfs 2.1.7 on Arch Linux). The umount trick by @siilike worked for me.

Such that, for /tank/home/user that is stuck/cannot be unmounted, running umount /tank/home; umount /tank/home/user resolves the stuck state.