Cannot use deferred destroy for snapshots: dataset is busy

[mpeter50]

System information

Type	Version/Name
Distribution Name	Proxmox VE
Distribution Version	8.2.4
Kernel Version	6.8.4-3-pve
Architecture	x86_64
OpenZFS Version	zfs-2.2.4-pve1

Describe the problem you're observing

I'm trying to delete a snapshot that I dont need anymore. It has been opened in a process (virtiofsd, used for sharing directories to a virtual machine) so I can not delete it with default options, but I figured I dont need to have the snapshot deleted right now, its fine if ZFS deletes it when it can, so I added the -d option to my command.

However, running this command results in no change to what is happening: ZFS tells me that the snapshot is in use:

$ sudo zfs destroy -d mypool/Archive/Backup@temp_archive2_athelyezett_fajlok_fel_archive2-mappaba_kesz
cannot destroy snapshot mypool/Archive/Backup@temp_archive2_athelyezett_fajlok_fel_archive2-mappaba_kesz: dataset is busy

I expected that the command would succeed, and that the defer_destroy property of the snapshot becomes on, however it errors and the property stays off.

Describe how to reproduce the problem

1. Create a snapshot of a dataset
2. Open a file in that snapshot in a process
3. Try to delete the snapshot with the -d option
4. The operation fails

Include any warning/errors/backtraces from the system logs

I dont receive such messages in the system logs

[rincebrain]

Neither on nor off is a valid value for feature@async_destroy.

If you have it set to disabled, it will act like the feature doesn't exist, including erroring if you try to make use of it. If it's enabled, the feature can be used, but none of the on-disk changes that it makes are present, e.g. there's no "todo: delete me" currently present. If it's active, then it both can be used and actively has changes present on disk.

Disabled can be set to enabled with zpool upgrade or a manual explicit zpool set, but enabled will never go back to disabled. (Note that zpool upgrade will, unless you set compatibility, set all features it knows about to enabled, so you probably don't want to do that just for a single feature flag...)

[mpeter50]

Neither on nor off is a valid value for feature@async_destroy.

Sorry, I meant the defer_destroy "dataset" property of the snapshot. The feature@async_destroy has been already enabled on the pool, according to zpool get.

[AllKind]

from zfs-destroy(8):

The given snapshots are destroyed immediately if and only if the zfs destroy command without the -d option would have destroyed it. Such immediate destruction would occur, for example, if the snapshot had no clones and the user-initiated reference count were zero.

So given that information, I would guess your snapshot does not meet above conditions, which would mark it for deferred destruction. So zfs destroy sees no reason to mark it as such. After doing the checks zfs simply tries to destroy the snapshot but fails because a process still uses it.

[mpeter50]

If a process still uses it, shouldnt that count as a user-initiated reference? Honestly, I dont know what that means.

But then, the documentation of the -d option says this:

-d  Destroy immediately.  If a snapshot cannot be destroyed now, mark it for deferred destruction.

As I understand, the snapshot should have been marked for deferred destruction, as it could not have been destroyed right at that moment.

[mpeter50]

But also this, between your excerpt and mine in the man page:

If a snapshot does not qualify for immediate destruction, it is marked for deferred deletion. In this state, it exists as a usable, visible snapshot until both of the preconditions listed above are met, at which point it is destroyed.

It tells either immediate or deferred destruction should happen.

[AllKind]

If a process still uses it, shouldnt that count as a user-initiated reference? Honestly, I dont know what that means.

Honestly me neither. Gotta love man pages ;-)

But i.e. a clone is some kind of reference. Something zfs can keep track of in itself. What zfs doesn't do is start some kind of running in the background loop to check, if that particular process is still running until it could delete the snapshot. That is up to the user.

So I think what happens is what I said before: It checks the (zfs internal) conditions which would qualify for deferred destroy and if it does not find them, try a normal "immediate" destroy. Then bails out in error because the filesystem is still in use.

What is not clear to me from reading the man page is if "deferred destruction" only happens if you use the -d parameter. If deferred destruction is also applied without -d, it would be redundant. So I think the man page is poorly written in that regard.

[mpeter50]

What is not clear to me from reading the man page is if "deferred destruction" only happens if you use the -d parameter. If deferred destruction is also applied without -d, it would be redundant. So I think the man page is poorly written in that regard.

From when I have first read it, I had to re-read it a few times too, but until you pointed this out I didnt notice this. The first paragraph tells when will an immediate destruction occur, the second paragraph tells that otherwise it will do a deferred deletion, and the -d option's documentation tells that it will do an immediate destruction or fall back to deferred deletion, which is the same as described what happens without the flag.

[rincebrain]

So, to be specific.

-d, if specified, sets cb_defer_destroy to TRUE when zfs destroy calls zfs_destroy_snaps_nvl, which returns EINVAL if you call it on not a snapshot (In other functions, that shouldn't come up in this specific one), and should mark it for deferred destruction if it's held/cloned otherwise. (Where deferred destruction means "it will be destroyed when the clones/holds go away", not the same "we're destroying it slowly in the background" sense of async_destroy. So not the same semantics as zfs destroy, without -d - that should, in theory, error out the way we're seeing here)

So EBUSY seems like a surprising response for that. Perhaps the logic handling that broke sometime. It wouldn't astonish me. Or maybe it's busy in a way that it doesn't defer for. (It's not clear to me if the intended sense of "hold" here is "zfs hold" or if it also intends to mean "snapshot in use" more broadly. Based on this experience, and the code comments, I would rather guess the former.)