search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.54k stars 1.74k forks source link

Support for FIDEDUPERANGE #11065

Open luke-jr opened 4 years ago

luke-jr commented 4 years ago

Linux 5.4 added an ioctl to inform filesystems of duplicate data, for deduplication. It would be nice if ZFS supported it.

Use case: a cron job could regularly find data to deduplicate without the memory overhead of current ZFS deduplication.

PrivatePuffin commented 3 years ago

Are you sure that said function is compatible with ZFS? Not part of LVM or something?

Rydako commented 3 years ago

https://man7.org/linux/man-pages/man2/ioctl_fideduperange.2.html

"This ioctl operation first appeared in Linux 4.5. It was previously known as BTRFS_IOC_FILE_EXTENT_SAME and was private to Btrfs."

Isn't LVM specific, and is already used by BTRFS it seems.

"Both files must reside within the same filesystem" I'm by no means an expert on the inner workings of ZFS, but wouldn't this caveat mean this could only be used to dedup within a single dataset?

luke-jr commented 3 years ago

Are you sure that said function is compatible with ZFS?

At least in some contexts, it should be? ZFS already deduplicates snapshots, for example.

I'm not familiar with the more generalised existing ZFS deduplication, but I don't know why it wouldn't prove this function is possible.

"Both files must reside within the same filesystem" I'm by no means an expert on the inner workings of ZFS, but wouldn't this caveat mean this could only be used to dedup within a single dataset?

If you add such a limitation, it might be nice to have an option to not enforce it. :)

shodanshok commented 3 years ago

This basically is a rehash of reflink support. A much needed feature, I must say...

strugee commented 1 year ago

Note that this issue is theoretically unblocked since reflink support has landed.