Enhance Scrub to Match File Storage to Current Dataset Property Configuration

fredcooke commented 3 years ago

Describe the feature would like to see added to OpenZFS

Something like scrub, fsck, resilver to scan a dataset with a known property change and correct the underlying storage. As an automation engineer it physically hurts my brain to do things like:

cp -ar dir dir2
rm -rf dir
mv dir2 dir

or similar

zfs snapshot
zfs clone
zfs rename
zfs destroy

Or something like that (if that would even work).

IMO it should be possible to scan a dataset and validate that each file's underlying storage honours the current properties of the dataset that it's in, and correct it if not.

This affects at least copies= but I thought I read the docs for other properties that had similar "only what is written after the change" which is totally fair as to set something like that and have the system go off on an automatic/instant binge "correcting" things is also ugly. Better to be able to trigger it on demand.

I envisage this as an extension to "scrub" which currently only verifies integrity, not config-to-implementation correctness.

How will this feature improve OpenZFS?

Allows space to be reclaimed in situations where free space is smaller than the dataset involved
Allows space to be reclaimed by actively used directory structures without stopping a running application and restarting it after manually correcting things
Allows space to be freed without undue wear and tear on the disks rewriting data that is already on them just to work around the inability to correct this with some sort of scan

Additional context

My situation is that I set up my first non-mirrored zpool on an SSD (4TB PNY CS3030 TLC high endurance) with two nested datasets having copies=2 not because that does any good at all on an SSD, but because I was going to be initially doing send/receive to a single un-mirrored 4TB spinning rust drive (ironwolf pro) that I don't trust much simply because spinning rust. In the event of a short-medium term SSD failure I wanted those two datasets to stand a better chance of being recovered from the single rust drive without being silently corrupted.

On order is a second 4TB ironwolf pro that I will add to the first to form a 2-way mirror for the SSD to sync snapshots to with send/receive.

Once that arrives I will set the two datasets in question to copies=1 and I just found out today that I will not recover any space without the manual hacks outlined above and was somewhat displeased :-D I need to write some more data to these datasets now and I'm faced with the following choice:

New data is written with copies=2 and space cannot be easily recovered (or at all if it gets too full)
Set copies=1 now and new data is written to what is still a 2-disk sync pair one of which is spinning rust such that if the SSD fails before the new drive is added to the vdev as a mirror I can no longer have faith in my ability to recover all of the data uncorrupted (I understand that ZFS will tell me if there are issues such as this and that I won't be ignorant of them).

Clearly this is not a fun choice to have to make.

Perhaps what I want is already possible with some combination of zdb commands and I've just failed to find it in my research? If so, great - it's not a frequent operation for anyone, so that would be just fine.

Finally, I'm happy to have a crack at this (and #12024) if someone more versed in the code base can work with me to formulate a plan and advise as I go. It would be my great pleasure to contribute to this fine project :-)

gmelikov commented 3 years ago

Technically it's a duplicate of at least https://github.com/openzfs/zfs/issues/9762 (it requires the same mechanism - block pointer rewrite).

fredcooke commented 3 years ago

Thanks @gmelikov - that issue is titled and primarily about compression (which I thought I recalled having the same "only after" characteristics but didn't bother to confirm and list) whereas I intentionally wrote this one up as "all storage configurations" despite only actually caring about copies= right now. Open sauce to the bone, you might say. On that basis I'd like it left open, however it's good to have that link on this and concerning that it's so old. If you think of any others that call out all or part of the description please do add them. Before I posted I specifically looked for copies in all open issues and found nothing relevant, hence bothering to write it up in detail.

Aside from copies and compression, are there any other properties that act this way, and if so, what are they? Cheers :-)

fredcooke commented 3 years ago

Encryption, too, perhaps.

AllKind commented 3 years ago

One of the topics, that pops up frequently in zfs-discussion list, is the ability to re-balance pools after extending storage. Always ending in pointing at the stalled development of block pointer rewrite.

fredcooke commented 3 years ago

Thanks @AllKind - please see my comment on the tail end of the ticket gmelikov mentioned: https://github.com/openzfs/zfs/issues/9762#issuecomment-841955353

fredcooke commented 3 years ago

I need to write some more data to these datasets now and I'm faced with the following choice:

Well, it turns out that I'm past that choice without realising it. The copies=2 was by-design inherited by my two real data sets from a parent dual and I renamed them out of there for a shorter/more convenient mount path rather than setting explicit mount paths and contrary to the word rename, they lost their properties on move to a new parent. Not a problem and the new disk is on the table, but no enclosure for it yet, so it'll be minimal time exposed to danger. Just not intuitive with that command naming, but in hindsight reasonable to expect that.

openzfs / zfs