bees exceptions plus some help needed to handle snapper read-only snapshots

[darkbasic]

Hi, I decided to try bees on my Fedora 33 system with a Samsung 970 EVO SSD, so I removed all my read-only snapshots and run btrfs filesystem defragment -r -v -czstd /mnt from a live usb. At this point I had a disk usage of about 18.36GiB, while after running bees I've got 17.12GiB, so it worked.

(btw, I'm not sure if the previous defrag command behaves like a compress-force=zstd mount option, but I guess it's not: how can I achieve the same result of compress-force when I defrag?)

Weirdely, it threw some exceptions (see attached file): bees_exceptions.txt

At this point I wanted to re-enable my snapper read-only snapshots: it snapshots my subvolumes on an hourly basis and I usually keep ~48 hourly snapshots, 7 daily snapshots, 4 weekly snapshots and 3 monthly snapshots. That means that a snapshot can take up to 3 months before it gets deleted. So far my experience with deduplication and read-only snapshots has been terrible, because I've always ended up using more space than before, but I've never tried bees: do you think it can handle something like this?

[Zygo]

btrfs fi defrag first filters extents by length, and the defaults are set to minimize work, so it won't recompress extents if they are above a particular size. You might need to set the target extent size (-t) very large, like 1GB. (This might not be sufficient. btrfs fi defrag uses a number of heuristics and they aren't all obvious.)

The exceptions look like a kernel regression present in:

• 5.4.64..5.4.65
• 5.8.8..5.8.9
• 4.19.144..4.19.145

Upgrade (or downgrade) to a kernel outside of those version ranges and the exceptions should go away. Normally these exceptions are there to allow bees to skip processing inconsistent data about a file if it is modified while bees is looking at it, so they may appear if the files are in active use; however, the kernel bug introduces garbage into search results so it will trigger a lot of extra exceptions for inactive files. The garbage doesn't harm any data, but it does reduce dedupe efficiency because bees can't analyze the affected portions of the filesystem structure.

bees can handle snapshots. They are treated like reflink copies and previously deduped files. You can enable special handling for read-only snapshots, or you can dedupe read-only snapshots the same as any other subvol.

Snapshots add references to extents, and adding more references makes bees run slower and use temporary space longer during dedupe; however, they shouldn't affect the final dedupe efficiency by more than a few percent once all the data has been scanned. An EVO SSD with a single-user workload should be fine--it is neither large nor slow.

When snapshots are created rapidly, we might want to use -m1 scanning mode. The incremental scanning feature of bees relies on completely scanning every subvol before updating the minimum transid pointer for future scans. If snapshots are created too quickly to scan completely, then the default -m0 mode keeps getting forced back to the beginning of the filesystem and no forward process is made. The -m1 scan mode guarantees all subvols make forward progress so it will eventually catch up.

You can use the --workaround-btrfs-send option to completely ignore read-only snapshots, and only dedupe within read-write subvols. This saves time if you intend to delete all the read-only subvols. Storage will increase while the snapshots exist because any extents that have to be split for dedupe will consume extra space on disk until the snapshots are deleted. In your case that probably isn't worthwhile, as it will take months to recover space and your disk is more than fast enough to process everything without excluding read-only snapshots. bees could be smarter about this (e.g. choose the read-only snapshot version of an extent when multiple versions are available) but that capability is planned for a future version.

[darkbasic]

The exceptions look like a kernel regression present in:

I'm already on 5.8.11 so I guess this is not the kernel regression which is causing the exceptions.

I'll try with -m1 scanning mode and without --workaround-btrfs-send and I'll let you know how it goes, thanks!

[Zygo]

OK if I look more closely I see an inode change on one of the exceptions (i.e. the file was being scanned while something else renamed a different file under the same name), one is a sqlite journal file that may only exist during a database update, and the others could be files that were modified as bees happened to scan. The one where a constraint check fails against max compressed extent length I've never seen without the kernel bug, but I guess it is possible under just the right conditions (if an old, no-longer valid compressed extent is recorded in the hash table, but a newer extent at the same location on disk is not, i.e. just a special case of data update).

The general bees approach is to be paranoid about checking everything, backing away quickly when any inconsistency is detected, no matter how harmless or routine such an inconsistency might be. This is done by throwing a C++ "exception", which is a technical term for non-local return from a function call, not necessarily an indication of an error or problem. As the exception is processed, the data leading up to the consistency check is logged for possible debugging later. Maybe it should simply be called something else in the log messages..."exception" seems to be a very scary word to a lot of users. Maybe after the first one we also log a few sentences of explanation or a doc URL that discusses the constraint checking. I don't want to disable them completely because these logs are the first step in debugging high resource usage and kernel bug issues.

An active filesystem will always throw a small number of constraint-check exceptions, usually more on smaller filesystems where bees scan loops are shorter and other applications are more likely to be accessing the same data. If there are so many exceptions that bees can't get any work done, or there's a bees bug that is creating exceptions where they shouldn't be, only then do they become a problem.