kakra
commented
2 days ago First, I think the progress table should be separated by a new line, or have a header like the other sections, e.g. PROGRESS: or ESTIMATES:.
Now I've updated bees, then changed scan mode to 4, and restarted it without touching any of the beeshome files. I've also setup various windows to watch beesstats, the bees log, and htop.
I do not see how it possibly starts scanning all the subvols and snapshots again, I cannot imagine it being that fast:
Switching to extent scan starts over from the beginning, even if previous subvol scans were up to date; however, extent scan mode is fast enough that this is not necessarily an inconvenience.
There are no new lines in beescrawl.dat. I'd expect a new type of line appearing:
Older bees versions will delete extent scan checkpoints from beescrawl.dat.
Maybe I just have to wait for some time? According to progress, it is probably done:
done 4.814T %done size transid 4059507 todo ETA
-------- -------- -------- ---- ------- ------- ---- ---
deferred 2.894T finished max 4059506 4059507 - -
deferred 799.222G finished 32M 4059506 4059507 - -
deferred 337.118G finished 8M 4059506 4059507 - -
deferred 226.404G finished 2M 4059506 4059507 - -
deferred 221.007G finished 512K 4059506 4059507 - -
deferred 382.818G finished 128K 4059505 4059506 - -I've restarted bees hoping that it forces writing the beescrawl.dat but still no new entry format. I'll let it sit for a while now.
According to htop, bees is mostly writing data and not reading anything. This is probably just writing to the state files.
According to journalctl, bees writes absolutely no logs (except version and option parsing) which comes at a surprise. I'm using --verbose 5 but I didn't expect it to be that silent now.
Is it maybe still scanning and the "progress" shows a bogus status until then? I can see the progress stats changing every now and then... It now looks like this:
done 4.814T %done size transid 4059523 todo ETA
-------- -------- -------- ---- ------- ------- ---- ---
deferred 2.429T finished max 4059522 4059523 - -
deferred 723.453G finished 32M 4059522 4059523 - -
deferred 455.517G finished 8M 4059522 4059523 - -
deferred 408.836G finished 2M 4059522 4059523 - -
deferred 346.682G finished 512K 4059522 4059523 - -
deferred 508.406G finished 128K 4059521 4059522 - -
lilydjwg
Zygo
manxorist
I'm getting ready to do the v0.11 release, and this one is larger than most of the previous releases, so I'd like to do more testing before putting a tag on it.
Please try it out, and comment here if it works for you, or open an issue if it doesn't.
Here's a list of the new things to try:
Extent scan mode
Extent scan mode replaces the historically problematic subvol scan modes.
Extent scan is now the default mode. If you are using a
-moption, remove it or replace it with-m4to enable extent scan.Extent scan enables several new features:
Extent size sorting
Skip over the small extents to dedupe the large extents first.
The filesystem is divided into 6 size groups: 0 to 128K (which includes all compressed extents), 512K, 2M, 8M, 32M, and larger than 32M.
There's a table in
bees-roots.ccwhich lists the size groups. Ambitious users who want to avoid small extent dedupes entirely can remove the table entry for the small extent sizes without breaking the dedupe.Progress reporting
One of the most requested features!
Estimates how much data is done and how much remains at each size level, how long the remaining data will take, and when it might be finished.
The columns are:
finishedat 100%Unfortunately, it's about as accurate as the estimated time remaining in Windows Copy, i.e. not very accurate at all...
The progress report appears in
$BEESSTATUS, the debug log, andbeesstats.txt. It is only available in extent scan mode.Duplicate read elimination
With extent scan mode, bees can read an extent once, then never read it again. That saves a lot of time when scanning a filesystem that has a lot of snapshots and not much duplicate data.
Extent scan doesn't have to start over when new snapshots are created. It doesn't have to read every snapshot or reflink multiple times as the subvol scanners do.
Duplicate data still needs to be reread over and over, but that's a limitation of the kernel API.
Minimal temporary space usage and unreachable extents
Extent scan adds copy extents to the hash table and processes them immediately, without waiting for later scans to clean them up. This avoids leaving a temporary extent sitting in the hash table for so long that it is evicted before the extent can be completely deduped.
No support for btrfs send workaround
Currently the extent scan mode can't effectively implement
--workaround-btrfs-send. Read-only snapshots are simply excluded from dedupe when the workaround is enabled.Whether the
--workaroundis enabled or not, ifbtrfs sendis active on the filesystem, no dedupe can be done on the subvol being sent. Any space savings will only appear when the read-only snapshots are deleted.One solution for active
btrfs sendis to terminate bees before startingbtrfs send, and restart bees afterbtrfs sendis finished.Unlike subvol scan modes, there is no way to go back to rescan a read-only subvol that becomes read-write (other than deleting
beescrawl.datand starting over from the very beginning of the filesystem).Extent scan vs subvol scan history
Switching to extent scan starts over from the earliest completed subvol scan. If your filesystem was completely up to date with subvol scans, it will also be up to date with extent scan; however, if subvol scans were only partially completed, then extent scan will restart from the beginning.
It is possible to switch from extent to subvol scan modes and back. Extent scan will pause while subvol scan is active, and vice versa. If you needed to switch back to subvol scan mode, please tell us why!
Older bees versions will delete extent scan checkpoints from
beescrawl.dat.The extent scan mode stores its checkpoint using virtual subvol IDs in
beescrawl.dat, withrootnumbers 250 through 255. This may change in the future.[edited to correct "from the beginning" and fill in some answers from the comments]
Other enhancements and bug fixes
Less fragmentation and nuisance dedupes
bees will no longer dedupe a partial extent if the total amount of data saved is less than half of the extent size, or if the extent requires more than 100 dedupe and copy operations to remove.
There will be no more "we can save 24K if we make 128M of copy extents" nuisance dedupes that happen in media files, or "we can save 80% of the space in this extent, but only if we break the extent into thousands of tiny fragments" dedupes that happen in VM image files.
bees will still split an extent when the ratio is the other way around, e.g. "create a few dozen new 4K fragments if it enables dedupe to save 100M of space" or "split a 4M chunk into a 3M chunk and a 1M chunk so we can dedupe the 3M chunk."
bees now picks the longest available extent for dedupe if multiple choices exist.
Removing expensive toxic extent workarounds
Toxic extents are fixed in kernel 5.7 and later, and the workarounds for toxic extents in bees were extremely expensive. The worst of these workarounds have been removed. Some filesystems will see improvements of 100x to 1000x on highly duplicated data.
Improved IO scheduling
Only one thread reads from disk at any time, avoiding thrashing on spinning drives (and it even improves performance on some solid-state drives).
Dedupes run on data that has been preloaded into page cache, making dedupe operations substantially faster since the sequential read buffers were removed from kernel 4.20.
Dedupes now pass the full extent length to the kernel, instead of cutting them into 16M fragments. The kernel has been able to handle any length of dedupe request since 4.18.
There is now a simple filter to prevent duplicate prereads.
Improved dedupe task scheduling
Crawlers are now much better at finding work that worker threads can do at the same time.
Lower extent reference count limit and kernel memory usage
Decreased the maximum number of extent refs from 699050 to 9999. This uses much smaller buffers for kernel ioctls, which avoids triggering some annoying kernel VM bugs that forcibly evict pages unrelated to bees from memory.
New minimum kernel requirements
The extent scan mode won't work at all on kernel 4.14 and earlier. Only subvol scans can be used there.
Toxic extent workarounds have been removed. Kernels earlier than 5.7 may experience slowdowns and lockups with large files that have many duplicate extents.