Closed mischaelschill closed 3 years ago
I just found out that it was actually different files named so similarily that I erroneously thought they to be one and the same
Also, it doesn't scan files but extents. As long as changes to the same file add new extents (which is normal in btrfs because it is cow), bees will scan that data: This data is actually new, it won't scan the same file over and over again.
It does seem to have to "scan" the file for the new extents, even if it does not read the data. These files might well have beyond a hundred thousand extents each.
btrfs provides a mechanism for scanning only "new" metadata in subvols; however, the resolution of "new" is a metadata page, each of which can hold references to up to 300 extents. So if you have a big file, and you're modifying random blocks, and you modify more than 0.3% of the file between scans, then all of the metadata pages are "new" and show up in the next scan's metadata stream. (edit: the overhead for this is pretty low, since we can eliminate the 99.7% of old extents very quickly and scan only the 0.3% new ones; however, it will increase the crawl time).
The "Scanning" status usually implies reading the blocks at the same time. By the time we get to "scanning" we already know the location of a new data block to be read. The metadata search where btrfs tells us where to find the new data is called "crawling" in bees, and it usually runs far too fast to see (milliseconds per hour).
If the dedupe splits extents into unique and non-unique extents, then the next pass over the subvol will scan the new unique extents. These will have the same locations within the file, but different physical locations on disk.
oops, wrong button ;)
I have fairly large VM images in snapshots on the drive running bees. I constantly grep the .status file for "Scanning " to see where the crawler is currently at. I discovered that the crawler scans the same image file (in the same snapshot) over and over, even though it is not in use and therefore not changing. It does so even in the (read-only) snapshots. Is this expected behavior? It seems it never gets to the point where it looks at another file, although a full scan of the file requires hours and I haven't yet observed it for days.
I use bees v0.6.3 on a btrfs volume with 6 (ssd) disks in raid1 and kernel 5.9.9