size of BTRFS images

[joergmlpts]

Hello, I cannot make sense of the size of images for one particular file system. Maybe you have a simple explanation.

The image file sizes for backups of BTRFS partitions appear large. This is an example:

$ ls -l nvme0n1p6.img
-rw-r----- 1 user user 21168964854 Aug 12 06:48 nvme0n1p6.img

where partclone.info reports these sizes:

$ partclone.info -L /dev/null -s nvme0n1p6.img
Partclone v0.3.24 http://partclone.org
Showing info of image (nvme0n1p6.img)
File system:  BTRFS
Device size:  420.0 GB = 25634752 Blocks
Space in use:  16.7 GB = 1020947 Blocks
Free Space:   403.3 GB = 24613805 Blocks
Block size:   16384 Byte

image format:    0002
created on a:    64 bits platform
with partclone:  v0.3.23
bitmap mode:     BIT
checksum algo:   CRC32
checksum size:   4
blocks/checksum: 640
reseed checksum: yes

I cannot see how the image file can be 21,168,964,854 bytes long for a partition with 1,020,947 blocks in use. Based on the file format description, the expected size calculates to:

           110  header
     3,204,332  bitmap = ceil(25634752 / 8)
             4  bitmap's CRC32
16,727,195,648  1020947 blocks in use * 16384 block size
         6,384  1596 CRC32's * 4
--------------
16,730,406,478

The actual image size is 26.5% larger than expected!

Do you understand what is going on? Is the number of in-use blocks incorrectly reported?

Thanks!

[Thomas-Tsai]

Hi,

Thank you for this detailed information and report. We know the in-use number is mismatched. The main reason is the number is different between btrfs superblock and bitmap. Usually, some of btrfs metadata are not included in used blocks from superblock information, but partclone will mark it as used.

The problem is that partclone does not provide the bitmap test result so we may add this feature in partclone.info.

Thank you!

[joergmlpts]

Thank you so much for your reply, Thomas!

I have looked into the source code and found that the number of in-use blocks from the bitmap is counted right after the bitmap is obtained. The bitmap's number of blocks in use is written as part of the image header.

The file format description calls the number at offset 76 bitmap's size, in bytes but actually this is the number of blocks in use in the bitmap, field used_bitmap.

In my example the number of blocks in use at offset 76 of the header is 1,291,855 and with this number of blocks the image file size makes sense.

Thanks again!

[Thomas-Tsai]

Hi,

We have a new commit for this issue. To get used blocks in the bitmap with the -d2 option.

partclone.info -d2 -s btrfs.img

Showing info of image (btrfs.img)
File system:  BTRFS
Device size:  108.3 GB = 6611925 Blocks
Space in use:  24.6 GB = 1500853 Blocks
Free Space:    83.7 GB = 5111072 Blocks
Block size:   16384 Byte
Used Blocks in Bitmap:   1580977 Blocks

image format:    0002
created on a:    64 bits platform
with partclone:  v0.3.25
bitmap mode:     BIT
checksum algo:   CRC32
checksum size:   4
blocks/checksum: 64
reseed checksum: yes

Thank you!

[joergmlpts]

Hi Thomas-Tsai, thank you for committing a fix!

The file format description has not been corrected yet. At offset 76 is the number of unsed blocks based on the bitmap. The number at that offset is certainly not the size of the bitmap in bytes. Since your commit did not cover the file format description as well I am going to send a merge request for it. Please kindly consider it.

Thank you!

[Thomas-Tsai]

Sure, no problem. Thanks for your contribution!

[joergmlpts]

Hi Thomas, your commit enables me, with a debug option, to print the number of blocks that are actually written.

But the broader issue remains that for BTRFS partclone reports incorrect used blocks and free space values. When partclone -c saves a partition it writes many more blocks than stated. One symptom is that its output remains for a long time at "Completed: 99.99%" - while it is writing all the blocks not accounted for.

A fix should make partclone report the used blocks and free space based on the bitmap instead of the super-block. The blocks based on the bitmap are what partclone actually reads and writes.

I just gave it a try and came up with this:

The main idea is to simply rename two fields in partclone.h. The field that represents the used blocks from the bitmap will be named usedblocks and the one from the super-block gets a different name.

Note that only field names within partclone's code are renamed. The image files stay exactly the same.

--- a/src/partclone.h
+++ b/src/partclone.h
@@ -181,10 +181,10 @@ typedef struct
        unsigned long long totalblock;

        /// Number of blocks in use as reported by the file system
-       unsigned long long usedblocks;
+       unsigned long long superBlockUsedBlocks;

        /// Number of blocks in use in the bitmap
-       unsigned long long used_bitmap;
+       unsigned long long usedblocks;

        /// Number of bytes in each block
        unsigned int  block_size;

With this change, partclone.c needs to assign the used blocks from the bitmap to the field with the new name:

-- a/src/partclone.c
+++ b/src/partclone.c
@@ -1222,7 +1222,7 @@ void update_used_blocks_count(file_system_info* fs_info, unsigned long* bitmap)
                        ++used;
        }

-       fs_info->used_bitmap = used;
+       fs_info->usedblocks = used;
 }

and your recent commit needs to be changed as well:

@@ -1907,7 +1907,7 @@ void print_file_system_info(file_system_info fs_info, cmd_opt opt) {
        unsigned int     block_s = fs_info.block_size;
        unsigned long long total = fs_info.totalblock;
        unsigned long long used  = fs_info.usedblocks;
-       unsigned long long used_bitmap  = fs_info.used_bitmap;
+       unsigned long long superBlockUsedBlocks = fs_info.superBlockUsedBlocks;
        int debug = opt.debug;
        char size_str[11];

@@ -1926,7 +1926,7 @@ void print_file_system_info(file_system_info fs_info, cmd_opt opt) {
        log_mesg(0, 0, 1, debug, _("Free Space:   %s = %llu Blocks\n"), size_str, (total-used));

        log_mesg(0, 0, 1, debug, _("Block size:   %i Byte\n"), block_s);
-       log_mesg(2, 0, 1, debug, _("Used Blocks in Bitmap:   %llu Blocks\n"), used_bitmap);
+       log_mesg(2, 0, 1, debug, _("Used Blocks in Super-Block:   %llu Blocks\n"), superBlockUsedBlocks);
 }

Then all what is left to do is make read_super_blocks assign the super block's idea of the used block count to the new field name. I decided against changing all the filesystem-specific read_super_blocks functions and instead added assignments after the 4 calls of this function in main.c:

--- a/src/main.c
+++ b/src/main.c
@@ -205,6 +205,7 @@ int main(int argc, char **argv) {

                /// get Super Block information from partition
                read_super_blocks(source, &fs_info);
+               fs_info.superBlockUsedBlocks = fs_info.usedblocks;

                if (img_opt.checksum_mode != CSM_NONE && img_opt.blocks_per_checksum == 0) {

@@ -308,6 +309,7 @@ int main(int argc, char **argv) {

                /// get Super Block information from partition
                read_super_blocks(source, &fs_info);
+               fs_info.superBlockUsedBlocks = fs_info.usedblocks;

                check_mem_size(fs_info, img_opt, opt);

@@ -339,6 +341,7 @@ int main(int argc, char **argv) {
                if (dfr != 0){
                    fs_info.device_size = get_partition_size(&dfr);
                    read_super_blocks(source, &fs_info);
+                   fs_info.superBlockUsedBlocks = fs_info.usedblocks;
                }else{
                    if (target_stdout) {
                        log_mesg(0, 1, 1, debug, "%s, %i, stdout is not supported\n", __func__, __LINE__);
@@ -352,6 +355,7 @@ int main(int argc, char **argv) {
                        fs_info.device_size = get_free_space(target);
                    }
                    read_super_blocks(target, &fs_info);
+                   fs_info.superBlockUsedBlocks = fs_info.usedblocks;
                }
                img_opt.checksum_mode = opt.checksum_mode;
                img_opt.checksum_size = get_checksum_size(opt.checksum_mode, opt.debug);

It seems to work! With very few code changes partclone finally reports the correct used blocks and free space:

$ partclone.btrfs -c ...
...
File system:  BTRFS
Device size:  275.0 GB = 16784320 Blocks
Space in use:  19.5 GB = 1189626 Blocks
Free Space:   255.5 GB = 15594694 Blocks
Block size:   16384 Byte
Elapsed: 00:18:20, Remaining: 00:00:00, Completed: 100.00%, Rate:   1.06GB/min, 
Current block:   16777217, Total block:   16784320, Complete: 100.00%           
Total Time: 00:18:20, Ave. Rate:   1.06GB/min, 100.00% completed!
Syncing... OK!
Partclone successfully cloned the device (/dev/sda4) to the image (-)
Cloned successfully.

$ partclone.info -d2
Partclone v0.3.25 http://partclone.org
Showing info of image (-)
File system:  BTRFS
Device size:  275.0 GB = 16784320 Blocks
Space in use:  19.5 GB = 1189626 Blocks
Free Space:   255.5 GB = 15594694 Blocks
Block size:   16384 Byte
Used Blocks in Super-Block:   995053 Blocks

image format:    0002
created on a:    64 bits platform
with partclone:  v0.3.25
bitmap mode:     BIT
checksum algo:   CRC32
checksum size:   4
blocks/checksum: 64
reseed checksum: yes

What do you think? Does this change make sense to you?

Thanks!

[Thomas-Tsai]

Thank You! I merged your code, and it makes sense for partclone.