search

Lakshmipathi / dduper

Fast block-level out-of-band BTRFS deduplication tool.
GNU General Public License v2.0
168 stars 18 forks source link

Assertion error #48

Open LoZio opened 3 years ago

LoZio commented 3 years ago

Running code from today's repo Gen-15-2021 (dduper 0.04)

btrfs_lookup_csums search failed.icrosoft.MicrosoftOfficeHub_8wekyb3d8bbwe/AC/Microsoft/CLR_v4.0/ngen.log
 Error: btrfs_lookup_csumextent buffer leak: start 87928078336 len 16384
extent buffer leak: start 55155425280 len 16384
extent buffer leak: start 305545216 len 16384
Traceback (most recent call last):
  File "/usr/local/bin/dduper", line 575, in <module>
    main(results)
  File "/usr/local/bin/dduper", line 465, in main
    dedupe_dir(results.dir_path, results.dry_run, results.recurse)
  File "/usr/local/bin/dduper", line 456, in dedupe_dir
    dedupe_files(file_list, dry_run)
  File "/usr/local/bin/dduper", line 410, in dedupe_files
    ret = do_dedupe(src_file, dst_file, dry_run)
  File "/usr/local/bin/dduper", line 225, in do_dedupe
    assert len(out2) != 0
AssertionError

It ran half a day analyzing a path then crashed. Command line is dduper --device /dev/sdq --dir /mypath/backup/ --analyze --recurse

LoZio commented 3 years ago

And anothere here, on a different path, but strangely again a a CLR file

/Microsoft.MicrosoftOfficeHub_8wekyb3d8bbwe/AC/Microsoft/CLR_v4.0btrfs_lookup_csums search failed.
 Error: btrfs_lookup_csumextent buffer leak: start 87928242176 len 16384
extent buffer leak: start 55155425280 len 16384
extent buffer leak: start 305545216 len 16384
Traceback (most recent call last):
  File "/usr/local/bin/dduper", line 575, in <module>
    main(results)
  File "/usr/local/bin/dduper", line 465, in main
    dedupe_dir(results.dir_path, results.dry_run, results.recurse)
  File "/usr/local/bin/dduper", line 456, in dedupe_dir
    dedupe_files(file_list, dry_run)
  File "/usr/local/bin/dduper", line 410, in dedupe_files
    ret = do_dedupe(src_file, dst_file, dry_run)
  File "/usr/local/bin/dduper", line 225, in do_dedupe
    assert len(out2) != 0
AssertionError
Lakshmipathi commented 3 years ago

Can you trying on these files individually and check the status? Are you using Subvolume? I added this assert especially when dduper runs into subvolume.

LoZio commented 3 years ago

There was no subvolume involved here, singole partition btrfs formatted. Currently I have no access to the server, probably next week. The other test I made was to check if files were somewhat damaged but I could download/zip/copy them with no problem.

VTiTux commented 3 years ago

Same issue. The -d "Dir" is inside a subvolume, but there is no nested subvolumes inside this Dir, and there is not snapshot anywhere in this FS. With BTRFS, everythings are inside subvolumes. The top-level subvolume (id 5) cannot be removed.

Lakshmipathi commented 3 years ago

Currently I have no access to the server, probably next week

Okay, thanks. Single partition - it should. I guess its something related to file. Can you share the output or stat filename and filefrag -e filename ?

Lakshmipathi commented 3 years ago

Same issue. The -d "Dir" is inside a subvolume, but there is no nested subvolumes inside this Dir, and there is not snapshot anywhere in this FS. With BTRFS, everythings are inside subvolumes. The top-level subvolume (id 5) cannot be removed.

dduper works fine with top-level subvolume. Can you share the output for failing file.


dduper --device /dev/sda1 --files /file1 /file2 --dry-run
``
aplund commented 3 years ago

I have the same problem. My btrfs.static has version 'btrfs-progs v5.7' and when you do the inspect-internal dump-csum on a file, there is no output to stdout, even though it returns success.

abadger commented 3 years ago

I may be having the same issue. But I don't seem to have output from dump-csum with any file on my btrfs filesystem. I'm using the btrfs.static that is in the git repo:

(run) [root@camelback bin]# btrfs inspect-internal dump-csum /bin/more /dev/mapper/luks-90a6412f-c588-46ca-9118-5aca35943d25
(run) [root@camelback bin]# stat more
  File: more
  Size: 45248           Blocks: 96         IO Block: 4096   regular file
Device: 23h/35d Inode: 105399      Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:bin_t:s0
Access: 2021-03-28 10:50:30.171868026 -0400
Modify: 2021-01-07 05:21:36.000000000 -0500
Change: 2021-03-28 04:20:54.485295462 -0400
 Birth: 2021-03-28 04:20:54.485295462 -0400
(run) [root@camelback bin]# filefrag -e more
Filesystem type is: 9123683e
File size of more is 45248 (12 blocks of 4096 bytes)
 ext:     logical_offset:        physical_offset: length:   expected: flags:
   0:        0..      11:    6781429..   6781440:     12:             last,encoded,eof
more: 1 extent found
(run) [root@camelback bin]# mount |grep '/ '
/dev/mapper/luks-90a6412f-c588-46ca-9118-5aca35943d25 on / type btrfs (rw,relatime,seclabel,compress=zstd:3,ssd,space_cache,subvolid=256,subvol=/root)

Something that may be helpful: I ran the test/verify.sh script and it worked. So I modified the script to keep the loopback btrfs img mounted. I then entered that directory and observed the following:

(run) [root@camelback btrfs_mnt]# cp /bin/more .
(run) [root@camelback btrfs_mnt]# btrfs inspect-internal dump-csum /tmp/btrfs_mnt/more /dev/loop1
(run) [root@camelback btrfs_mnt]# sync
(run) [root@camelback btrfs_mnt]# btrfs inspect-internal dump-csum /tmp/btrfs_mnt/more /dev/loop1
cc2b2ca9 817f228e d39b1b93 27ea92ed 836a09df 3340f8bc b89342ae 6950ff3 
60936024 c576f12b 2cd90fa3 49c1878e (run) [root@camelback btrfs_mnt]#

I thought that it might be one of the mount options on my main filesystem that is causing this but mounting with these options doesn't appear to have made a difference:

(run) [root@camelback tmp]# mount /tmp/img /tmp/btrfs_mnt/ -orw,relatime,seclabel,compress=zstd:3,ssd,space_cache
(run) [root@camelback tmp]# cd btrfs_mnt/
(run) [root@camelback btrfs_mnt]# cp /bin/more .
(run) [root@camelback btrfs_mnt]# btrfs inspect-internal dump-csum /tmp/btrfs_mnt/more /dev/loop1
(run) [root@camelback btrfs_mnt]# sync
(run) [root@camelback btrfs_mnt]# btrfs inspect-internal dump-csum /tmp/btrfs_mnt/more /dev/loop1
7414aaf2 d953ad9d 841da37f 15906a01 2e82545 6add4ec4 fffffff6 ffffffff 
d0600080 3 3f8300 1800 (run) [root@camelback btrfs_mnt]#

My main filesystem is on a luks2 device but I haven't tested whether using luks2 with a loop back device will reproduce this problem.

bugsyb commented 6 months ago

More errors like this, as I did run into it too and reported issue here: https://github.com/Lakshmipathi/dduper/issues/69#issuecomment-2060954380

Did run couple of tests:

For all tests BTRFS sits on LUKS2 and it is aarch64 (OSCM/Debian based distro).

bugsyb commented 6 months ago

Further investigation shows that for file located on root volume, csum can be obtained and for the file on subvolume it doesn't return anything.

Lakshmipathi commented 6 months ago

@bugsyb : Subvolume issue listed here https://github.com/Lakshmipathi/dduper?tab=readme-ov-file#known-issues Sorry I was unable to figure out the issue, its been years passed. I'll see whether I can allocate sometime on this issue.

bugsyb commented 6 months ago

Appreciate it. Yes whilst doing research did end up on Known Issues though hopes were created as under one of the tickets you've mentioned that work is being done on it and... it is more and more common to use subvolumes.

Thanks!