abort in find_dupes:584

KrissN commented 5 years ago

Installed latest (eda1437e3) duperemove on Gentoo and gave it around 350G to work with. Unfortunately duperemove aborted:

Thread 2 "pool" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff70b5700 (LWP 22627)]
0x00007ffff790082b in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff790082b in raise () from /lib64/libc.so.6
#1  0x00007ffff78e9855 in abort () from /lib64/libc.so.6
#2  0x00005555555614f1 in compare_extents (orig_file=orig_file@entry=0x55555583ecc0, orig_file_block=orig_file_block@entry=0x5555613a09c0, walk_file=walk_file@entry=0x55555583ecc0, walk_file_block=walk_file_block@entry=0x555561414880, search_len=<optimized out>,
    res=res@entry=0x7fffffffda70) at find_dupes.c:584
#3  0x0000555555561786 in search_extent (file=file@entry=0x55555583ecc0, extent=0x7ffff00b0c90, dupe_hashes=dupe_hashes@entry=0x7fffffffda40, dupe_extents=dupe_extents@entry=0x7fffffffda70, db=db@entry=0x7ffff0001348) at find_dupes.c:704
#4  0x0000555555561a0e in search_file_extents (file=file@entry=0x55555583ecc0, dupe_hashes=dupe_hashes@entry=0x7fffffffda40, dupe_extents=dupe_extents@entry=0x7fffffffda70) at find_dupes.c:760
#5  0x0000555555561a79 in find_dupes_thread (ctxt=<optimized out>, priv=<optimized out>) at find_dupes.c:786
#6  0x00007ffff7ea3833 in ?? () from /usr/lib64/libglib-2.0.so.0
#7  0x00007ffff7eac735 in ?? () from /usr/lib64/libglib-2.0.so.0
#8  0x00007ffff763c0e3 in start_thread () from /lib64/libpthread.so.0
#9  0x00007ffff79ced5f in clone () from /lib64/libc.so.6
(gdb) frame 2
#2  0x00005555555614f1 in compare_extents (orig_file=orig_file@entry=0x55555583ecc0, orig_file_block=orig_file_block@entry=0x5555613a09c0, walk_file=walk_file@entry=0x55555583ecc0, walk_file_block=walk_file_block@entry=0x555561414880, search_len=<optimized out>,
    res=res@entry=0x7fffffffda70) at find_dupes.c:584
584             abort_on(block->b_parent != orig->b_parent);
(gdb) p *block
$1 = {b_parent = 0x555561391a20, b_file = 0x55555583ecc0, b_loff = 2298855424, b_flags = 0, b_list = {next = 0x555561419a60, prev = 0x5555614199a0}, b_file_next = {__rb_parent_color = 93825134229200, rb_right = 0x0, rb_left = 0x0}, b_head_list = {next = 0x555561419a88,
    prev = 0x5555614199c8}}
(gdb) p *orig
$2 = {b_parent = 0x55555dc37740, b_file = 0x55555583ecc0, b_loff = 354025472, b_flags = 0, b_list = {next = 0x55555dc4b8c0, prev = 0x55555dc4b800}, b_file_next = {__rb_parent_color = 93825065777841, rb_right = 0x5555613a5bb0, rb_left = 0x0}, b_head_list = {
    next = 0x55555dc4b8e8, prev = 0x55555dc4b828}}
(gdb) p *block->b_parent
$3 = {dl_node = {__rb_parent_color = 93825195924449, rb_right = 0x0, rb_left = 0x0}, dl_size_list = {next = 0x5555613914b8, prev = 0x555561795bf8}, dl_num_elem = 43864, dl_list = {next = 0x5555613919e0, prev = 0x555561795b40}, dl_num_files = 1, dl_files_root = {
    rb_node = 0x555561391aa8}, dl_hash = "\305uq\367\312\t\322V0R\312ڧL\006\326", '\000' <repeats 15 times>}
(gdb) p *orig->b_parent
$4 = {dl_node = {__rb_parent_color = 93825133669761, rb_right = 0x55555e03b960, rb_left = 0x55555dc37300}, dl_size_list = {next = 0x55555dc37318, prev = 0x55555e03b978}, dl_num_elem = 43865, dl_list = {next = 0x55555dc37700, prev = 0x55555e03b8c0}, dl_num_files = 1,
  dl_files_root = {rb_node = 0x55555dc377c8}, dl_hash = "t\203!a(\\G\t\b\341\000\016\326^1\022", '\000' <repeats 15 times>}
(gdb)

markfasheh commented 5 years ago

Hi, can you try the fixes in the issue#222 branch of duperemove? Thanks!

KrissN commented 5 years ago

Sure, I've built duperemove from your branch and I've started it on the partition, where it failed previously.

KrissN commented 5 years ago

My directory has grown to about 900GB and I've run duperemove with your patch on it. It's been running for over 11h now on 2 threads and is consuming 5GB of RAM (and increasing slowly). Something is not right - the stable version would have been done by now.

KrissN commented 5 years ago

I have stopped duperemove after it went over 6.5GM of RAM.

markfasheh commented 5 years ago

If you're willing to try the latest master it should be much faster and better on memory as the length extent search has been put behind an option. I'd appreciate any feedback, thanks.

markfasheh commented 5 years ago

I'm going to track this all in issue #216

markfasheh / duperemove

abort in find_dupes:584 #222