search

markfasheh / duperemove

Tools for deduping file systems
GNU General Public License v2.0
816 stars 81 forks source link

ASAN: heap use after free #73

Closed matthiaskrgr closed 9 years ago

matthiaskrgr commented 9 years ago

I attached some test-files, the heap-use-after-free occurs after running (as root)

duperemove --debug -rhd <path>

on the extracted directory for the second time.

I compiled with

CFLAGS = -Wall -fsanitize=address,undefined -g3

use-after-free:

Skipping small file /home/matthias/temp/new/apps//orage_ice.xpm
Skipping small file /home/matthias/temp/new/apps//orage_round.xpm
Warning: murmur3 support is experimental!
Using 128K blocks
Using hash: murmur3
Using 4 threads for file hashing phase
csum: /home/matthias/temp/new/apps//orage.xpm   [1/2] (50.00%)
csum: /home/matthias/temp/new/apps//orage_sun.xpm   [2/2] (100.00%)
Hashing completed. Calculating duplicate extents - this may take some time.
[%                                       ]Duplicated extent of 1 blocks in files:
/home/matthias/temp/new/apps//orage.xpm     /home/matthias/temp/new/apps//orage_sun.xpm
0-1     0-1
[########################################]
Search completed with no errors.             
Simple read and compare of file data found 1 instances of extents that might benefit from deduplication.
2 extents had length 1 Blocks (174520) for a score of 174520.
Hash is: 6889a645b56db6a460d03785adcd17f8
Start       Length      Filename (2 extents)
0.0 170.4K  "/home/matthias/temp/new/apps//orage.xpm"
0.0 170.4K  "/home/matthias/temp/new/apps//orage_sun.xpm"

Removing overlapping extents

Simple read and compare of file data found 1 instances of extents that might benefit from deduplication.
2 extents had length 1 Blocks (174520) for a score of 174520.
Hash is: 6889a645b56db6a460d03785adcd17f8
Start       Length      Filename (2 extents)
0.0 170.4K  "/home/matthias/temp/new/apps//orage.xpm"
0.0 170.4K  "/home/matthias/temp/new/apps//orage_sun.xpm"
Using 4 threads for dedupe phase
(fiemap) /home/matthias/temp/new/apps//orage.xpm: start: 0, len: 174520
(fiemap) [0] fe_logical: 0, fe_length: 131072, fe_physical: 52006060032, fe_flags: 0x2008
(fiemap) [1] fe_logical: 131072, fe_length: 45056, fe_physical: 51978125312, fe_flags: 0x2001
(fiemap) /home/matthias/temp/new/apps//orage_sun.xpm: start: 0, len: 174520
(fiemap) [0] fe_logical: 0, fe_length: 131072, fe_physical: 52006060032, fe_flags: 0x2008
(fiemap) [1] fe_logical: 131072, fe_length: 40960, fe_physical: 51978125312, fe_flags: 0x2000
(fiemap) [2] fe_logical: 172032, fe_length: 4096, fe_physical: 51979546624, fe_flags: 0x2001
=================================================================
==11147==ERROR: AddressSanitizer: heap-use-after-free on address 0x60b00000afc8 at pc 0x00000041fbc7 bp 0x7f6306cfea30 sp 0x7f6306cfea20
READ of size 8 at 0x60b00000afc8 thread T1 (pool)
    #0 0x41fbc6 in clean_deduped /home/matthias/duperemove/run_dedupe.c:161
    #1 0x420203 in dedupe_extent_list /home/matthias/duperemove/run_dedupe.c:217
    #2 0x42103d in dedupe_worker /home/matthias/duperemove/run_dedupe.c:365
    #3 0x7f630b4b0a3d  (/lib64/libglib-2.0.so.0+0x71a3d)
    #4 0x7f630b4b00a4  (/lib64/libglib-2.0.so.0+0x710a4)
    #5 0x7f6309b84554 in start_thread (/lib64/libpthread.so.0+0x7554)
    #6 0x7f6309e9af3c in __clone (/lib64/libc.so.6+0x101f3c)

0x60b00000afc8 is located 56 bytes inside of 104-byte region [0x60b00000af90,0x60b00000aff8)
freed by thread T1 (pool) here:
    #0 0x7f630b81170a in __interceptor_free (/lib64/libasan.so.2+0x9870a)
    #1 0x4094bf in free_dupe_extents /home/matthias/duperemove/results-tree.c:36
    #2 0x40b40d in remove_extent /home/matthias/duperemove/results-tree.c:274
    #3 0x41fb69 in clean_deduped /home/matthias/duperemove/run_dedupe.c:175
    #4 0x420203 in dedupe_extent_list /home/matthias/duperemove/run_dedupe.c:217
    #5 0x42103d in dedupe_worker /home/matthias/duperemove/run_dedupe.c:365
    #6 0x7f630b4b0a3d  (/lib64/libglib-2.0.so.0+0x71a3d)

previously allocated by thread T0 here:
    #0 0x7f630b811ba9 in __interceptor_calloc (/lib64/libasan.so.2+0x98ba9)
    #1 0x409432 in calloc_dupe_extents /home/matthias/duperemove/results-tree.c:36
    #2 0x40a6ea in dupe_extents_new /home/matthias/duperemove/results-tree.c:183
    #3 0x40ac0d in insert_result /home/matthias/duperemove/results-tree.c:216
    #4 0x41d199 in record_match /home/matthias/duperemove/find_dupes.c:62
    #5 0x41dd41 in walk_dupe_block /home/matthias/duperemove/find_dupes.c:140
    #6 0x41dfe5 in lookup_walk_file_hash_head /home/matthias/duperemove/find_dupes.c:169
    #7 0x41e131 in find_file_dupes /home/matthias/duperemove/find_dupes.c:195
    #8 0x41e185 in compare_files /home/matthias/duperemove/find_dupes.c:202
    #9 0x41e54e in walk_dupe_hashes /home/matthias/duperemove/find_dupes.c:248
    #10 0x41ea17 in find_all_dupes /home/matthias/duperemove/find_dupes.c:328
    #11 0x405bea in main /home/matthias/duperemove/duperemove.c:430
    #12 0x7f6309db978f in __libc_start_main (/lib64/libc.so.6+0x2078f)

Thread T1 (pool) created by T0 here:
    #0 0x7f630b7af703 in pthread_create (/lib64/libasan.so.2+0x36703)
    #1 0x7f630b4cdcaf  (/lib64/libglib-2.0.so.0+0x8ecaf)

SUMMARY: AddressSanitizer: heap-use-after-free /home/matthias/duperemove/run_dedupe.c:161 clean_deduped
Shadow bytes around the buggy address:
  0x0c167fff95a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c167fff95b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c167fff95c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c167fff95d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c167fff95e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c167fff95f0: fa fa fd fd fd fd fd fd fd[fd]fd fd fd fd fd fa
  0x0c167fff9600: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c167fff9610: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c167fff9620: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c167fff9630: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c167fff9640: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
==11147==ABORTING

when skipping "-d", there are only some memleaks being reported:

Skipping small file /home/matthias/temp/new/apps//orage_ice.xpm
Skipping small file /home/matthias/temp/new/apps//orage_round.xpm
Warning: murmur3 support is experimental!
Using 128K blocks
Using hash: murmur3
Using 4 threads for file hashing phase
csum: /home/matthias/temp/new/apps//orage.xpm   [1/2] (50.00%)
csum: /home/matthias/temp/new/apps//orage_sun.xpm   [2/2] (100.00%)
Hashing completed. Calculating duplicate extents - this may take some time.
[%                                       ]Duplicated extent of 1 blocks in files:
/home/matthias/temp/new/apps//orage.xpm     /home/matthias/temp/new/apps//orage_sun.xpm
0-1     0-1
[########################################]
Search completed with no errors.             
Simple read and compare of file data found 1 instances of extents that might benefit from deduplication.
2 extents had length 1 Blocks (174520) for a score of 174520.
Hash is: 6889a645b56db6a460d03785adcd17f8
Start       Length      Filename (2 extents)
0.0 170.4K  "/home/matthias/temp/new/apps//orage.xpm"
0.0 170.4K  "/home/matthias/temp/new/apps//orage_sun.xpm"

Removing overlapping extents

Simple read and compare of file data found 1 instances of extents that might benefit from deduplication.
2 extents had length 1 Blocks (174520) for a score of 174520.
Hash is: 6889a645b56db6a460d03785adcd17f8
Start       Length      Filename (2 extents)
0.0 170.4K  "/home/matthias/temp/new/apps//orage.xpm"
0.0 170.4K  "/home/matthias/temp/new/apps//orage_sun.xpm"
Duperemove memory usage statistics:
struct file_block num: 4 sizeof: 96 total: 384
struct dupe_blocks_list num: 2 sizeof: 120 total: 240
struct dupe_extents num: 1 sizeof: 104 total: 104
struct extent num: 2 sizeof: 88 total: 176
struct filerec num: 0 sizeof: 136 total: 0
struct filerec_token num: 0 sizeof: 32 total: 0

=================================================================
==11171==ERROR: LeakSanitizer: detected memory leaks

Indirect leak of 104 byte(s) in 1 object(s) allocated from:
    #0 0x7fa90ef4dba9 in __interceptor_calloc (/lib64/libasan.so.2+0x98ba9)
    #1 0x409432 in calloc_dupe_extents /home/matthias/duperemove/results-tree.c:36
    #2 0x40a6ea in dupe_extents_new /home/matthias/duperemove/results-tree.c:183
    #3 0x40ac0d in insert_result /home/matthias/duperemove/results-tree.c:216
    #4 0x41d199 in record_match /home/matthias/duperemove/find_dupes.c:62
    #5 0x41dd41 in walk_dupe_block /home/matthias/duperemove/find_dupes.c:140
    #6 0x41dfe5 in lookup_walk_file_hash_head /home/matthias/duperemove/find_dupes.c:169
    #7 0x41e131 in find_file_dupes /home/matthias/duperemove/find_dupes.c:195
    #8 0x41e185 in compare_files /home/matthias/duperemove/find_dupes.c:202
    #9 0x41e54e in walk_dupe_hashes /home/matthias/duperemove/find_dupes.c:248
    #10 0x41ea17 in find_all_dupes /home/matthias/duperemove/find_dupes.c:328
    #11 0x405bea in main /home/matthias/duperemove/duperemove.c:430
    #12 0x7fa90d4f578f in __libc_start_main (/lib64/libc.so.6+0x2078f)

Indirect leak of 88 byte(s) in 1 object(s) allocated from:
    #0 0x7fa90ef4dba9 in __interceptor_calloc (/lib64/libasan.so.2+0x98ba9)
    #1 0x40954e in calloc_extent /home/matthias/duperemove/results-tree.c:37
    #2 0x409666 in alloc_extent /home/matthias/duperemove/results-tree.c:41
    #3 0x40aa28 in insert_result /home/matthias/duperemove/results-tree.c:205
    #4 0x41d199 in record_match /home/matthias/duperemove/find_dupes.c:62
    #5 0x41dd41 in walk_dupe_block /home/matthias/duperemove/find_dupes.c:140
    #6 0x41dfe5 in lookup_walk_file_hash_head /home/matthias/duperemove/find_dupes.c:169
    #7 0x41e131 in find_file_dupes /home/matthias/duperemove/find_dupes.c:195
    #8 0x41e185 in compare_files /home/matthias/duperemove/find_dupes.c:202
    #9 0x41e54e in walk_dupe_hashes /home/matthias/duperemove/find_dupes.c:248
    #10 0x41ea17 in find_all_dupes /home/matthias/duperemove/find_dupes.c:328
    #11 0x405bea in main /home/matthias/duperemove/duperemove.c:430
    #12 0x7fa90d4f578f in __libc_start_main (/lib64/libc.so.6+0x2078f)

Indirect leak of 88 byte(s) in 1 object(s) allocated from:
    #0 0x7fa90ef4dba9 in __interceptor_calloc (/lib64/libasan.so.2+0x98ba9)
    #1 0x40954e in calloc_extent /home/matthias/duperemove/results-tree.c:37
    #2 0x409666 in alloc_extent /home/matthias/duperemove/results-tree.c:41
    #3 0x40aad4 in insert_result /home/matthias/duperemove/results-tree.c:206
    #4 0x41d199 in record_match /home/matthias/duperemove/find_dupes.c:62
    #5 0x41dd41 in walk_dupe_block /home/matthias/duperemove/find_dupes.c:140
    #6 0x41dfe5 in lookup_walk_file_hash_head /home/matthias/duperemove/find_dupes.c:169
    #7 0x41e131 in find_file_dupes /home/matthias/duperemove/find_dupes.c:195
    #8 0x41e185 in compare_files /home/matthias/duperemove/find_dupes.c:202
    #9 0x41e54e in walk_dupe_hashes /home/matthias/duperemove/find_dupes.c:248
    #10 0x41ea17 in find_all_dupes /home/matthias/duperemove/find_dupes.c:328
    #11 0x405bea in main /home/matthias/duperemove/duperemove.c:430
    #12 0x7fa90d4f578f in __libc_start_main (/lib64/libc.so.6+0x2078f)

SUMMARY: AddressSanitizer: 280 byte(s) leaked in 3 allocation(s).
matthiaskrgr commented 9 years ago

Looks like I can't upload zips to github; files found here: http://s000.tinyupload.com/index.php?file_id=95296634016149252021

markfasheh commented 9 years ago

Thanks for the report, I actually saw this yesterday too doing my large run. My guess is that it's somewhere in dedupe_extent_list() on down - I saw it right after the dedupe stage started. We had some changes in that area recently too.

markfasheh commented 9 years ago

If you have time to check out the 'dedupe_fixes' branch and report back whether that fixes these issues for you that'd be great. I'm seeing a much more stable dedupe stage now as a result of those.

matthiaskrgr commented 9 years ago

I'm still having issues with the given test files on 3f2184cd76bca7c3462ac318de2c143ef840486c

 [root@t420 duperemove]# ./duperemove -rdh /home/matthias/temp/ --debug 
Skipping small file /home/matthias/temp//hicolor/.git/refs/heads/master
Skipping small file /home/matthias/temp//hicolor/.git/hooks/applypatch-msg.sample
Skipping small file /home/matthias/temp//hicolor/.git/hooks/commit-msg.sample
Skipping small file /home/matthias/temp//hicolor/.git/hooks/post-update.sample
Skipping small file /home/matthias/temp//hicolor/.git/hooks/pre-applypatch.sample
Skipping small file /home/matthias/temp//hicolor/.git/hooks/pre-commit.sample
Skipping small file /home/matthias/temp//hicolor/.git/hooks/pre-push.sample
Skipping small file /home/matthias/temp//hicolor/.git/hooks/pre-rebase.sample
Skipping small file /home/matthias/temp//hicolor/.git/hooks/prepare-commit-msg.sample
Skipping small file /home/matthias/temp//hicolor/.git/hooks/update.sample
Skipping small file /home/matthias/temp//hicolor/.git/info/exclude
Skipping small file /home/matthias/temp//hicolor/.git/info/refs
Skipping small file /home/matthias/temp//hicolor/.git/description
Skipping small file /home/matthias/temp//hicolor/.git/HEAD
Skipping small file /home/matthias/temp//hicolor/.git/config
Skipping small file /home/matthias/temp//hicolor/.git/objects/pack/pack-f5c489d95b7655420175486f2245eb5dfd1dade6.idx
Skipping small file /home/matthias/temp//hicolor/.git/objects/info/packs
Skipping small file /home/matthias/temp//hicolor/.git/objects/b9/17efe920406d3293f99d4d7ea30e30c525f9e7
Skipping small file /home/matthias/temp//hicolor/.git/objects/83/9214ffaf7a71cc4353d5754e629d2d39788d94
Skipping small file /home/matthias/temp//hicolor/.git/COMMIT_EDITMSG
Skipping small file /home/matthias/temp//hicolor/.git/logs/refs/heads/master
Skipping small file /home/matthias/temp//hicolor/.git/logs/HEAD
Skipping small file /home/matthias/temp//hicolor/.git/ORIG_HEAD
Skipping small file /home/matthias/temp//hicolor/.git/index
Skipping small file /home/matthias/temp//hicolor/160x160/apps/orage_ice.xpm
Skipping small file /home/matthias/temp//hicolor/160x160/apps/orage_round.xpm
Skipping small file /home/matthias/temp//new/apps/orage_ice.xpm
Skipping small file /home/matthias/temp//new/apps/orage_round.xpm
Warning: murmur3 support is experimental!
Using 128K blocks
Using hash: murmur3
Using 4 threads for file hashing phase
csum: /home/matthias/temp//hicolor/.git/objects/pack/pack-f5c489d95b7655420175486f2245eb5dfd1dade6.pack     [1/7] (14.29%)
csum: /home/matthias/temp//hicolor/160x160/apps/orage.xpm   [2/7] (28.57%)
csum: /home/matthias/temp//hicolor/160x160/apps/orage_sun.xpm   [3/7] (42.86%)
csum: /home/matthias/temp//bla.gz   [4/7] (57.14%)
csum: /home/matthias/temp//new/files.zip    [5/7] (71.43%)
csum: /home/matthias/temp//new/apps/orage.xpm   [6/7] (85.71%)
csum: /home/matthias/temp//new/apps/orage_sun.xpm   [7/7] (100.00%)
Hashing completed. Calculating duplicate extents - this may take some time.
[#################%                      ]Duplicated extent of 1 blocks in files:
/home/matthias/temp//hicolor/160x160/apps/orage.xpm     /home/matthias/temp//hicolor/160x160/apps/orage_sun.xpm
0-1     0-1
Duplicated extent of 1 blocks in files:
/home/matthias/temp//hicolor/160x160/apps/orage_sun.xpm     /home/matthias/temp//new/apps/orage_sun.xpm
0-1     0-1
Duplicated extent of 1 blocks in files:
/home/matthias/temp//new/apps/orage_sun.xpm     /home/matthias/temp//new/apps/orage.xpm
0-1     0-1
[########################################]
Search completed with no errors.             
Simple read and compare of file data found 1 instances of extents that might benefit from deduplication.
4 extents had length 1 Blocks (174520) for a score of 523560.
Hash is: 6889a645b56db6a460d03785adcd17f8
Start       Length      Filename (4 extents)
0.0 170.4K  "/home/matthias/temp//hicolor/160x160/apps/orage.xpm"
0.0 170.4K  "/home/matthias/temp//hicolor/160x160/apps/orage_sun.xpm"
0.0 170.4K  "/home/matthias/temp//new/apps/orage_sun.xpm"
0.0 170.4K  "/home/matthias/temp//new/apps/orage.xpm"

Removing overlapping extents

Simple read and compare of file data found 1 instances of extents that might benefit from deduplication.
4 extents had length 1 Blocks (174520) for a score of 523560.
Hash is: 6889a645b56db6a460d03785adcd17f8
Start       Length      Filename (4 extents)
0.0 170.4K  "/home/matthias/temp//hicolor/160x160/apps/orage.xpm"
0.0 170.4K  "/home/matthias/temp//hicolor/160x160/apps/orage_sun.xpm"
0.0 170.4K  "/home/matthias/temp//new/apps/orage_sun.xpm"
0.0 170.4K  "/home/matthias/temp//new/apps/orage.xpm"
Using 4 threads for dedupe phase
(fiemap) /home/matthias/temp//hicolor/160x160/apps/orage.xpm: start: 0, len: 174520
(fiemap) [0] fe_logical: 0, fe_length: 131072, fe_physical: 50084708352, fe_flags: 0x2008
(fiemap) [1] fe_logical: 131072, fe_length: 45056, fe_physical: 50121392128, fe_flags: 0x2001
(fiemap) /home/matthias/temp//hicolor/160x160/apps/orage_sun.xpm: start: 0, len: 174520
(fiemap) [0] fe_logical: 0, fe_length: 131072, fe_physical: 51890085888, fe_flags: 0x2008
(fiemap) [1] fe_logical: 131072, fe_length: 45056, fe_physical: 51866259456, fe_flags: 0x2001
(fiemap) /home/matthias/temp//new/apps/orage_sun.xpm: start: 0, len: 174520
(fiemap) [0] fe_logical: 0, fe_length: 131072, fe_physical: 52006060032, fe_flags: 0x2008
(fiemap) [1] fe_logical: 131072, fe_length: 40960, fe_physical: 51978125312, fe_flags: 0x2000
(fiemap) [2] fe_logical: 172032, fe_length: 4096, fe_physical: 51979546624, fe_flags: 0x2001
(fiemap) /home/matthias/temp//new/apps/orage.xpm: start: 0, len: 174520
(fiemap) [0] fe_logical: 0, fe_length: 131072, fe_physical: 52006060032, fe_flags: 0x2008
(fiemap) [1] fe_logical: 131072, fe_length: 45056, fe_physical: 51978125312, fe_flags: 0x2001
=================================================================
==11868==ERROR: AddressSanitizer: heap-use-after-free on address 0x60800000bcb8 at pc 0x000000420780 bp 0x7f7d0ecfeb30 sp 0x7f7d0ecfeb20
READ of size 8 at 0x60800000bcb8 thread T1 (pool)
    #0 0x42077f in clean_deduped /home/matthias/duperemove/run_dedupe.c:157
    #1 0x42077f in dedupe_extent_list /home/matthias/duperemove/run_dedupe.c:210
    #2 0x42077f in dedupe_worker /home/matthias/duperemove/run_dedupe.c:373
    #3 0x7f7d134e4a3d  (/lib64/libglib-2.0.so.0+0x71a3d)
    #4 0x7f7d134e40a4  (/lib64/libglib-2.0.so.0+0x710a4)
    #5 0x7f7d11bb8554 in start_thread (/lib64/libpthread.so.0+0x7554)
    #6 0x7f7d11ecef3c in __clone (/lib64/libc.so.6+0x101f3c)

0x60800000bcb8 is located 24 bytes inside of 88-byte region [0x60800000bca0,0x60800000bcf8)
freed by thread T1 (pool) here:
    #0 0x7f7d1384570a in __interceptor_free (/lib64/libasan.so.2+0x9870a)
    #1 0x40bd78 in free_extent /home/matthias/duperemove/results-tree.c:37
    #2 0x40bd78 in remove_extent /home/matthias/duperemove/results-tree.c:260
    #3 0x41fb2a in clean_deduped /home/matthias/duperemove/run_dedupe.c:177
    #4 0x41fb2a in dedupe_extent_list /home/matthias/duperemove/run_dedupe.c:210
    #5 0x41fb2a in dedupe_worker /home/matthias/duperemove/run_dedupe.c:373
    #6 0x7f7d134e4a3d  (/lib64/libglib-2.0.so.0+0x71a3d)

previously allocated by thread T0 here:
    #0 0x7f7d13845ba9 in __interceptor_calloc (/lib64/libasan.so.2+0x98ba9)
    #1 0x4095d1 in calloc_extent /home/matthias/duperemove/results-tree.c:37
    #2 0x4095d1 in alloc_extent /home/matthias/duperemove/results-tree.c:41
    #3 0x40a545 in insert_result /home/matthias/duperemove/results-tree.c:206
    #4 0x41e301 in record_match /home/matthias/duperemove/find_dupes.c:62
    #5 0x41e301 in walk_dupe_block /home/matthias/duperemove/find_dupes.c:140
    #6 0x41e301 in lookup_walk_file_hash_head /home/matthias/duperemove/find_dupes.c:169
    #7 0x41e301 in find_file_dupes /home/matthias/duperemove/find_dupes.c:195
    #8 0x41e301 in compare_files /home/matthias/duperemove/find_dupes.c:202
    #9 0x41e301 in walk_dupe_hashes /home/matthias/duperemove/find_dupes.c:248
    #10 0x41e301 in find_all_dupes /home/matthias/duperemove/find_dupes.c:328
    #11 0x405472 in main /home/matthias/duperemove/duperemove.c:430
    #12 0x7f7d11ded78f in __libc_start_main (/lib64/libc.so.6+0x2078f)

Thread T1 (pool) created by T0 here:
    #0 0x7f7d137e3703 in pthread_create (/lib64/libasan.so.2+0x36703)
    #1 0x7f7d13501caf  (/lib64/libglib-2.0.so.0+0x8ecaf)

SUMMARY: AddressSanitizer: heap-use-after-free /home/matthias/duperemove/run_dedupe.c:157 clean_deduped
Shadow bytes around the buggy address:
  0x0c107fff9740: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c107fff9750: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c107fff9760: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c107fff9770: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c107fff9780: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
=>0x0c107fff9790: fa fa fa fa fd fd fd[fd]fd fd fd fd fd fd fd fa
  0x0c107fff97a0: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fa
  0x0c107fff97b0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
  0x0c107fff97c0: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fa
  0x0c107fff97d0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
  0x0c107fff97e0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
==11868==ABORTING
markfasheh commented 9 years ago

Closing - we figured this out in irc, resulting in one more patch fix.

petechap commented 9 years ago

Fixed for me too, thanks.