Closed matthiaskrgr closed 9 years ago
Looks like I can't upload zips to github; files found here: http://s000.tinyupload.com/index.php?file_id=95296634016149252021
Thanks for the report, I actually saw this yesterday too doing my large run. My guess is that it's somewhere in dedupe_extent_list() on down - I saw it right after the dedupe stage started. We had some changes in that area recently too.
If you have time to check out the 'dedupe_fixes' branch and report back whether that fixes these issues for you that'd be great. I'm seeing a much more stable dedupe stage now as a result of those.
I'm still having issues with the given test files on 3f2184cd76bca7c3462ac318de2c143ef840486c
[root@t420 duperemove]# ./duperemove -rdh /home/matthias/temp/ --debug
Skipping small file /home/matthias/temp//hicolor/.git/refs/heads/master
Skipping small file /home/matthias/temp//hicolor/.git/hooks/applypatch-msg.sample
Skipping small file /home/matthias/temp//hicolor/.git/hooks/commit-msg.sample
Skipping small file /home/matthias/temp//hicolor/.git/hooks/post-update.sample
Skipping small file /home/matthias/temp//hicolor/.git/hooks/pre-applypatch.sample
Skipping small file /home/matthias/temp//hicolor/.git/hooks/pre-commit.sample
Skipping small file /home/matthias/temp//hicolor/.git/hooks/pre-push.sample
Skipping small file /home/matthias/temp//hicolor/.git/hooks/pre-rebase.sample
Skipping small file /home/matthias/temp//hicolor/.git/hooks/prepare-commit-msg.sample
Skipping small file /home/matthias/temp//hicolor/.git/hooks/update.sample
Skipping small file /home/matthias/temp//hicolor/.git/info/exclude
Skipping small file /home/matthias/temp//hicolor/.git/info/refs
Skipping small file /home/matthias/temp//hicolor/.git/description
Skipping small file /home/matthias/temp//hicolor/.git/HEAD
Skipping small file /home/matthias/temp//hicolor/.git/config
Skipping small file /home/matthias/temp//hicolor/.git/objects/pack/pack-f5c489d95b7655420175486f2245eb5dfd1dade6.idx
Skipping small file /home/matthias/temp//hicolor/.git/objects/info/packs
Skipping small file /home/matthias/temp//hicolor/.git/objects/b9/17efe920406d3293f99d4d7ea30e30c525f9e7
Skipping small file /home/matthias/temp//hicolor/.git/objects/83/9214ffaf7a71cc4353d5754e629d2d39788d94
Skipping small file /home/matthias/temp//hicolor/.git/COMMIT_EDITMSG
Skipping small file /home/matthias/temp//hicolor/.git/logs/refs/heads/master
Skipping small file /home/matthias/temp//hicolor/.git/logs/HEAD
Skipping small file /home/matthias/temp//hicolor/.git/ORIG_HEAD
Skipping small file /home/matthias/temp//hicolor/.git/index
Skipping small file /home/matthias/temp//hicolor/160x160/apps/orage_ice.xpm
Skipping small file /home/matthias/temp//hicolor/160x160/apps/orage_round.xpm
Skipping small file /home/matthias/temp//new/apps/orage_ice.xpm
Skipping small file /home/matthias/temp//new/apps/orage_round.xpm
Warning: murmur3 support is experimental!
Using 128K blocks
Using hash: murmur3
Using 4 threads for file hashing phase
csum: /home/matthias/temp//hicolor/.git/objects/pack/pack-f5c489d95b7655420175486f2245eb5dfd1dade6.pack [1/7] (14.29%)
csum: /home/matthias/temp//hicolor/160x160/apps/orage.xpm [2/7] (28.57%)
csum: /home/matthias/temp//hicolor/160x160/apps/orage_sun.xpm [3/7] (42.86%)
csum: /home/matthias/temp//bla.gz [4/7] (57.14%)
csum: /home/matthias/temp//new/files.zip [5/7] (71.43%)
csum: /home/matthias/temp//new/apps/orage.xpm [6/7] (85.71%)
csum: /home/matthias/temp//new/apps/orage_sun.xpm [7/7] (100.00%)
Hashing completed. Calculating duplicate extents - this may take some time.
[#################% ]Duplicated extent of 1 blocks in files:
/home/matthias/temp//hicolor/160x160/apps/orage.xpm /home/matthias/temp//hicolor/160x160/apps/orage_sun.xpm
0-1 0-1
Duplicated extent of 1 blocks in files:
/home/matthias/temp//hicolor/160x160/apps/orage_sun.xpm /home/matthias/temp//new/apps/orage_sun.xpm
0-1 0-1
Duplicated extent of 1 blocks in files:
/home/matthias/temp//new/apps/orage_sun.xpm /home/matthias/temp//new/apps/orage.xpm
0-1 0-1
[########################################]
Search completed with no errors.
Simple read and compare of file data found 1 instances of extents that might benefit from deduplication.
4 extents had length 1 Blocks (174520) for a score of 523560.
Hash is: 6889a645b56db6a460d03785adcd17f8
Start Length Filename (4 extents)
0.0 170.4K "/home/matthias/temp//hicolor/160x160/apps/orage.xpm"
0.0 170.4K "/home/matthias/temp//hicolor/160x160/apps/orage_sun.xpm"
0.0 170.4K "/home/matthias/temp//new/apps/orage_sun.xpm"
0.0 170.4K "/home/matthias/temp//new/apps/orage.xpm"
Removing overlapping extents
Simple read and compare of file data found 1 instances of extents that might benefit from deduplication.
4 extents had length 1 Blocks (174520) for a score of 523560.
Hash is: 6889a645b56db6a460d03785adcd17f8
Start Length Filename (4 extents)
0.0 170.4K "/home/matthias/temp//hicolor/160x160/apps/orage.xpm"
0.0 170.4K "/home/matthias/temp//hicolor/160x160/apps/orage_sun.xpm"
0.0 170.4K "/home/matthias/temp//new/apps/orage_sun.xpm"
0.0 170.4K "/home/matthias/temp//new/apps/orage.xpm"
Using 4 threads for dedupe phase
(fiemap) /home/matthias/temp//hicolor/160x160/apps/orage.xpm: start: 0, len: 174520
(fiemap) [0] fe_logical: 0, fe_length: 131072, fe_physical: 50084708352, fe_flags: 0x2008
(fiemap) [1] fe_logical: 131072, fe_length: 45056, fe_physical: 50121392128, fe_flags: 0x2001
(fiemap) /home/matthias/temp//hicolor/160x160/apps/orage_sun.xpm: start: 0, len: 174520
(fiemap) [0] fe_logical: 0, fe_length: 131072, fe_physical: 51890085888, fe_flags: 0x2008
(fiemap) [1] fe_logical: 131072, fe_length: 45056, fe_physical: 51866259456, fe_flags: 0x2001
(fiemap) /home/matthias/temp//new/apps/orage_sun.xpm: start: 0, len: 174520
(fiemap) [0] fe_logical: 0, fe_length: 131072, fe_physical: 52006060032, fe_flags: 0x2008
(fiemap) [1] fe_logical: 131072, fe_length: 40960, fe_physical: 51978125312, fe_flags: 0x2000
(fiemap) [2] fe_logical: 172032, fe_length: 4096, fe_physical: 51979546624, fe_flags: 0x2001
(fiemap) /home/matthias/temp//new/apps/orage.xpm: start: 0, len: 174520
(fiemap) [0] fe_logical: 0, fe_length: 131072, fe_physical: 52006060032, fe_flags: 0x2008
(fiemap) [1] fe_logical: 131072, fe_length: 45056, fe_physical: 51978125312, fe_flags: 0x2001
=================================================================
==11868==ERROR: AddressSanitizer: heap-use-after-free on address 0x60800000bcb8 at pc 0x000000420780 bp 0x7f7d0ecfeb30 sp 0x7f7d0ecfeb20
READ of size 8 at 0x60800000bcb8 thread T1 (pool)
#0 0x42077f in clean_deduped /home/matthias/duperemove/run_dedupe.c:157
#1 0x42077f in dedupe_extent_list /home/matthias/duperemove/run_dedupe.c:210
#2 0x42077f in dedupe_worker /home/matthias/duperemove/run_dedupe.c:373
#3 0x7f7d134e4a3d (/lib64/libglib-2.0.so.0+0x71a3d)
#4 0x7f7d134e40a4 (/lib64/libglib-2.0.so.0+0x710a4)
#5 0x7f7d11bb8554 in start_thread (/lib64/libpthread.so.0+0x7554)
#6 0x7f7d11ecef3c in __clone (/lib64/libc.so.6+0x101f3c)
0x60800000bcb8 is located 24 bytes inside of 88-byte region [0x60800000bca0,0x60800000bcf8)
freed by thread T1 (pool) here:
#0 0x7f7d1384570a in __interceptor_free (/lib64/libasan.so.2+0x9870a)
#1 0x40bd78 in free_extent /home/matthias/duperemove/results-tree.c:37
#2 0x40bd78 in remove_extent /home/matthias/duperemove/results-tree.c:260
#3 0x41fb2a in clean_deduped /home/matthias/duperemove/run_dedupe.c:177
#4 0x41fb2a in dedupe_extent_list /home/matthias/duperemove/run_dedupe.c:210
#5 0x41fb2a in dedupe_worker /home/matthias/duperemove/run_dedupe.c:373
#6 0x7f7d134e4a3d (/lib64/libglib-2.0.so.0+0x71a3d)
previously allocated by thread T0 here:
#0 0x7f7d13845ba9 in __interceptor_calloc (/lib64/libasan.so.2+0x98ba9)
#1 0x4095d1 in calloc_extent /home/matthias/duperemove/results-tree.c:37
#2 0x4095d1 in alloc_extent /home/matthias/duperemove/results-tree.c:41
#3 0x40a545 in insert_result /home/matthias/duperemove/results-tree.c:206
#4 0x41e301 in record_match /home/matthias/duperemove/find_dupes.c:62
#5 0x41e301 in walk_dupe_block /home/matthias/duperemove/find_dupes.c:140
#6 0x41e301 in lookup_walk_file_hash_head /home/matthias/duperemove/find_dupes.c:169
#7 0x41e301 in find_file_dupes /home/matthias/duperemove/find_dupes.c:195
#8 0x41e301 in compare_files /home/matthias/duperemove/find_dupes.c:202
#9 0x41e301 in walk_dupe_hashes /home/matthias/duperemove/find_dupes.c:248
#10 0x41e301 in find_all_dupes /home/matthias/duperemove/find_dupes.c:328
#11 0x405472 in main /home/matthias/duperemove/duperemove.c:430
#12 0x7f7d11ded78f in __libc_start_main (/lib64/libc.so.6+0x2078f)
Thread T1 (pool) created by T0 here:
#0 0x7f7d137e3703 in pthread_create (/lib64/libasan.so.2+0x36703)
#1 0x7f7d13501caf (/lib64/libglib-2.0.so.0+0x8ecaf)
SUMMARY: AddressSanitizer: heap-use-after-free /home/matthias/duperemove/run_dedupe.c:157 clean_deduped
Shadow bytes around the buggy address:
0x0c107fff9740: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c107fff9750: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c107fff9760: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c107fff9770: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c107fff9780: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
=>0x0c107fff9790: fa fa fa fa fd fd fd[fd]fd fd fd fd fd fd fd fa
0x0c107fff97a0: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fa
0x0c107fff97b0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
0x0c107fff97c0: fa fa fa fa fd fd fd fd fd fd fd fd fd fd fd fa
0x0c107fff97d0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
0x0c107fff97e0: fa fa fa fa 00 00 00 00 00 00 00 00 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Heap right redzone: fb
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack partial redzone: f4
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
==11868==ABORTING
Closing - we figured this out in irc, resulting in one more patch fix.
Fixed for me too, thanks.
I attached some test-files, the heap-use-after-free occurs after running (as root)
on the extracted directory for the second time.
I compiled with
use-after-free:
when skipping "-d", there are only some memleaks being reported: