search

sahib / rmlint

Extremely fast tool to remove duplicates and other lint from your filesystem
http://rmlint.rtfd.org
GNU General Public License v3.0
1.86k stars 128 forks source link

SIGABRT: double free or corruption (out) #562

Closed Suika closed 1 year ago

Suika commented 2 years ago

Long story short, I was deduping two locations with rmlint DUPES // ORIGINAL --keep-all-tagged --must-match-tagged -g -T 'all,-badids' --xattr-read --xattr-write --write-unfinished These locations are MergerFS FUSE mounts, but everything worked fine. MergerFS config is as follows ergerfs -o defaults,allow_other,category.create=epmfs,minfreespace=50G,dropcacheonclose=true,moveonenospc=true The FS on disk to which MergerFS is pointing to, is BTRFS.

After I was done deduping the said locations, I wanted to dedupe the "DUPES" location, where all the duplicates were, so I run rmlint DUPES -g -T all,-badids --xattr-read --xattr-write --write-unfinished on it and the application stopped working with a "double free or corruption (out)" message, the json file is cutoff in the middle.

To make sure that it is not MergerFS/FUSE fuckery, I run rmlint against the disk mount directly and got the same "double free or corruption (out)". Seems to be the same thing as in #447.

rmlint built with scons DEBUG=1 VERBOSE=0 GDB=1

# free -l
              total        used        free      shared  buff/cache   available
Mem:      197868360    59962692     3884100     1849184   134021568   134479704
Low:      197868360   193984260     3884100
High:             0           0           0
Swap:             0           0           0

# rpm -qa | grep btrfs
libbtrfs0-4.19.1-16.1.x86_64
btrfsprogs-4.19.1-16.1.x86_64
btrfsmaintenance-0.4.2-3.3.1.noarch
btrfsprogs-udev-rules-4.19.1-16.1.noarch

# uname -a
Linux 5.3.18-59.24-default #1 SMP Mon Sep 13 15:06:42 UTC 2021 (2f872ea) x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/SUSE-brand
openSUSE
VERSION = 15.3

# rmlint --version
version 2.10.1 compiled: Feb 19 2022 at [22:16:06] "Ludicrous Lemur" (rev 854af40f)
compiled with: +mounts +nonstripped +fiemap +sha512 +bigfiles +intl +replay +xattr +btrfs-support

rmlint was written by Christopher <sahib> Pahl and Daniel <SeeSpotRun> Thomas.
The code at https://github.com/sahib/rmlint is licensed under the terms of the GPLv3.

Point to MergerFS:

GNU gdb (GDB; SUSE Linux Enterprise 15) 11.1
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://bugs.opensuse.org/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from rmlint...
Starting program: /usr/local/bin/rmlint -vvv /mnt/temp/jd2/download -g -T all,-badids --xattr-read --xattr-write --write-unfinished
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff5161700 (LWP 10663)]
[New Thread 0x7ffff4960700 (LWP 10664)]
▕░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░[Thread 0x7ffff4960700 (LWP 10664) exited]                                                                                                      Traversing (13310 usable files / 0 + 0 ignored files / folders)
▕░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▏                                                                          Traversing (13513 usable files / 0 + 0 ignored files / folders)
▕▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒New Thread 0x7ffff4960700 (LWP 10867)]
▕░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▏                                                                                                          Preprocessing (reduces files to 10706 / found 0 other lint)
▕▒░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░New Thread 0x7fffeffff700 (LWP 10870)]
[New Thread 0x7fffef7fe700 (LWP 10871)]
▕░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▏                                                                                            Matching (3318 dupes of 2654 originals; 0 B to scan in 0 files, ETA: ...)
▕▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓Thread 0x7fffef7fe700 (LWP 10871) exited]
[Thread 0x7ffff4960700 (LWP 10867) exited]
[Thread 0x7ffff5161700 (LWP 10663) exited]
▕░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▏                                                         Merging files into directories (stand by...)

==> In total 13513 files, whereof 3318 are duplicates in 2654 groups.
==> This equals 893.56 GB of duplicates which could be removed.
==> Scanning took in total 4.827s.

Wrote a json file to: /usr/local/src/rmlint/rmlint.json
Wrote a sh file to: /usr/local/src/rmlint/rmlint.sh
double free or corruption (out)

Thread 1 "rmlint" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51      }
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffff64f7585 in __GI_abort () at abort.c:79
#2  0x00007ffff653a2f7 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff6659cd8 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#3  0x00007ffff65417ea in malloc_printerr (str=str@entry=0x7ffff665be90 "double free or corruption (out)") at malloc.c:5347
#4  0x00007ffff6543580 in _int_free (av=0x7ffff688a9e0 <main_arena>, p=0x7fffe8132630, have_lock=<optimized out>) at malloc.c:4314
#5  0x00007ffff70623b9 in g_free (mem=0x7fffe8132640) at ../glib/gmem.c:192
#6  0x000000000042d462 in rm_file_destroy (file=0x7fffe8131c50) at lib/file.c:141
#7  0x000000000040f100 in rm_fmt_group_destroy (self=self@entry=0x661080, group=0xa66aa0) at lib/formats.c:52
#8  0x000000000040fc9e in rm_fmt_close (self=0x661080) at lib/formats.c:438
#9  0x00000000004167cd in rm_session_clear (session=session@entry=0x7fffffffde10) at lib/session.c:131
#10 0x0000000000407db1 in main (argc=9, argv=0x7fffffffe218) at src/rmlint.c:146
(gdb)

Pointing to Mount directly:

GNU gdb (GDB; SUSE Linux Enterprise 15) 11.1
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://bugs.opensuse.org/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from rmlint...
Starting program: /usr/local/bin/rmlint -vvv /mnt/otherserrver/D1/jd2/download -g -T all,-badids --xattr-read --xattr-write --write-unfinished
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff5161700 (LWP 12046)]
[New Thread 0x7ffff4960700 (LWP 12047)]
▕░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░Thread 0x7ffff5161700 (LWP 12046) exited]░░▏                                                                                                                                                               Traversing (4864 usable files / 0 + 0 ignored files / folders)
▕░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▏                                                                                                                   Traversing (5267 usable files / 0 + 0 ignored files / folders)
▕░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░New Thread 0x7ffff5161700 (LWP 12055)]
▕░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▏                                                                                                      Preprocessing (reduces files to 0 / found 0 other lint)
▕▒░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░New Thread 0x7fffeffff700 (LWP 12056)]
[New Thread 0x7fffef7fe700 (LWP 12057)]
▕░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▏                                                                                                                                                     Matching (1053 dupes of 979 originals; 0 B to scan in 0 files, ETA: ...)
▕▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓Thread 0x7fffef7fe700 (LWP 12057) exited]
[Thread 0x7ffff5161700 (LWP 12055) exited]
[Thread 0x7ffff4960700 (LWP 12047) exited]
▕░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▏                                                                                                                 Merging files into directories (stand by...)

==> In total 5267 files, whereof 1053 are duplicates in 979 groups.
==> This equals 159.17 GB of duplicates which could be removed.
==> Scanning took in total 0.343s.

Wrote a json file to: /usr/local/src/rmlint/rmlint.json
Wrote a sh file to: /usr/local/src/rmlint/rmlint.sh
double free or corruption (out)

Thread 1 "rmlint" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51      }
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffff64f7585 in __GI_abort () at abort.c:79
#2  0x00007ffff653a2f7 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff6659cd8 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#3  0x00007ffff65417ea in malloc_printerr (str=str@entry=0x7ffff665be90 "double free or corruption (out)") at malloc.c:5347
#4  0x00007ffff6543580 in _int_free (av=0x7ffff688a9e0 <main_arena>, p=0x7ffff00ddde0, have_lock=<optimized out>) at malloc.c:4314
#5  0x00007ffff70623b9 in g_free (mem=0x7ffff00dddf0) at ../glib/gmem.c:192
#6  0x000000000042d462 in rm_file_destroy (file=0x7ffff00e1c50) at lib/file.c:141
#7  0x000000000040f100 in rm_fmt_group_destroy (self=self@entry=0x661080, group=0x8b7860) at lib/formats.c:52
#8  0x000000000040fc9e in rm_fmt_close (self=0x661080) at lib/formats.c:438
#9  0x00000000004167cd in rm_session_clear (session=session@entry=0x7fffffffde00) at lib/session.c:131
#10 0x0000000000407db1 in main (argc=9, argv=0x7fffffffe208) at src/rmlint.c:146
(gdb)
Suika commented 2 years ago

I tested every single directory in jd2/download with rmlint and it seems to be fine, but not when running rmlint against jd2/download.

cebtenzzre commented 2 years ago

Build with CFLAGS='-fsanitize=address' LDFLAGS='-fsanitize=address' scons DEBUG=1 VERBOSE=0 GDB=1 and paste the full ASAN report.

Suika commented 2 years ago 
GNU gdb (GDB; SUSE Linux Enterprise 15) 11.1
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://bugs.opensuse.org/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from rmlint...
Starting program: /usr/local/src/rmlint/rmlint -vvv /mnt/temp/jd2/download -g -T all,-badids --xattr-read --xattr-write --write-unfinished
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff10f6700 (LWP 23650)]
[New Thread 0x7ffff08f5700 (LWP 23651)]
▕░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░Thread 0x7ffff08f5700 (LWP 23651) exited]                                                                                                                       Traversing (13000 usable files / 0 + 0 ignored files / folders)
▕░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▏                                                                      Traversing (13030 usable files / 0 + 0 ignored files / folders)
▕▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒New Thread 0x7ffff08f5700 (LWP 24041)]
▕░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▏                                                                                                                                                                  Preprocessing (reduces files to 11145 / found 0 other lint)
▕▒░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░New Thread 0x7ffff00f4700 (LWP 24046)]
[New Thread 0x7fffef8f3700 (LWP 24051)]
▕░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▏                                                                                                                                                    Matching (3076 dupes of 2515 originals; 0 B to scan in 0 files, ETA: ...)
▕▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓Thread 0x7ffff00f4700 (LWP 24046) exited]
[Thread 0x7ffff08f5700 (LWP 24041) exited]
[Thread 0x7ffff10f6700 (LWP 23650) exited]
▕░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▏                                                                                                                 Merging files into directories (stand by...)

==> In total 13030 files, whereof 3076 are duplicates in 2515 groups.
==> This equals 853.19 GB of duplicates which could be removed.
==> Scanning took in total 8.099s.

Wrote a json file to: /usr/local/src/rmlint/rmlint.json
Wrote a sh file to: /usr/local/src/rmlint/rmlint.sh
=================================================================
==23589==ERROR: AddressSanitizer: attempting double-free on 0x60d00007c2d0 in thread T0:
    #0 0x7ffff6efd1a8 in __interceptor_free (/usr/lib64/libasan.so.4+0xdc1a8)
    #1 0x7ffff60af3b8 in g_free (/usr/lib64/libglib-2.0.so.0+0x583b8)
    #2 0x45e115 in rm_file_destroy lib/file.c:141
    #3 0x41b92c in rm_fmt_group_destroy lib/formats.c:52
    #4 0x41cf04 in rm_fmt_close lib/formats.c:438
    #5 0x42bbed in rm_session_clear lib/session.c:131
    #6 0x408bd2 in main src/rmlint.c:146
    #7 0x7ffff552e34c in __libc_start_main (/lib64/libc.so.6+0x2534c)
    #8 0x4084c9 in _start (/usr/local/src/rmlint/rmlint+0x4084c9)

0x60d00007c2d0 is located 0 bytes inside of 129-byte region [0x60d00007c2d0,0x60d00007c351)
freed by thread T0 here:
    #0 0x7ffff6efd1a8 in __interceptor_free (/usr/lib64/libasan.so.4+0xdc1a8)
    #1 0x7ffff60af3b8 in g_free (/usr/lib64/libglib-2.0.so.0+0x583b8)

previously allocated by thread T2 (pool-rmlint) here:
    #0 0x7ffff6efd500 in malloc (/usr/lib64/libasan.so.4+0xdc500)
    #1 0x7ffff60af2b8 in g_malloc (/usr/lib64/libglib-2.0.so.0+0x582b8)

Thread T2 (pool-rmlint) created by T0 here:
    #0 0x7ffff6e5ac80 in pthread_create (/usr/lib64/libasan.so.4+0x39c80)
    #1 0x7ffff60f500b  (/usr/lib64/libglib-2.0.so.0+0x9e00b)

SUMMARY: AddressSanitizer: double-free (/usr/lib64/libasan.so.4+0xdc1a8) in __interceptor_free
==23589==ABORTING
[Thread 0x7fffef8f3700 (LWP 24051) exited]
[Inferior 1 (process 23589) exited with code 01]
No stack.
Missing separate debuginfos, use: zypper install libasan4-debuginfo-7.5.0+r278197-4.25.1.x86_64 libgcc_s1-debuginfo-10.3.0+git1587-1.6.4.x86_64 libstdc++6-debuginfo-10.3.0+git1587-1.6.4.x86_64
(gdb)
cebtenzzre commented 2 years ago

That's helpful, but the backtraces are incomplete. If you change the build command to CFLAGS='-fsanitize=address -fno-omit-frame-pointer' ... does the report change?

Suika commented 2 years ago

It does change a bit. One entry has a bit more information.

GNU gdb (GDB; SUSE Linux Enterprise 15) 11.1
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-suse-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://bugs.opensuse.org/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from rmlint...
Starting program: /usr/local/src/rmlint/rmlint -vvv /mnt/temp/jd2/download -g -T all,-badids --xattr-read --xattr-write --write-unfinished
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
[New Thread 0x7ffff10f6700 (LWP 28886)]
[New Thread 0x7ffff08f5700 (LWP 28887)]
▕░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░Thread 0x7ffff10f6700 (LWP 28886) exited]░▒░░▒░░▏                                                                                                                                                              Traversing (13003 usable files / 0 + 0 ignored files / folders)
▕░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▏                                                                                                                    Traversing (13030 usable files / 0 + 0 ignored files / folders)
▕░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░▒░░New Thread 0x7ffff10f6700 (LWP 29636)]
▕░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▏                                                                                                                                                                  Preprocessing (reduces files to 11282 / found 0 other lint)
▕▒░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░New Thread 0x7ffff00f4700 (LWP 29637)]
[New Thread 0x7fffef8f3700 (LWP 29639)]
▕░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▏                                                                                                                                                    Matching (3076 dupes of 2515 originals; 0 B to scan in 0 files, ETA: ...)
▕▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓Thread 0x7fffef8f3700 (LWP 29639) exited]
[Thread 0x7ffff10f6700 (LWP 29636) exited]
[Thread 0x7ffff08f5700 (LWP 28887) exited]
▕░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▏                                                                                                                 Merging files into directories (stand by...)

==> In total 13030 files, whereof 3076 are duplicates in 2515 groups.
==> This equals 853.19 GB of duplicates which could be removed.
==> Scanning took in total 8.905s.

Wrote a json file to: /usr/local/src/rmlint/rmlint.json
Wrote a sh file to: /usr/local/src/rmlint/rmlint.sh
=================================================================
==28750==ERROR: AddressSanitizer: attempting double-free on 0x60d00007c2d0 in thread T0:
    #0 0x7ffff6efd1a8 in __interceptor_free (/usr/lib64/libasan.so.4+0xdc1a8)
    #1 0x7ffff60af3b8 in g_free (/usr/lib64/libglib-2.0.so.0+0x583b8)
    #2 0x4607c2 in rm_file_destroy lib/file.c:141
    #3 0x41bf06 in rm_fmt_group_destroy lib/formats.c:52
    #4 0x41d55a in rm_fmt_close lib/formats.c:438
    #5 0x42c865 in rm_session_clear lib/session.c:131
    #6 0x408bd6 in main src/rmlint.c:146
    #7 0x7ffff552e34c in __libc_start_main (/lib64/libc.so.6+0x2534c)
    #8 0x4084c9 in _start (/usr/local/src/rmlint/rmlint+0x4084c9)

0x60d00007c2d0 is located 0 bytes inside of 129-byte region [0x60d00007c2d0,0x60d00007c351)
freed by thread T0 here:
    #0 0x7ffff6efd1a8 in __interceptor_free (/usr/lib64/libasan.so.4+0xdc1a8)
    #1 0x7ffff60af3b8 in g_free (/usr/lib64/libglib-2.0.so.0+0x583b8)
    #2 0x41bf06 in rm_fmt_group_destroy lib/formats.c:52
    #3 0x41d55a in rm_fmt_close lib/formats.c:438
    #4 0x42c865 in rm_session_clear lib/session.c:131
    #5 0x408bd6 in main src/rmlint.c:146
    #6 0x7ffff552e34c in __libc_start_main (/lib64/libc.so.6+0x2534c)

previously allocated by thread T2 (pool-rmlint) here:
    #0 0x7ffff6efd500 in malloc (/usr/lib64/libasan.so.4+0xdc500)
    #1 0x7ffff60af2b8 in g_malloc (/usr/lib64/libglib-2.0.so.0+0x582b8)

Thread T2 (pool-rmlint) created by T0 here:
    #0 0x7ffff6e5ac80 in pthread_create (/usr/lib64/libasan.so.4+0x39c80)
    #1 0x7ffff60f500b  (/usr/lib64/libglib-2.0.so.0+0x9e00b)

SUMMARY: AddressSanitizer: double-free (/usr/lib64/libasan.so.4+0xdc1a8) in __interceptor_free
==28750==ABORTING
[Thread 0x7ffff00f4700 (LWP 29637) exited]
[Inferior 1 (process 28750) exited with code 01]
No stack.
Missing separate debuginfos, use: zypper install libasan4-debuginfo-7.5.0+r278197-4.25.1.x86_64 libgcc_s1-debuginfo-10.3.0+git1587-1.6.4.x86_64 libstdc++6-debuginfo-10.3.0+git1587-1.6.4.x86_64
(gdb)
cebtenzzre commented 2 years ago

The closest I have gotten to reproducing the issue is this:

ERROR:lib/shredder.c:752:rm_shred_group_free: assertion failed: (self->num_pending == 0)

Which does not occur on the develop branch, so maybe your bug is already fixed?

To debug further: If you download this patch, apply it with patch -Np1 -i /path/to/patch and then rebuild rmlint, it should crash earlier and print a message to explain what is happening. I am assuming the --xattr-read option is the most important, but it might be helpful if you could narrow down which rmlint modes are necessary to reproduce this - is -T df,dd, or even -T df enough?

Suika commented 2 years ago

I've compiled the develop branch and it seems to work just fine. Guess this is only on master. Also, I've used the patch on master and rmlint finished without problems. So I guess the can be closed. If you need info, do reopen the ticket. Otherwise, thanks for the help.

cebtenzzre commented 2 years ago

If you can't reproduce on master anymore, then running develop with --xattr-write must have changed the files on disk so that the issue no longer occurs. My patch cannot prevent the double-free.

@SeeSpotRun It looks like we need a new release soon or a backport of whatever changes may have fixed this, including dcdab3b8c25bc7b9bf67c3472c0afa1940c5d4ec, which removes the implementation of --write-unfinished that uses rm_fmt_write and possibly could have caused an RmFile instance to be written to the formatter twice.

cebtenzzre commented 2 years ago

Minimal reproducer:

$ mkdir -p repr562/a
$ echo 'foo' >repr562/a/1.txt
$ cp repr562/a/1.txt repr562/a/2.txt
$ cp -r repr562/a repr562/b
$ rmlint -o sh:/dev/null -T df,dd --write-unfinished repr562
dupped output file: lib/shredder.c:1713 and lib/shredder.c:1713

lib/shredder.c:1713 is in rm_shred_output_tm_results. Two backtraces for a given file (there should only be one):

#0  rm_shred_output_tm_results (file=0x621000051860, data=0x7fffffffcd80) at lib/shredder.c:1709
#1  0x00005555555b00af in rm_tm_output_file (self=self@entry=0x621000041800, file=0x621000051860) at lib/treemerge.c:661
#2  0x00005555555b034b in rm_tm_write_unfinished_cksums (self=self@entry=0x621000041800, directory=directory@entry=0x60d000007470) at lib/treemerge.c:670
#3  0x00005555555b0c5d in rm_tm_extract (self=self@entry=0x621000041800) at lib/treemerge.c:917
#4  0x00005555555b288b in rm_tm_finish (self=0x621000041800) at lib/treemerge.c:1024
#5  0x000055555558f3b9 in rm_cmd_main (session=session@entry=0x7fffffffcd80) at lib/cmdline.c:1828
#6  0x000055555557d2ad in main (argc=<optimized out>, argv=<optimized out>) at src/rmlint.c:142
#0  rm_shred_output_tm_results (file=0x621000051860, data=0x7fffffffcd80) at lib/shredder.c:1709
#1  0x00005555555b00af in rm_tm_output_file (self=self@entry=0x621000041800, file=0x621000051860) at lib/treemerge.c:661
#2  0x00005555555b062e in rm_tm_output_group (self=self@entry=0x621000041800, group=0x619000002580) at lib/treemerge.c:830
#3  0x00005555555b1101 in rm_tm_extract (self=self@entry=0x621000041800) at lib/treemerge.c:971
#4  0x00005555555b288b in rm_tm_finish (self=0x621000041800) at lib/treemerge.c:1024
#5  0x000055555558f3b9 in rm_cmd_main (session=session@entry=0x7fffffffcd80) at lib/cmdline.c:1828
#6  0x000055555557d2ad in main (argc=<optimized out>, argv=<optimized out>) at src/rmlint.c:142

It looks like the combination of -T df,dd (or just -D) with --write-unfinished just plain doesn't work as it ends up writing files to the formatter twice, once as unique files and again as duplicate files. If this option weren't already removed I would suggest skipping rm_tm_write_unfinished_cksums if find_duplicates is enabled since everything is going to be sent to the formatter anyway.