DrMemory crashes - Githubissues

Hello, i wanted to use DrMemory to search for memory leaks in a python module (Which I did in Valgrind and wanted to check out drmemory in comparison)

To be able to test the module, we have a test which consists of a binary that loads the module into a embedded python interpreter and tests the module. Sourcecode: https://github.com/bareos/bareos/blob/master/core/src/plugins/filed/python/test/python-fd-module-tester.cc

Describe the bug I tried different Versions of DrMemory releases, and unfortunately it crashes in all versions I tired (1.10, 2.2, 2.3. 2.3-18696)

To Reproduce Steps to reproduce the behavior:

Build the bareos sourcecode from https://github.com/bareos/bareos
run the module-test verbose to see what PYHTONPATH is required: ctest -V -R bareosfd-python3-module-tester
set the PYTHONPATH: export PYTHONPATH=/home/pstorz/git/master/core/src/plugins/filed/python/test/:/home/pstorz/git/master/b/core/src/plugins/filed/python/python3modules:/home/pstorz/git/master/core/src/plugins/filed/python/pyfiles
run the binary in drmemory: drmemory -- /home/pstorz/git/master/b/core/src/plugins/filed/python/bareosfd-python3-module-tester
- Does the problem go away when running in light mode (pass -light to Dr. Memory)? NO
- Does the problem go away when running with the options -leaks_only -no_count_leaks -no_track_allocs? NO
- Does the problem go away when running under plain DynamoRIO? Do this by running dynamorio/bin32/drrun -- <application and args> or dynamorio/bin64/drrun -- <application and args> depending on the bitwidth of your applicaiton. (Ignore warnings about "incomplete installation".) YES
- What happens with the debug version of Dr. Memory and of its underlying engine DynamoRIO? Try this by passing -debug -dr_debug -pause_at_assert. Are any messages reported? NO, crash still happens

Expected behavior I would expect that drmemory does not crash.

Screenshots or Pasted Text

Running without special switches:

drmemory  --  /home/pstorz/git/master/b/core/src/plugins/filed/python/bareosfd-python3-module-tester
<Application /home/pstorz/git/master/b/core/src/plugins/filed/python/bareosfd-python3-module-tester (1526009).  Dr. Memory internal crash at PC 0x000000007382b176.  Please report this at http://drmemory.org/issues along with the results of running '-debug -dr_debug'.  Program aborted.
Received SIGSEGV at client library pc 0x000000007382b176 in thread 1526009
Base: 0x00007fcdc2a1a000
Registers:eax=0x000000000000002d ebx=0x0000000073b70338 ecx=0x00000000000000ad edx=0x0000000000000000
        esi=0x00007ffc56b80920 edi=0x00007fcdc2e15f40 esp=0x00007ffc56b808d0 ebp=0x00007fcdc2e15f40
        r8 =0x00000000ffffff51 r9 =0x0000000000000020 r10=0x0000000000000000 r11=0x00007fcdc2b7f477
        r12=0x0000000000000000 r13=0x000000000000002d r14=0x00007fcdc2e15f40 r15=0x0000000000000000
        eflags=0x0000000000010207
2.3.18696-0-(Mar 20 2021 02:40:59) 
-no_dynamic_options -logdir '/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/drmemory/logs/dynamorio' -client_lib '/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/bin64/release/libdrmemorylib.so;0;-logdir `/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/drmemory/logs` -symcache_dir `/home/pstorz/drmemory/DrMemory-Linux-2.3.
0x00007fcdc2e15f40 0x702f656d6f682f60
/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/dynamorio/lib64/release/libdynamorio.so=0x00007fcdc2a1a000
/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/bin64/release/libdrmemorylib.so=0x0000000073800000
/usr/lib64/libc.so.6=0x00007fcbc250b000
/usr/lib64/ld-linux-x86-64.so.2=0x00007fcbc29bb000>

Running with debug version:

drmemory -debug -dr_debug -pause_at_assert  --  /home/pstorz/git/master/b/core/src/plugins/filed/python/bareosfd-python3-module-tester
<Starting application /home/pstorz/git/master/b/core/src/plugins/filed/python/bareosfd-python3-module-tester (1525650)>
<Initial options = -no_dynamic_options -logdir '/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/drmemory/logs/dynamorio' -client_lib '/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/bin64/debug/libdrmemorylib.so;0;`-pause_at_assert` -logdir `/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/drmemory/logs` -symcache_dir `/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/drmemory/logs/symcache` -resfile 1525650 ' -client_lib64 '/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/bin64/debug/libdrmemorylib.so;0;`-pause_at_assert` -logdir `/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/drmemory/logs` -symcache_dir `/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/drmemory/logs/symcache` -resfile 1525650 ' -code_api -stack_size 56K -signal_stack_size 32K -disable_traces -no_enable_traces -max_elide_jmp 0 -max_elide_call 0 -no_shared_traces -bb_ibl_targets -bb_single_restore_prefix -no_shared_trace_ibl_routine -no_enable_reset -no_reset_at_switch_to_os_at_vmm_limit -reset_at_vmm_percent_free_limit 0 -no_reset_at_vmm_full -reset_at_commit_free_limit 0B -reset_every_nth_pending 0 -vm_size 256M -early_inject -emulate_brk -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct >
<Paste into GDB to debug DynamoRIO clients:
set confirm off
add-symbol-file '/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/bin64/debug/libdrmemorylib.so' 0x000000007381d000
add-symbol-file '/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/dynamorio/lib64/debug/libdynamorio.so' 0x00007fabebe10f70
add-symbol-file '/usr/lib64/libc.so.6' 0x00007fa9eb8de690
add-symbol-file '/usr/lib64/ld-linux-x86-64.so.2' 0x00007fa9ebd69090
>
<Application /home/pstorz/git/master/b/core/src/plugins/filed/python/bareosfd-python3-module-tester (1525650).  Dr. Memory internal crash at PC 0x000000007386fad6.  Please report this at http://drmemory.org/issues.  Program aborted.
Received SIGSEGV at client library pc 0x000000007386fad6 in thread 1525650
Base: 0x00007fabebdc7000
Registers:eax=0x00000000000000c0 ebx=0x00007fabec4308d0 ecx=0x0000000000000020 edx=0x0000000000000000
        esi=0x00007ffd9547b610 edi=0x00007fabec41dac0 esp=0x00007ffd9547b5d0 ebp=0x00007ffd9547b5f0
        r8 =0x00007fa9ebe1d700 r9 =0x00007fa9ebdf7268 r10=0x00007fabec0da5fd r11=0x0000000000000246
        r12=0x0000000000000000 r13=0x0000000000000000 r14=0x0000000000000000 r15=0x0000000000000000
        eflags=0x0000000000010206
2.3.18696-0-(Mar 20 2021 02:42:18) 
-no_dynamic_options -logdir '/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/drmemory/logs/dynamorio' -client_lib '/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/bin64/debug/libdrmemorylib.so;0;`-pause_at_assert` -logdir `/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/drmemory/logs` -symcache_dir `/home/pstorz/drmemory/Dr
0x00007ffd9547b5f0 0x000000007387142d
0x00007ffd9547be40 0x000000007381dbbc
0x00007ffd9547be60 0x0000000073827fc5
0x00007ffd9547c070 0x00007fabebfb4999
0x00007ffd9547c0b0 0x00007fabebe22fe2
0x00007ffd9547c8e0 0x00007fabec0c672d
0x00007ffd9547d990 0x00007fabec074217
/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/dynamorio/lib64/debug/libdynamorio.so=0x00007fabebdc7000
/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/bin64/debug/libdrmemorylib.so=0x0000000073800000
/usr/lib64/libc.so.6=0x00007fa9eb8b8000
/usr/lib64/ld-linux-x86-64.so.2=0x00007fa9ebd68000>

Versions

What version of Dr. Memory are you using?DrMemory-Linux-2.3.18696
Does the latest build from https://github.com/DynamoRIO/drmemory/wiki/Latest-Build solve the problem? No, as this is already the latest build
- What operating system version are you running on?Fedora Linux 33 64Bit
- Is your application 32-bit or 64-bit?64Bit

Additional context I have the feeling that it has to do with python itself, as it also crashes when I run drmemory on python3 executing a script containing print ("hello") also crashes:

drmemory -- /bin/python3.9 ./test.py 
<Application /usr/bin/python3.9 (1527055).  Dr. Memory internal crash at PC 0x00007fc69e596176.  Please report this at http://drmemory.org/issues along with the results of running '-debug -dr_debug'.  Program aborted.
Received SIGSEGV at client library pc 0x00007fc69e596176 in thread 1527055
Base: 0x0000000071000000
Registers:eax=0x000000000000002d ebx=0x00007fc69e8db338 ecx=0x00000000000000ad edx=0x0000000000000000
        esi=0x00007fff052c0850 edi=0x00000000713fbf40 esp=0x00007fff052c0800 ebp=0x00000000713fbf40
        r8 =0x00000000ffffff51 r9 =0x0000000000000020 r10=0x0000000000000000 r11=0x0000000071165477
        r12=0x0000000000000000 r13=0x000000000000002d r14=0x00000000713fbf40 r15=0x0000000000000000
        eflags=0x0000000000010207
2.3.18696-0-(Mar 20 2021 02:40:59) 
-no_dynamic_options -logdir '/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/drmemory/logs/dynamorio' -client_lib '/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/bin64/release/libdrmemorylib.so;0;-logdir `/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/drmemory/logs` -symcache_dir `/home/pstorz/drmemory/DrMemory-Linux-2.3.
0x00000000713fbf40 0x702f656d6f682f60
/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/dynamorio/lib64/release/libdynamorio.so=0x0000000071000000
/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/bin64/release/libdrmemorylib.so=0x00007fc69e56b000
/usr/lib64/libc.so.6=0x00007fc6b2061000
/usr/lib64/ld-linux-x86-64.so.2=0x00007fc6b2511000>

Hello Derek, thanks for this hint.

Running it in gdb and passing the SIGILL, I get the following output from gdb:

gdb drmemory 
GNU gdb (GDB) Fedora 10.1-4.fc33
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
add symbol table from file "/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/bin64/debug/libdrmemorylib.so" at
        .text_addr = 0x7381d000
add symbol table from file "/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/dynamorio/lib64/debug/libdynamorio.so" at
        .text_addr = 0x71049f70
add symbol table from file "/usr/lib64/libc.so.6" at
        .text_addr = 0xf7a95690
add symbol table from file "/usr/lib64/ld-linux-x86-64.so.2" at
        .text_addr = 0xf7fa2090
Reading symbols from drmemory...
(gdb) run -debug -dr_debug  -- /home/pstorz/git/master/b/core/src/plugins/filed/python/bareosfd-python3-module-tester
Starting program: /home/pstorz/drmemory/DrMemory-Linux-2.3.18696/bin64/drmemory -debug -dr_debug  -- /home/pstorz/git/master/b/core/src/plugins/filed/python/bareosfd-python3-module-tester
process 221746 is executing new program: /home/pstorz/drmemory/DrMemory-Linux-2.3.18696/dynamorio/lib64/debug/libdynamorio.so
<Starting application /home/pstorz/git/master/b/core/src/plugins/filed/python/bareosfd-python3-module-tester (221746)>

Program received signal SIGILL, Illegal instruction.
syscall_ready () at /home/runner/work/drmemory/drmemory/dynamorio/core/drlibc/drlibc_x86.asm:184
184     /home/runner/work/drmemory/drmemory/dynamorio/core/drlibc/drlibc_x86.asm: Datei oder Verzeichnis nicht gefunden.
(gdb) c
Continuing.
<Initial options = -no_dynamic_options -logdir '/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/drmemory/logs/dynamorio' -client_lib '/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/bin64/debug/libdrmemorylib.so;0;-logdir `/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/drmemory/logs` -symcache_dir `/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/drmemory/logs/symcache` -resfile 221746 ' -client_lib64 '/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/bin64/debug/libdrmemorylib.so;0;-logdir `/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/drmemory/logs` -symcache_dir `/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/drmemory/logs/symcache` -resfile 221746 ' -code_api -stack_size 56K -signal_stack_size 32K -disable_traces -no_enable_traces -max_elide_jmp 0 -max_elide_call 0 -no_shared_traces -bb_ibl_targets -bb_single_restore_prefix -no_shared_trace_ibl_routine -no_enable_reset -no_reset_at_switch_to_os_at_vmm_limit -reset_at_vmm_percent_free_limit 0 -no_reset_at_vmm_full -reset_at_commit_free_limit 0B -reset_every_nth_pending 0 -vm_size 256M -early_inject -emulate_brk -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct >
<Paste into GDB to debug DynamoRIO clients:
set confirm off
add-symbol-file '/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/bin64/debug/libdrmemorylib.so' 0x000000007381d000
add-symbol-file '/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/dynamorio/lib64/debug/libdynamorio.so' 0x0000000071049f70
add-symbol-file '/usr/lib64/libc.so.6' 0x00007ffdf7a95690
add-symbol-file '/usr/lib64/ld-linux-x86-64.so.2' 0x00007ffff7fa2090
>

Program received signal SIGSEGV, Segmentation fault.
0x000000007386fad6 in get_option_word (
    s=0x71656ac0 <client_libs+2592> "-logdir `/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/drmemory/logs` -symcache_dir `/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/drmemory/logs/symcache` -resfile 221746 ", 
    buf=0x7fffffffb530 "/debug/libdrmemorylib.so' 0x000000007381d000\nadd-symbol-file '/home/pstorz/drmemory/DrMemory-Linux-2.3.18696/dynamorio/lib64/deb") at /home/runner/work/drmemory/drmemory/drmemory/options.c:174
174     /home/runner/work/drmemory/drmemory/drmemory/options.c: Datei oder Verzeichnis nicht gefunden.

Unfortunately, it is not possible to do anything at this point as any gdb command results in:

(gdb) n
Warning:
Cannot insert breakpoint -46.
Cannot access memory at address 0x3d701
Cannot insert breakpoint -45.
Cannot access memory at address 0x3d797
Cannot insert breakpoint -44.
Cannot access memory at address 0x1113d1

Command aborted.

It seems to crash in ``` /home/runner/work/drmemory/drmemory/drmemory/options.c:174

DynamoRIO / drmemory

DrMemory crashes #2395