CRASH Segfault with DrMemory

[Desperado17]

Received SIGSEGV at client library pc 0x00007f36e6555fa9 in thread 15974 Base: 0x00007f36ff0c5000 Registers:eax=0x0000000000000033 ebx=0x00007f34e72ed298 ecx=0x0000000000000050 edx=0x0000000000000000 esi=0x0000000000000050 edi=0x00007f34e72e5e50 esp=0x00007f34e72e5d30 ebp=0x00007f34e72e5e50 r8 =0x00007f35beeaf5b0 r9 =0x0000000000000020 r10=0x00007f36ff2a3000 r11=0x000000000000008f r12=0x0000000000000033 r13=0x00007f350d36f054 r14=0x0000000000000074 r15=0x0000000000000082 eflags=0x0000000000010203 2.5.19327-0-(Dec 1 2022 15:41:46) -no_dynamic_options -logdir '/home/user/Downloads/DrMemory-Linux-2.5.19327/drmemory/logs/dynamorio' -client_lib '/home/user/Downloads/DrMemory-Linux-2.5.19327/bin64/debug/libdrmemorylib.so;0;-light -no_check_uninitialized -logdir /home/user/Downloads/DrMemory-Linux-2.5.19327/drmemory/logs -symcache_dir `/home/user/ 0x00007f34e72e5e50 0x0000000000000008 /home/user/Downloads/DrMemory-Linux-2.5.19327/dynamorio/lib64/release/libdynamorio.so=0x00007f36ff0c5000 /home/user/Downloads/DrMemory-Linux-2.5.19327/bin64/debug/libdrmemorylib.so=0x00007f36e6302000 /usr/local/lib/libunwind.so.8=0x00007f36fec7e000 /lib/x86_64-linux-gnu/liblzma.so.5=0x00007f36fe438000 /lib/x86_64-linux-gnu/libpthread.so.0=0x00007f36fe013000 /lib/x86_64-linux-gnu/libdl.so.2=0x00007f36fe233000 /lib/x86_64-linux-gnu/libc.so.6=0x000>

~/Downloads/DrMemory-Linux-2.5.19327/bin64/drmemory -debug -light -no_check_uninitialized -- ./myapp

or

drrun -debug -t drmemory -light -no_check_uninitialized -- ./myapp

Only with

-leaks_only -no_count_leaks -no_track_allocs

program will start correctly, even one missing will trigger a fatal sigsegv.

[Desperado17]

See this discussion:

https://groups.google.com/g/dynamorio-users/c/IAFjO9gBPJ0

Ok,the nullptr in rdx seems to come via rbx from the rax register which is last written here:

0 0x00007ffff7611374 in __ctype_b_loc () at ../include/ctype.h:42

1 0x00007fffd992573f in cpp_demangle_read_uqname (ddata=0x7ffddaaee140) at libelftc_dem_gnu3.c:3109

https://sourceforge.net/p/elftoolchain/code/3530/tree//trunk/libelftc/libelftc_dem_gnu3.c#l3109

/* source name */
if (ELFTC_ISDIGIT(*ddata->cur) != 0)

I suspect that the lookup table inside the isdigit function is not initialized similar to isspace in this ticket:

https://github.com/DynamoRIO/drmemory/issues/2425

[derekbruening]

This doesn't have enough information for reproducing: no platform or OS or application information. There are bug templates that ask all these questions (https://github.com/DynamoRIO/dynamorio/issues/new?assignees=&labels=&template=bug_report.md&title= and https://github.com/DynamoRIO/drmemory/issues/new?assignees=&labels=&template=bug_report.md&title=) ; looks like they've been bypassed here which doesn't help. Given that it doesn't say it happens with hello,world and every application tried, it sounds like it is limited to "myapp".

If it is an issue with the private loader, it is becoming impractical to maintain it in the face of hidden dependencies in ld.so, libc, and pthreads. If you are not able to figure out what libc internal weirdness is going on here probably the thing to do is link everything the tool needs statically and bail on the private loader.

[Desperado17]

Ok, some infos about the system I currently use:

ii libc-bin 2.35-0ubuntu3.1 amd64 GNU C Library: Binaries ii libc-dev-bin 2.35-0ubuntu3.1 amd64 GNU C Library: Development binaries ii libc-devtools 2.35-0ubuntu3.1 amd64 GNU C Library: Development tools ii libc6:amd64 2.35-0ubuntu3.1 amd64 GNU C Library: Shared libraries ii libc6:i386 2.35-0ubuntu3.1 i386 GNU C Library: Shared libraries ii libc6-dbg:amd64 2.35-0ubuntu3.1 amd64 GNU C Library: detached debugging symbols ii libc6-dev:amd64 2.35-0ubuntu3.1 amd64 GNU C Library: Development Libraries and Header Files ii libc6-i386 2.35-0ubuntu3.1 amd64 GNU C Library: 32-bit shared libraries for AMD64

No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04.2 LTS Release: 22.04 Codename: jammy

I cannot say a lot about the target application except that it is multithreaded (150+ threads) and proprietary. I tried to reproduce the problem by building elftools locally then put the string in addr2line but to no avail.

I suspect that the tables where the locales are stored in disappear for some reason: https://elixir.bootlin.com/glibc/glibc-2.35/source/locale/C-ctype.c

[Desperado17]

Addendum:

• The app uses dlopen, dlsym to open and address an external library. Could this be a problem?
• Is there a way to avoid this codepath with decreased functionality? All I need are call stacks, they don't have to be symbolized.

[Desperado17]

Greetings, can you tell me which parameters you use for building elftools for the nightlies? I would like to try defining isdigit locally with the default implementation which should suffice for my x64 environment.

Regards

Btw: Do these dwarf strings really need the locale capable version of isdigit? Wouldn't the standard version ala c - '0' <= 9 work for most cases?

[derekbruening]

I think you're looking for: https://github.com/DynamoRIO/dynamorio/blob/master/ext/drsyms/libelftc/rebuild_elftc.sh

[Desperado17]

I'll try. Btw is there a simple method to build dynamorio with drmemory from dynamorio master?

[derekbruening]

It is built in the automated packages so mirroring its steps should do it: https://github.com/DynamoRIO/dynamorio/blob/master/.github/workflows/ci-package.yml#L68

[Desperado17]

Okay, so the source directory has a drmemory folder now. Whats missing is the cmake / build command I tried calling ./suite/runsuite_wrapper.pl automated_ci but it finished with errors. Just calling cmake .. on the existing dynamorio build folder doesn't suffice.

[Desperado17]

Ok, I built drmemory with default cmake ../ and modified elftoolchain that replaces the libc isdigit call with a simple internal ascii definition:

int isdigit_custom ( int c ) { return c >= '0' && c <= '9'; }

drmemory did not crash. drmemory wasn't very fast though. Will cmake .. build in release mode? x64 Ubuntu 22.04.

[Desperado17]

Would this be an acceptable quickfix for you?

[jackgallagher-arm]

I have seen what looks like the same crash running the code_api|client.drsyms-test test on AArch64 RHEL 8. It looks like its being caused by privload_os_finalize() not calling __libc_early_init() because RHEL 8 has glibc 2.28 so it fails the 2.34+ version check.

#0  0x000000007602b35c in cpp_demangle_read_uqname (ddata=ddata@entry=0xffffffffc488) at libelftc_dem_gnu3.c:3109
#1  0x000000007602837c in cpp_demangle_read_nested_name (ddata=0xffffffffc488) at libelftc_dem_gnu3.c:1629
#2  cpp_demangle_read_name (ddata=ddata@entry=0xffffffffc488) at libelftc_dem_gnu3.c:1499
#3  0x0000000076028668 in cpp_demangle_read_encoding (ddata=ddata@entry=0xffffffffc488) at libelftc_dem_gnu3.c:1432
#4  0x000000007602beec in cpp_demangle_gnu3 (org=org@entry=0xfffdbdbe910f "_ZN12_GLOBAL__N_13Foo3BarEi.isra.0")
    at libelftc_dem_gnu3.c:290
#5  0x0000000076022860 in demangle (rc=<optimized out>, style=<optimized out>, s=0xfffdbdbe910f "_ZN12_GLOBAL__N_13Foo3BarEi.isra.0")
    at elftc_demangle.c:91
#6  elftc_demangle (mangledname=0xfffdbdbe910f "_ZN12_GLOBAL__N_13Foo3BarEi.isra.0", 
    buffer=0xfffdbe226260 '\315' <repeats 199 times>, <incomplete sequence \315>..., bufsize=1024, flags=<optimized out>)
    at elftc_demangle.c:91
#7  0x000000007600976c in drsym_unix_demangle_symbol (dst=0xfffdbe226260 '\315' <repeats 199 times>, <incomplete sequence \315>..., 
    dst_sz=1024, mangled=0xfffdbdbe910f "_ZN12_GLOBAL__N_13Foo3BarEi.isra.0", flags=2)
    at dynamorio/ext/drsyms/drsyms_unix_common.c:761
#8  0x0000000076007e94 in drsym_demangle_symbol (dst=0xfffdbe226260 '\315' <repeats 199 times>, <incomplete sequence \315>..., 
    dst_sz=1024, mangled=0xfffdbdbe910f "_ZN12_GLOBAL__N_13Foo3BarEi.isra.0", flags=2)
    at dynamorio/ext/drsyms/drsyms_unix_frontend.c:315
#9  0x0000000076009114 in drsym_demangle_helper (sym=0xfffdbdbe910f "_ZN12_GLOBAL__N_13Foo3BarEi.isra.0", flags=2)
    at dynamorio/ext/drsyms/drsyms_unix_common.c:506
#10 0x0000000076009318 in drsym_fill_symtable_cb (sym=0xfffdbdbe910f "_ZN12_GLOBAL__N_13Foo3BarEi.isra.0", modoffs=4912, data=0x4ec0e698)
    at dynamorio/ext/drsyms/drsyms_unix_common.c:568
#11 0x0000000076008e28 in symsearch_symtab (mod=0x4ec0e698, callback=0x760091d8 <drsym_fill_symtable_cb>, callback_ex=0x0, 
    info_size=104, data=0x4ec0e698, flags=0) at dynamorio/ext/drsyms/drsyms_unix_common.c:419
#12 0x00000000760094b8 in drsym_unix_lookup_symbol (mod_in=0x4ec0e698, symbol=0xffffffffc820 "client.drsyms-test!exe_export", 
    modoffs=0xffffffffc920, flags=1) at /home/jacgal01/dynamorio/ext/drsyms/drsyms_unix_common.c:654
#13 0x00000000760079fc in drsym_lookup_symbol_local (modpath=0xfffdbe1ddab0 "suite/tests/bin/client.drsyms-test", 
    symbol=0xffffffffc820 "client.drsyms-test!exe_export", modoffs=0xffffffffc920, flags=1)
    at dynamorio/ext/drsyms/drsyms_unix_frontend.c:127
#14 0x0000000076007d38 in drsym_lookup_symbol (modpath=0xfffdbe1ddab0 "debug/suite/tests/bin/client.drsyms-test", 
    symbol=0xffffffffc820 "client.drsyms-test!exe_export", modoffs=0xffffffffc920, flags=1)
    at dynamorio/ext/drsyms/drsyms_unix_frontend.c:256
#15 0x00000000720021b0 in lookup_and_wrap (modpath=0xfffdbe1ddab0 "debug/suite/tests/bin/client.drsyms-test", 
    modbase=0x400000 "\177ELF\002\001\001", modname=0xffffffffc948 "client.drsyms-test", symbol=0x72003c88 "exe_export", flags=1)
    at dynamorio/suite/tests/client-interface/drsyms-test.dll.cpp:269
#16 0x0000000072002448 in lookup_exe_syms () at dynamorio/suite/tests/client-interface/drsyms-test.dll.cpp:317
#17 0x0000000072001d60 in dr_init (id=0) at dynamorio/suite/tests/client-interface/drsyms-test.dll.cpp:90
#18 0x0000ffffbe40c7a4 in instrument_init () at dynamorio/core/lib/instrument.c:774
#19 0x0000ffffbe26469c in dynamorio_app_init_part_two_finalize () at dynamorio/core/dynamo.c:716
#20 0x0000ffffbe6a0f34 in privload_early_inject (sp=0xffffffffe7f0, old_libdr_base=0x0, old_libdr_size=0)
    at dynamorio/core/unix/loader.c:2279
#21 0x0000ffffbe662c7c in _start () at dynamorio/core/arch/aarchxx/aarchxx.asm:68