Closed derekbruening closed 7 months ago
Debug build of DR works fine.
DrMemory doesn't seem to be running release-build tests pre-commit? What about DR which does do release-build tests: did drcov tests all pass? What is different here?
I spent some time digging further into this. There is some weird pthread lock behavior.
__readers == 0xb has 2 set for this single-threaded app!
It is set to 0x8 and then 0xb here:
(gdb) p &cu_die->cu.abbrev_hash.resize_rwl.__data.__readers
$8 = (unsigned int *) 0x7ffde464b940
(gdb) p sizeof(cu_die->cu.abbrev_hash.resize_rwl.__data.__readers)
$9 = 4
(gdb) watch *(int*)0x7ffde464b940
Hardware watchpoint 6: *(int*)0x7ffde464b940
(gdb) x/2wx 0x7ffde464b940
0x7ffde464b940: 0x00000000 0x00000000
(gdb) c
Continuing.
Hardware watchpoint 6: *(int*)0x7ffde464b940
Old value = 0
New value = 8
0x00007ffff78d41c2 in ___pthread_rwlock_tryrdlock (rwlock=rwlock@entry=0x7ffde464b940) at ./nptl/pthread_rwlock_tryrdlock.c:88
88 ./nptl/pthread_rwlock_tryrdlock.c: No such file or directory.
(gdb) x/2wx 0x7ffde464b940
0x7ffde464b940: 0x00000008 0x00000000
(gdb) c
Continuing.
Hardware watchpoint 6: *(int*)0x7ffde464b940
Old value = 8
New value = 11
0x00007ffff78d43ba in __pthread_rwlock_wrunlock (rwlock=0x7ffde464b940) at ./nptl/pthread_rwlock_common.c:568
568 ./nptl/pthread_rwlock_common.c: No such file or directory.
(gdb) bt
#0 0x00007ffff78d43ba in __pthread_rwlock_wrunlock (rwlock=0x7ffde464b940) at ./nptl/pthread_rwlock_common.c:568
#1 ___pthread_rwlock_unlock (rwlock=rwlock@entry=0x7ffde464b940) at ./nptl/pthread_rwlock_unlock.c:40
#2 0x00007fffe3e3a15b in Dwarf_Abbrev_Hash_find (htab=htab@entry=0x7ffde464b8f0, hval=hval@entry=5)
at /home/runner/work/drmemory/drmemory/dynamorio/third_party/elfutils/lib/dynamicsizehash_concurrent.c:486
#3 0x00007fffe3e3baa4 in __libdw_findabbrev (cu=0x7ffde464b8a8, code=5) at /home/runner/work/drmemory/drmemory/dynamorio/third_party/elfutils/libdw/dwarf_tag.c:48
#4 0x00007fffe3e3a8aa in __libdw_dieabbrev (readp=<synthetic pointer>, die=0x7ffde3c9c830)
at /home/runner/work/drmemory/drmemory/dynamorio/third_party/elfutils/libdw/libdwP.h:762
#5 __libdw_find_attr (die=die@entry=0x7ffde3c9c830, search_name=search_name@entry=16, codep=codep@entry=0x7ffde3c9c6d0, formp=formp@entry=0x7ffde3c9c6d4)
(gdb) memquery $pc
7ffff786c000-7ffff79c1000 r-xp 00026000 fe:01 21234345 /usr/lib/x86_64-linux-gnu/libc.so.6
That's that same Dwarf_Abbrev_Hash_find calling:
pthread_rwlock_unlock(&htab->resize_rwl);
Which apparently will unlock either a write or read lock. But it seems weird that it did tryrdlock and then a write unlock??
A higher-level approach: it reproduces on DR on the client.drcallstack-test
test on my local machine in release build only; works in debug there. That test passes on GA CI in release build, however, so it is limited to certain libraries or toolchains: and as later work shows it seems to depend on the glibc version.
I tracked it down by tweaking debug build to look like release. When I disabled DEBUG_MEMORY and HEAP_ACCOUNTING, debug build then reproduced the hang, even at -checklevel 0
. I then narrowed it down to the memset to 0xab in heap_mmap_ex() which is what prevents the hang. Enabling just that memset in release build: works! A memset to 0 hangs; any non-0 value avoids the hang.
Narrowing further: it's the memset for the mmap in privload_tls_init(). Narrowing within that 2-page region: it's offset 0x19d0, which turns into %fs:0x2d0
, which is the pthread tid
field.
It seems that if the tid field is zero, the pthread lock checks for whether a lock is owned incorrectly think the lock is indeed owned since the owner field of the lock is zero and that matches the (invalid) tid of zero. This explains the weird lock behavior in the callstack above at the hang point.
I thought this may be related to #5437 and the libc/ld GLRO vars: but the variables initialized there are not in the TLS mmap and are unrelated. This is related to that issue in the sense that it's coming from the merging of pthreads into libc and having too tight of a coupling with ld.so with undocumented initialization between them, making if very hard to replace ld.so as DR is doing.
For a short term solution, my plan is to locate this field (%fs:0x2d0
for my glibc 2.37) programmatically by decoding an exported routine known to reference it up front such as pthread_mutex_consistent
. Long-term solutions are discussed in #5437.
Noticed this in Dr. Memory where with the update to use elfutils libdw, release build seem to hang when looking up line numbers:
Let it run w/ breakpoints in drsyms_dw.c -- never returns out of libdw.
Seems to be stuck here inside Dwarf_Abbrev_Hash_find at third_party/elfutils/lib/dynamicsizehash_concurrent.c:476:
htab is maybe not initialized properly?