search

bsc-performance-tools / extrae

Instrumentation framework to generate execution traces of the most used parallel runtimes.
https://tools.bsc.es/extrae
GNU Lesser General Public License v2.1
58 stars 35 forks source link

TLS free() interception causes a segfault with extrae 4.0.0 #66

Closed rodarima closed 2 years ago

rodarima commented 2 years ago

The new version seems to have introduced a problem with the free interception. I'm using a OmpSs-2 program with the libnanosmpitrace.so library, following this example.

It seems that the function xtr_mem_tracked_allocs_remove is called in the free wrapper, which in turn executes the TLS implementation of the glibc, which calls the free function, causing an infinite loop, and later a segfault. I believe this problem is caused by the access to the mallocentries variable:

https://github.com/bsc-performance-tools/extrae/blob/f5f80f6314a8e87e75f2c068b73c73e084760e34/src/tracer/wrappers/MALLOC/malloc_wrapper.c#L143

https://github.com/bsc-performance-tools/extrae/blob/f5f80f6314a8e87e75f2c068b73c73e084760e34/src/tracer/wrappers/MALLOC/malloc_wrapper.c#L87

Here is the backtrace:

(gdb) bt 20
#0  0x00007f2b49132e63 in __tls_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#1  0x00007f2b486cfec5 in xtr_mem_tracked_allocs_remove () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#2  0x00007f2b486d0019 in free () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#3  0x00007f2b4912d7d6 in _dl_update_slotinfo () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#4  0x00007f2b4912d8fc in update_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#5  0x00007f2b49132e68 in __tls_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#6  0x00007f2b486cfec5 in xtr_mem_tracked_allocs_remove () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#7  0x00007f2b486d0019 in free () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#8  0x00007f2b4912d7d6 in _dl_update_slotinfo () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#9  0x00007f2b4912d8fc in update_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#10 0x00007f2b49132e68 in __tls_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#11 0x00007f2b486cfec5 in xtr_mem_tracked_allocs_remove () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#12 0x00007f2b486d0019 in free () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#13 0x00007f2b4912d7d6 in _dl_update_slotinfo () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#14 0x00007f2b4912d8fc in update_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#15 0x00007f2b49132e68 in __tls_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#16 0x00007f2b486cfec5 in xtr_mem_tracked_allocs_remove () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#17 0x00007f2b486d0019 in free () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#18 0x00007f2b4912d7d6 in _dl_update_slotinfo () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#19 0x00007f2b4912d8fc in update_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
(More stack frames follow...)
(gdb) bt -20
#29489 0x00007f2b4912d8fc in update_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29490 0x00007f2b49132e68 in __tls_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29491 0x00007f2b486cfec5 in xtr_mem_tracked_allocs_remove () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#29492 0x00007f2b486d0019 in free () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#29493 0x00007f2b4912d7d6 in _dl_update_slotinfo () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29494 0x00007f2b4912d8fc in update_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29495 0x00007f2b49132e68 in __tls_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29496 0x00007f2b486cfec5 in xtr_mem_tracked_allocs_remove () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#29497 0x00007f2b486d0019 in free () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#29498 0x00007f2b4912d7d6 in _dl_update_slotinfo () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29499 0x00007f2b4912d8fc in update_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29500 0x00007f2b49132e68 in __tls_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29501 0x00007f2b486cfec5 in xtr_mem_tracked_allocs_remove () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#29502 0x00007f2b486d0019 in free () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#29503 0x00007f2b4912d7d6 in _dl_update_slotinfo () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29504 0x00007f2b4912d8fc in update_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29505 0x00007f2b49132e68 in __tls_get_addr () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
#29506 0x00007f2b486cfec5 in xtr_mem_tracked_allocs_remove () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#29507 0x00007f2b486d0019 in free () from /nix/store/pl4zhp9sbklcgj0031f17n12gbwszhqv-extrae-4.0.0/lib/libnanosmpitrace.so
#29508 0x00007f2b4912d7d6 in _dl_update_slotinfo () from /nix/store/33idnvrkvfgd5lsx2pwgwwi955adl6sk-glibc-2.31/lib/ld-linux-x86-64.so.2
ubersawmedic commented 2 years ago

Hello,

Attached you will find a version of Extrae with the calls made to xtr_mem_tracked_allocs_remove inside the instrumented zone, avoiding this infinite loop. Try if this solution works for you, and if you are working with MareNostrum we can set up an install there if you want.

TLSfreeFix.tar.gz