google / sanitizers

AddressSanitizer, ThreadSanitizer, MemorySanitizer
Other
11.56k stars 1.04k forks source link

ASAN and TSAN hang when use with wxWidgets linked against gtk >= 3.6 #1270

Open Amanda-M-UK opened 4 years ago

Amanda-M-UK commented 4 years ago

I am experiencing a problem with my c++ application hanging at start when using ASAN or TSAN.

Version Details

Details

If I compile my application with wxWidgets 3.1.3 linked against gtk2 or gtk 3 <= 3.4 then it ASAN and TSAN works correctly. However, if I use gtk >= 3.6 then the application hangs at wxEntry(argc, argv);

This behaviour can be reproduced by using the standard wxWidgets "Hello World" example. (available from https://docs.wxwidgets.org/stable/overview_helloworld.html)

The output of running gdb ./my_test_application is as follows:

[nm@nm-primary Test]$ gdb ./my_test_application
warning: Found custom handler for signal 7 (Bus error) preinstalled.
warning: Found custom handler for signal 8 (Floating point exception) preinstalled.
warning: Found custom handler for signal 11 (Segmentation fault) preinstalled.
Some signal dispositions inherited from the environment (SIG_DFL/SIG_IGN)
won't be propagated to spawned programs.
GNU gdb (GDB) Fedora 9.1-5.fc32
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./my_test_application...
r(gdb) r
Starting program: .. my_test_application 
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.31-2.fc32.x86_64
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

The application has hung at _Using host libthread_db library "/lib64/libthreaddb.so.1" Can anybody recommend a fix for this problem?

Kind regards Amanda

kcc commented 4 years ago

At the very least, try to interrupt the program and see where it has stuck (in gdb, first press ctrl-C, then type bt<enter>)

Amanda-M-UK commented 4 years ago

Many thanks - here's the output from gdb.

[nm@nm-primary Linux_R_Race]$ gdb ./test
GNU gdb (GDB) Fedora 9.1-5.fc32
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./test...
(gdb) r
Starting program: /home/nm/PROJECTS/Test/Linux_R_Race/test 
Missing separate debuginfos, use: dnf debuginfo-install glibc-2.31-2.fc32.x86_64
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
^C
Program received signal SIGINT, Interrupt.
0x00007ffff58e1b2d in pthread_rwlock_wrlock () from /lib64/libpthread.so.0
Missing separate debuginfos, use: dnf debuginfo-install at-spi2-atk-2.34.2-1.fc32.x86_64 at-spi2-core-2.36.0-1.fc32.x86_64 atk-2.36.0-1.fc32.x86_64 bzip2-libs-1.0.8-2.fc32.x86_64 cairo-1.16.0-8.fc32.x86_64 cairo-gobject-1.16.0-8.fc32.x86_64 cyrus-sasl-lib-2.1.27-4.fc32.x86_64 dbus-libs-1.12.18-1.fc32.x86_64 expat-2.2.8-2.fc32.x86_64 fontconfig-2.13.92-9.fc32.x86_64 freetype-2.10.1-2.fc32.x86_64 fribidi-1.0.9-1.fc32.x86_64 gdk-pixbuf2-2.40.0-2.fc32.x86_64 glib2-2.64.3-2.fc32.x86_64 graphite2-1.3.14-1.fc32.x86_64 gtk3-3.24.20-1.fc32.x86_64 harfbuzz-2.6.4-3.fc32.x86_64 keyutils-libs-1.6-4.fc32.x86_64 krb5-libs-1.18.2-9.fc32.x86_64 libX11-1.6.9-3.fc32.x86_64 libXcomposite-0.4.5-2.fc32.x86_64 libXcursor-1.2.0-2.fc32.x86_64 libXdamage-1.1.5-2.fc32.x86_64 libXext-1.3.4-3.fc32.x86_64 libXfixes-5.0.3-11.fc32.x86_64 libXi-1.7.10-3.fc32.x86_64 libXinerama-1.1.4-5.fc32.x86_64 libXtst-1.2.3-11.fc32.x86_64 libblkid-2.35.2-1.fc32.x86_64 libbrotli-1.0.7-10.fc32.x86_64 libcurl-7.69.1-4.fc32.x86_64 libdatrie-0.2.9-11.fc32.x86_64 libepoxy-1.5.4-2.fc32.x86_64 libgcc-10.1.1-1.fc32.x86_64 libgcrypt-1.8.5-3.fc32.x86_64 libglvnd-1.3.1-1.fc32.x86_64 libgomp-10.1.1-1.fc32.x86_64 libnghttp2-1.41.0-1.fc32.x86_64 libpng-1.6.37-3.fc32.x86_64 libpsl-0.21.0-4.fc32.x86_64 libselinux-3.0-3.fc32.x86_64 libssh-0.9.4-2.fc32.x86_64 libstdc++-10.1.1-1.fc32.x86_64 libthai-0.1.28-4.fc32.x86_64 libtool-ltdl-2.4.6-33.fc32.x86_64 libtsan-10.1.1-1.fc32.x86_64 libwayland-cursor-1.18.0-1.fc32.x86_64 libxcb-1.13.1-4.fc32.x86_64 libxcrypt-4.4.16-3.fc32.x86_64 libxkbcommon-0.10.0-2.fc32.x86_64 mesa-libGLU-9.0.1-2.fc32.x86_64 openldap-2.4.47-4.fc32.x86_64 pango-1.44.7-2.fc32.x86_64 systemd-libs-245.6-2.fc32.x86_64 zlib-1.2.11-21.fc32.x86_64
(gdb) bt
#0  0x00007ffff58e1b2d in pthread_rwlock_wrlock () at /lib64/libpthread.so.0
#1  0x00007ffff573e44e in set_binding_values.part () at /lib64/libc.so.6
#2  0x00007ffff573e895 in bindtextdomain () at /lib64/libc.so.6
#3  0x00007ffff69ea36a in gettext_initialization () at /lib64/libgtk-3.so.0
#4  0x00007ffff69eb493 in gtk_parse_args () at /lib64/libgtk-3.so.0
#5  0x00007ffff69eb55b in gtk_init_check () at /lib64/libgtk-3.so.0
#6  0x0000000000dfcee0 in wxApp::Initialize(int&, wchar_t**) (this=0x7b5800000000, argc_=@0x19d5a90: 1, argv_=0x7b04000008c0)
    at ../src/gtk/app.cpp:375
#7  0x00000000010850fd in wxEntryStart(int&, wchar_t**) (argc=@0x19d5a90: 1, argv=0x7b04000008c0) at ../src/common/init.cpp:333
#8  0x0000000001085552 in wxInitialize(int&, wchar_t**) (argc=@0x19d5a90: 1, argv=0x7b04000008c0) at ../src/common/init.cpp:544
#9  0x000000000108575b in wxInitializer::wxInitializer(int&, wchar_t**) (this=0x7fffffffdb0f, argc=@0x19d5a90: 1, argv=0x7b04000008c0)
    at ../include/wx/init.h:92
#10 0x0000000001085396 in wxEntry(int&, wchar_t**) (argc=@0x19d5a90: 1, argv=0x7b04000008c0) at ../src/common/init.cpp:477
#11 0x00000000010854d6 in wxEntry(int&, char**) (argc=@0x7fffffffdb5c: 1, argv=0x7fffffffdc58) at ../src/common/init.cpp:519
#12 0x000000000049a4d5 in main(int, char**) (argc=<optimized out>, argv=0x7fffffffdc58) at ../src/common/main.cpp:130

The line of code at 130 is:

int ret = wxEntry(argc, argv);

kcc commented 4 years ago

Looks like maybe a deadlock. Since ASAN and TSAN are slower than the usual execution, they are more likely to trigger a deadlock if there is a potential for it. Check out what other threads are doing (in gdb, "thread all apply bt") and most likely you will see that another thread is also stuck while trying to grab a lock.

Amanda-M-UK commented 4 years ago

Thanks kcc

thread apply all bt doesn't give any additional information.

kcc commented 4 years ago

Sure, it's 'apply all', sorry. If it doesn't show anything, there are no more threads, and a deadlock is unlikely. Another hypothesis: uninitialized mutex? Under asan/tsan the memory is initialized differently than under normal glibc and it may cause a (buggy) code to behave differently. Although it's in libc, which is presumably well tested.

If you can extract a minimal test case (no wx*, no gtk, just pure libc cal to bindtextdomain, I can take a look)

jcelerier commented 2 years ago

I'm encountering the same issue in an entirely unrelated case (software not using wx at all), when trying to run PipeWire or Pulseaudio apps with asan / ubsan: so it seems that the issue is deeper. Also in this case no thread has been created yet so it can't be a "normal" deadlock, but maybe bindtextdomain does not initialize memory properly or something like this...

1   ??                                                                                                                                                                                                                                                                                                                                                                                                                                                                                0x7fffe6a89096 
2   pthread_rwlock_wrlock                                                                                                                                                                                                                                                                                                                                                                                                                                                             0x7fffe6a9298f 
3   ??                                                                                                                                                                                                                                                                                                                                                                                                                                                                                0x7fffe6a37916 
4   bindtextdomain                                                                                                                                                                                                                                                                                                                                                                                                                                                                    0x7fffe6a37dac 
5   pa_init_i18n                                                                                                                                                                                                                                                                                                                                                                                                                                                                      0x7fffd99a5bef 
6   pa_threaded_mainloop_new                                                                                                                                                                                                                                                                                                                                                                                                                                                          0x7fffeaf1477b 
7   pulse_new                                                                                                                                                                                                                                                                                                                                                                                                                                                                         0x7fffbb64ea38 
8   _snd_pcm_pulse_open                                                                                                                                                                                                                                                                                                                                                                                                                                                               0x7fffbb64efb7 
9   ??                                                                                                                                                                                                                                                                                                                                                                                                                                                                                0x7fffeb558673 
10  snd_pcm_open                                                                                                                                                                                                                                                                                                                                                                                                                                                                      0x7fffeb55aed6 
11  ??                                                                                                                                                                                                                                                                                                                                                                                                                                                                                0x7ffff78aaabc 
12  ??                                                                                                                                                                                                                                                                                                                                                                                                                                                                                0x7ffff78ab768 
13  PaAlsa_Initialize                                                                                                                                                                                                                                                                                                                                                                                                                                                                 0x7ffff78aed76 
14  Pa_Initialize                                                                                                                                                                                                                                                                                                                                                                                                                                                                     0x7ffff78a0d18 
rouse2617 commented 10 months ago

It look like this issue. https://sourceware.org/bugzilla/show_bug.cgi?id=27653