search

ivmai / bdwgc

The Boehm-Demers-Weiser conservative C/C++ Garbage Collector (bdwgc, also known as bdw-gc, boehm-gc, libgc)
https://www.hboehm.info/gc/
Other
2.98k stars 407 forks source link

gctest hangs rarely on Linux if compiled with TSan #236

Open ivmai opened 6 years ago

ivmai commented 6 years ago

Build link: https://travis-ci.org/ivmai/bdwgc/jobs/425865116

Source master (or 8.0.0) Host: Linux/x64

How to build and run: ./configure --disable-parallel-mark && make check CFLAGS_EXTRA="-fsanitize=thread -D NO_CANCEL_SAFE -D NO_INCREMENTAL -D USE_SPIN_LOCK -fno-omit-frame-pointer -D NTHREADS=15"

ivmai commented 5 years ago

Reproduced on latest master (a8c5ee41a) Build link: https://travis-ci.org/ivmai/bdwgc/jobs/523219610

ivmai commented 3 years ago

Reproduced on latest master (dd1d0bc) Build link: https://travis-ci.com/github/ivmai/bdwgc/jobs/529655035

ivmai commented 2 years ago

Source: master (https://github.com/ivmai/bdwgc/commit/7aa23f4e849520c229ea3c26d84afd39ebe83711)

Build: https://app.travis-ci.com/github/ivmai/bdwgc/jobs/564340350 gctest hang.

ivmai commented 2 years ago

Build: https://app.travis-ci.com/github/ivmai/bdwgc/jobs/565405309 Source: master (0eb8d2f9)

ivmai commented 2 years ago

Build: https://app.travis-ci.com/github/ivmai/bdwgc/jobs/565562062 Source: master (e34da92557)

ivmai commented 2 years ago

gctest still hangs sometimes (even after commit ba0616b), e.g. https://app.travis-ci.com/github/ivmai/bdwgc/jobs/565659019

ivmai commented 2 years ago

Reproduced even w/o GC_ENABLE_SUSPEND_THREAD - should be fixed by 09dd6a6

ivmai commented 2 years ago

Related issue #181

ivmai commented 2 years ago

Latest build: https://app.travis-ci.com/github/ivmai/bdwgc/jobs/566687404 Source: master (09dd6a6)

ivmai commented 2 years ago

The issue is still reproduced rarely, e.g.:

(gdb) t 18
#0  0x00000000004a0067 in __sanitizer::internal_sched_yield() ()
#1  0x000000000047ec45 in __tsan::Mutex::Lock() ()
#2  0x00000000004957a7 in __tsan::MetaMap::GetAndLock(__tsan::ThreadState*, unsigned long, unsigned long, bool, bool) ()
#3  0x000000000048dad5 in __tsan::MutexUnlock(__tsan::ThreadState*, unsigned long, unsigned long, unsigned int) ()
#4  0x0000000000443ca0 in pthread_mutex_unlock ()
#5  0x00000000004ea041 in GC_suspend_thread (thread=<optimized out>) at pthread_stop_world.c:650
#6  0x00000000004b82be in fork_a_thread () at tests/gctest.c:657

(gdb) t 23
#0  0x00007f7f5a5524ba in __GI___sigsuspend (set=0xfed438 <suspend_handler_mask>)
    at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
#1  0x000000000042c485 in sigsuspend ()
#2  0x00000000004e9a8a in GC_suspend_handler_inner (dummy=<optimized out>, context=<optimized out>)
    at pthread_stop_world.c:396
#3  0x00000000004e98ae in GC_suspend_sigaction (sig=<optimized out>, info=<optimized out>, context=<optimized out>)
    at pthread_stop_world.c:276
#4  0x000000000042c96c in __tsan::CallUserSignalHandler(__tsan::ThreadState*, bool, bool, bool, int, __sanitizer::__sanitizer_siginfo*, void*) ()
#5  0x000000000046459c in rtl_sigaction(int, __sanitizer::__sanitizer_siginfo*, void*) ()
#6  <signal handler called>
#7  0x0000000000499217 in __sanitizer::DD::MutexBeforeLock(__sanitizer::DDCallback*, __sanitizer::DDMutex*, bool) ()
#8  0x000000000048d210 in __tsan::MutexPreLock(__tsan::ThreadState*, unsigned long, unsigned long, unsigned int) ()
#9  0x0000000000443b10 in pthread_mutex_lock ()
#10 0x00000000004e8fae in GC_generic_lock (lock=0xfec6d0 <GC_allocate_ml>) at pthread_support.c:2354
#11 0x00000000004e4b03 in GC_lock () at pthread_support.c:2454

Unclear how to workaround this.

ivmai commented 2 years ago

Latest build (hang): https://app.travis-ci.com/github/ivmai/bdwgc/jobs/567979958 Source: master (fd4e579)

ivmai commented 2 years ago

Latest build: https://app.travis-ci.com/github/ivmai/bdwgc/jobs/573526977 Source: b75f1aa (master)

ivmai commented 2 years ago

Build: https://app.travis-ci.com/github/ivmai/bdwgc/jobs/576835719 Source: release-8_2 (4919305150)

ivmai commented 1 year ago

Build: https://app.travis-ci.com/github/ivmai/bdwgc/jobs/588637564 Source: release-8_2 (2b342c4)

ivmai commented 1 year ago

Cannot reproduce locally on this config (as of commit 6dfd81e on master): ./configure --enable-werror --enable-gc-assertions --disable-shared make -j check CFLAGS_EXTRA="-O0 -g -fsanitize=thread -fno-omit-frame-pointer" GC_PRINT_STATS=1 gdb ./gctest

ivmai commented 1 year ago

Reproduced locally. Source: master (7334cc94a04) Host: Linux 5.15.0-69-generic / x86_64 Compiler: gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0 How to build: ./configure --enable-static && make -j check CFLAGS_EXTRA="-O3 -fsanitize=thread -fno-omit-frame-pointer" Connect to gctest process: LD_LIBRARY_PATH=.libs gdb .libs/gctest -pid \<pid>

See stack trace of thread 18: __tsan::MutexUnlock

(gdb) t 1
[Switching to thread 1 (Thread 0x7f732b238340 (LWP 2947513))]
#0  futex_wait (private=0, expected=2, futex_word=0x7f732b5f7400 <GC_allocate_ml>) at ../sysdeps/nptl/futex-internal.h:146
146     ../sysdeps/nptl/futex-internal.h: No such file or directory.
(gdb) bt
#0  futex_wait (private=0, expected=2, futex_word=0x7f732b5f7400 <GC_allocate_ml>) at ../sysdeps/nptl/futex-internal.h:146
#1  __GI___lll_lock_wait (futex=futex@entry=0x7f732b5f7400 <GC_allocate_ml>, private=0) at ./nptl/lowlevellock.c:49
#2  0x00007f732b3bb082 in lll_mutex_lock_optimized (mutex=0x7f732b5f7400 <GC_allocate_ml>) at ./nptl/pthread_mutex_lock.c:48
#3  ___pthread_mutex_lock (mutex=0x7f732b5f7400 <GC_allocate_ml>) at ./nptl/pthread_mutex_lock.c:93
#4  0x00007f732b64c994 in __interceptor_pthread_mutex_lock (m=m@entry=0x7f732b5f7400 <GC_allocate_ml>) at ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:4242
#5  0x00007f732b5a0ae3 in GC_generic_lock (lock=0x7f732b5f7400 <GC_allocate_ml>) at pthread_support.c:2608
#6  0x00007f732b5a1ac4 in GC_lock () at pthread_support.c:2718
#7  0x00007f732b58bb45 in GC_malloc_kind_global (lb=lb@entry=937, k=k@entry=1) at malloc.c:319
#8  0x00007f732b5a441b in GC_malloc_kind (bytes=bytes@entry=937, kind=kind@entry=1) at thread_local_alloc.c:187
#9  0x00007f732b58bbf7 in GC_malloc (lb=lb@entry=937) at malloc.c:363
#10 0x00005632ff0b8c5a in cons (x=0x7f732490ae40, y=0x7f731ffc9000) at tests/gctest.c:320
#11 0x00005632ff0b906e in reverse1 (y=0x7f731ffc9000, x=0x7f7327be3c00) at tests/gctest.c:449
#12 reverse1 (x=<optimized out>, y=<optimized out>) at tests/gctest.c:449
#13 0x00005632ff0b90d6 in reverse1 (y=<optimized out>, x=0x7f7327beb000) at tests/gctest.c:449
#14 reverse1 (y=<optimized out>, x=<optimized out>) at tests/gctest.c:449
#15 reverse1 (y=<optimized out>, x=<optimized out>) at tests/gctest.c:449
#16 reverse1 (x=<optimized out>, y=<optimized out>) at tests/gctest.c:449
#17 0x00005632ff0bb1de in reverse1 (y=<optimized out>, x=0x7f73249ac800) at tests/gctest.c:449
#18 reverse1 (y=<optimized out>, x=<optimized out>) at tests/gctest.c:449
#19 reverse1 (y=<optimized out>, x=<optimized out>) at tests/gctest.c:449
#20 reverse1 (y=<optimized out>, x=<optimized out>) at tests/gctest.c:449
#21 reverse1 (y=<optimized out>, x=<optimized out>) at tests/gctest.c:449
#22 reverse1 (y=<optimized out>, x=<optimized out>) at tests/gctest.c:449
#23 reverse1 (y=0x0, x=<optimized out>) at tests/gctest.c:449
#24 reverse (x=<optimized out>) at tests/gctest.c:458
#25 reverse_test_inner (data=data@entry=0x1) at tests/gctest.c:944
#26 0x00007f732b5a2818 in GC_call_with_gc_active (fn=fn@entry=0x5632ff0ba780 <reverse_test_inner>, client_data=client_data@entry=0x1) at pthread_support.c:2034
#27 0x00005632ff0bb491 in reverse_test_inner (data=0x0) at tests/gctest.c:847
#28 0x00007f732b5a230b in GC_do_blocking_inner (data=0x7ffdcafe28a0 "\200\247\v\377\062V", context=<optimized out>) at pthread_support.c:1854
#29 0x00007f732b58a79a in GC_with_callee_saves_pushed (fn=0x7f732b5a2230 <GC_do_blocking_inner>, arg=<optimized out>, arg@entry=0x7ffdcafe28a0 "\200\247\v\377\062V") at mach_dep.c:421
#30 0x00007f732b59544e in GC_do_blocking (fn=fn@entry=0x5632ff0ba780 <reverse_test_inner>, client_data=client_data@entry=0x0) at misc.c:2314
#31 0x00005632ff0bd418 in reverse_test () at tests/gctest.c:979
#32 run_one_test () at tests/gctest.c:1845
#33 0x00005632ff0b815e in main () at tests/gctest.c:2643

(gdb) t 18
[Switching to thread 18 (Thread 0x7f7320ff8640 (LWP 2947796))]
#0  __sanitizer::internal_sched_yield () at ../../../../src/libsanitizer/sanitizer_common/sanitizer_syscall_linux_x86_64.inc:19
19      ../../../../src/libsanitizer/sanitizer_common/sanitizer_syscall_linux_x86_64.inc: No such file or directory.
(gdb) bt
#0  __sanitizer::internal_sched_yield () at ../../../../src/libsanitizer/sanitizer_common/sanitizer_syscall_linux_x86_64.inc:19
#1  0x00007f732b686bd5 in __tsan::Backoff::Do (this=<synthetic pointer>) at ../../../../src/libsanitizer/tsan/tsan_mutex.cpp:194
#2  __tsan::Mutex::Lock (this=0x7f732a530008) at ../../../../src/libsanitizer/tsan/tsan_mutex.cpp:234
#3  0x00007f732b6a15b5 in __tsan::MetaMap::GetAndLock (this=0x7f732b728408 <__tsan::ctx_placeholder+8>, thr=thr@entry=0x7f7320fb8ac0, pc=pc@entry=140132625895417,
    addr=addr@entry=140132625642496, write_lock=write_lock@entry=true, create=create@entry=true) at ../../../../src/libsanitizer/tsan/tsan_sync.cpp:226
#4  0x00007f732b6a19c3 in __tsan::MetaMap::GetOrCreateAndLock (this=<optimized out>, thr=thr@entry=0x7f7320fb8ac0, pc=pc@entry=140132625895417, addr=addr@entry=140132625642496,
    write_lock=write_lock@entry=true) at ../../../../src/libsanitizer/tsan/tsan_sync.cpp:198
#5  0x00007f732b699963 in __tsan::MutexUnlock (thr=thr@entry=0x7f7320fb8ac0, pc=pc@entry=140132625895417, addr=addr@entry=140132625642496, flagz=flagz@entry=0)
    at ../../../../src/libsanitizer/tsan/tsan_rtl_mutex.cpp:217
#6  0x00007f732b63507b in __interceptor_pthread_mutex_unlock (m=0x7f732b5f7400 <GC_allocate_ml>) at ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:4255
#7  0x00007f732b5a53fc in GC_suspend_thread (thread=<optimized out>) at pthread_stop_world.c:646
#8  0x00005632ff0ba48c in fork_a_thread () at tests/gctest.c:734
#9  0x00005632ff0bb211 in reverse_test_inner (data=data@entry=0x1) at tests/gctest.c:937
#10 0x00007f732b5a2818 in GC_call_with_gc_active (fn=fn@entry=0x5632ff0ba780 <reverse_test_inner>, client_data=client_data@entry=0x1) at pthread_support.c:2034
#11 0x00005632ff0bb491 in reverse_test_inner (data=0x0) at tests/gctest.c:847
#12 0x00007f732b5a230b in GC_do_blocking_inner (data=0x7f7320fb7f30 "\200\247\v\377\062V", context=<optimized out>) at pthread_support.c:1854
#13 0x00007f732b58a79a in GC_with_callee_saves_pushed (fn=0x7f732b5a2230 <GC_do_blocking_inner>, arg=<optimized out>, arg@entry=0x7f7320fb7f30 "\200\247\v\377\062V") at mach_dep.c:421
#14 0x00007f732b59544e in GC_do_blocking (fn=fn@entry=0x5632ff0ba780 <reverse_test_inner>, client_data=client_data@entry=0x0) at misc.c:2314
#15 0x00005632ff0bd418 in reverse_test () at tests/gctest.c:979
#16 run_one_test () at tests/gctest.c:1845
#17 0x00005632ff0bd90c in thr_run_one_test (arg=<optimized out>) at tests/gctest.c:2535
#18 0x00007f732b5a05d8 in GC_pthread_start_inner (sb=sb@entry=0x7f7320fb81b0, arg=arg@entry=0x7ffdcafe29c0) at pthread_start.c:55
#19 0x00007f732b5953a7 in GC_call_with_stack_base (fn=0x7f732b5a0530 <GC_pthread_start_inner>, arg=arg@entry=0x7ffdcafe29c0) at misc.c:2180
#20 0x00007f732b5a09f9 in GC_pthread_start (arg=0x7ffdcafe29c0) at pthread_support.c:2425
#21 0x00007f732b627600 in __tsan_thread_start_func (arg=0x7ffdcafe28b0) at ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:959
#22 0x00007f732b3b7b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#23 0x00007f732b449a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

(gdb) t 23
[Switching to thread 23 (Thread 0x7f731cf0e640 (LWP 2949801))]
#0  0x00007f732b36580a in __GI___sigsuspend (set=0x7f732b5f8140 <suspend_handler_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
26      ../sysdeps/unix/sysv/linux/sigsuspend.c: No such file or directory.
(gdb) bt
#0  0x00007f732b36580a in __GI___sigsuspend (set=0x7f732b5f8140 <suspend_handler_mask>) at ../sysdeps/unix/sysv/linux/sigsuspend.c:26
#1  0x00007f732b62bfb3 in __interceptor_sigsuspend (mask=mask@entry=0x7f732b5f8140 <suspend_handler_mask>) at ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1872
#2  0x00007f732b5a4a48 in GC_suspend_handler_inner (dummy=0x0, context=<optimized out>) at pthread_stop_world.c:398
#3  GC_suspend_sigaction (sig=<optimized out>, info=<optimized out>, context=<optimized out>) at pthread_stop_world.c:276
#4  0x00007f732b6267f7 in __tsan::CallUserSignalHandler (thr=thr@entry=0x7f731ceceac0, sync=<optimized out>, acquire=acquire@entry=true, sigact=sigact@entry=true, sig=sig@entry=31,
    info=info@entry=0x7f731cecd170, uctx=0x7f731cecd040) at ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1922
#5  0x00007f732b628df5 in rtl_generic_sighandler (ctx=0x7f731cecd040, info=0x7f731cecd170, sig=31, sigact=true) at ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:2016
#6  rtl_sigaction (sig=<optimized out>, info=0x7f731cecd170, ctx=0x7f731cecd040) at ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:2041
#7  <signal handler called>
#8  0x00007f732b698473 in __tsan::MutexPreLock (thr=thr@entry=0x7f731ceceac0, pc=pc@entry=140132625991945, addr=addr@entry=140132625642496, flagz=flagz@entry=0)
    at ../../../../src/libsanitizer/tsan/tsan_rtl_mutex.cpp:149
#9  0x00007f732b64c98b in __interceptor_pthread_mutex_lock (m=m@entry=0x7f732b5f7400 <GC_allocate_ml>) at ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:4241
#10 0x00007f732b5a0ae3 in GC_generic_lock (lock=0x7f732b5f7400 <GC_allocate_ml>) at pthread_support.c:2608
#11 0x00007f732b5a1ac4 in GC_lock () at pthread_support.c:2718
#12 0x00007f732b58bb45 in GC_malloc_kind_global (lb=lb@entry=16, k=k@entry=0) at malloc.c:319
#13 0x00007f732b5a441b in GC_malloc_kind (bytes=bytes@entry=16, kind=kind@entry=0) at thread_local_alloc.c:187
#14 0x00007f732b58bba4 in GC_malloc_atomic (lb=lb@entry=16) at malloc.c:357
#15 0x00005632ff0b9348 in small_cons_leaf (x=10) at tests/gctest.c:401
#16 ints (low=<optimized out>, up=10) at tests/gctest.c:507
#17 0x00005632ff0b933b in ints (low=<optimized out>, up=10) at tests/gctest.c:507
#18 0x00005632ff0b933b in ints (low=<optimized out>, up=10) at tests/gctest.c:507
#19 0x00005632ff0b933b in ints (low=<optimized out>, up=10) at tests/gctest.c:507
#20 0x00005632ff0b933b in ints (low=<optimized out>, up=10) at tests/gctest.c:507
#21 0x00005632ff0b933b in ints (low=<optimized out>, up=10) at tests/gctest.c:507
#22 0x00005632ff0b933b in ints (low=<optimized out>, up=10) at tests/gctest.c:507
#23 0x00005632ff0b933b in ints (low=<optimized out>, up=10) at tests/gctest.c:507
#24 0x00005632ff0b9f6a in ints (up=10, low=2) at tests/gctest.c:507
#25 ints (low=1, up=10) at tests/gctest.c:507
#26 tiny_reverse_test_inner () at tests/gctest.c:645
#27 0x00005632ff0ba323 in tiny_reverse_test (p_resumed=0x0) at tests/gctest.c:671
#28 0x00007f732b5a05d8 in GC_pthread_start_inner (sb=sb@entry=0x7f731cece1b0, arg=arg@entry=0x7f7320fb7770) at pthread_start.c:55
#29 0x00007f732b5953a7 in GC_call_with_stack_base (fn=0x7f732b5a0530 <GC_pthread_start_inner>, arg=arg@entry=0x7f7320fb7770) at misc.c:2180
#30 0x00007f732b5a09f9 in GC_pthread_start (arg=0x7f7320fb7770) at pthread_support.c:2425
#31 0x00007f732b627600 in __tsan_thread_start_func (arg=0x7f7320fb7660) at ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:959
#32 0x00007f732b3b7b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#33 0x00007f732b449a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
ivmai commented 1 year ago

Source: release-8_2 (bef858c) Build 1: https://app.travis-ci.com/github/ivmai/bdwgc/jobs/601194806 Build 2: https://app.travis-ci.com/github/ivmai/bdwgc/jobs/601194807 Host: Ubuntu 16.04.7 LTS / x64 Compiler: clang version 7.0.0 (tags/RELEASE_700/final) Config 1: CFLAGS_EXTRA="-fsanitize=thread -D NO_CANCEL_SAFE -D NO_INCREMENTAL -D USE_SPIN_LOCK -fno-omit-frame-pointer -D TEST_FORK_WITHOUT_ATFORK" CONF_OPTIONS="--enable-gc-assertions --enable-gc-debug --enable-handle-fork=manual --enable-large-config --with-libatomic-ops=no" Config 2: CFLAGS_EXTRA="-fsanitize=thread -D NO_CANCEL_SAFE -D NO_INCREMENTAL -D USE_SPIN_LOCK -fno-omit-frame-pointer -D NTHREADS=15" CONF_OPTIONS="--disable-parallel-mark"

ivmai commented 1 year ago

Build: https://app.travis-ci.com/github/ivmai/bdwgc/jobs/609522449 Source: release-8_2 (399005e) Host: Ubuntu 16.04.7 LTS / amd64 Compiler: clang version 7.0.0 (tags/RELEASE_700/final) Config: CFLAGS_EXTRA="-fsanitize=thread -D NO_CANCEL_SAFE -D NO_INCREMENTAL -D USE_SPIN_LOCK -fno-omit-frame-pointer -D NTHREADS=15" CONF_OPTIONS="--disable-parallel-mark"