Failures with ThreadSanitizer

[timwoj]

I've been adding support for ThreadSanitizer to the zeek project (which uses libkqueue) and it was reporting issues out of libkqueue. It appears the same issues happen with the built-in tests. ctest+tsan output is below. I haven't looked at whether any of these are easy fixes.

tim@nuc:~/kqueue/libkqueue/build$ ctest -V
UpdateCTestConfiguration  from :/home/tim/kqueue/libkqueue/build/DartConfiguration.tcl
UpdateCTestConfiguration  from :/home/tim/kqueue/libkqueue/build/DartConfiguration.tcl
Test project /home/tim/kqueue/libkqueue/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
    Start 1: libkqueue-test

1: Test command: /home/tim/kqueue/libkqueue/build/test/libkqueue-test
1: Test timeout computed to be: 10000000
1: ==================
1: WARNING: ThreadSanitizer: data race (pid=16276)
1:   Atomic write of size 8 at 0x7fee4c600040 by thread T3 (mutexes: write M4):
1:     #0 __tsan_atomic64_compare_exchange_strong ../../../../src/libsanitizer/tsan/tsan_interface_atomic.cpp:786 (libtsan.so.0+0x83a41)
1:     #1 map_delete ../src/common/map.c:125 (libkqueue.so.2.4.0+0xb7f7)
1:     #2 kqueue_free ../src/common/kqueue.c:143 (libkqueue.so.2.4.0+0xa90f)
1:     #3 monitoring_thread_kq_cleanup ../src/linux/platform.c:126 (libkqueue.so.2.4.0+0xc0e3)
1:     #4 monitoring_thread_loop ../src/linux/platform.c:245 (libkqueue.so.2.4.0+0xcae9)
1: 
1:   Previous read of size 8 at 0x7fee4c600040 by main thread:
1:     #0 map_lookup ../src/common/map.c:110 (libkqueue.so.2.4.0+0xb6fc)
1:     #1 kqueue_lookup ../src/common/kqueue.c:170 (libkqueue.so.2.4.0+0xaab0)
1:     #2 kevent ../src/common/kevent.c:266 (libkqueue.so.2.4.0+0x6419)
1:     #3 kevent_add ../test/kevent.c:217 (libkqueue-test+0x4e85)
1:     #4 test_cleanup ../test/main.c:228 (libkqueue-test+0x5f08)
1:     #5 test_harness ../test/main.c:292 (libkqueue-test+0x64ab)
1:     #6 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Mutex M4 (0x7fee4f11b320) created at:
1:     #0 pthread_mutex_lock ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:4165 (libtsan.so.0+0x526fc)
1:     #1 kqueue ../src/common/kqueue.c:189 (libkqueue.so.2.4.0+0xaaec)
1:     #2 test_kqueue ../test/main.c:150 (libkqueue-test+0x5817)
1:     #3 test_harness ../test/main.c:288 (libkqueue-test+0x6409)
1:     #4 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Thread T3 (tid=16280, running) created by main thread at:
1:     #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:962 (libtsan.so.0+0x5ea79)
1:     #1 linux_kqueue_start_thread ../src/linux/platform.c:267 (libkqueue.so.2.4.0+0xcc71)
1:     #2 pthread_once ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1442 (libtsan.so.0+0x41964)
1:     #3 linux_kqueue_init ../src/linux/platform.c:369 (libkqueue.so.2.4.0+0xd73a)
1:     #4 kqueue ../src/common/kqueue.c:200 (libkqueue.so.2.4.0+0xab99)
1:     #5 test_cleanup ../test/main.c:225 (libkqueue-test+0x5e63)
1:     #6 test_harness ../test/main.c:292 (libkqueue-test+0x64ab)
1:     #7 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1: SUMMARY: ThreadSanitizer: data race ../src/common/map.c:125 in map_delete
1: ==================
1: ==================
1: WARNING: ThreadSanitizer: data race (pid=16276)
1:   Write of size 1 at 0x7b940001e848 by thread T3 (mutexes: write M4):
1:     #0 pthread_mutex_destroy ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1237 (libtsan.so.0+0x3725c)
1:     #1 kqueue_free ../src/common/kqueue.c:147 (libkqueue.so.2.4.0+0xa941)
1:     #2 monitoring_thread_kq_cleanup ../src/linux/platform.c:126 (libkqueue.so.2.4.0+0xc0e3)
1:     #3 monitoring_thread_loop ../src/linux/platform.c:245 (libkqueue.so.2.4.0+0xcae9)
1: 
1:   Previous atomic read of size 1 at 0x7b940001e848 by main thread (mutexes: write M172680036309133384):
1:     #0 pthread_mutex_unlock ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:4179 (libtsan.so.0+0x3aadc)
1:     #1 kevent ../src/common/kevent.c:285 (libkqueue.so.2.4.0+0x66fe)
1:     #2 kevent_add ../test/kevent.c:217 (libkqueue-test+0x4e85)
1:     #3 test_cleanup ../test/main.c:228 (libkqueue-test+0x5f08)
1:     #4 test_harness ../test/main.c:292 (libkqueue-test+0x64ab)
1:     #5 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Location is heap block of size 8376 at 0x7b940001e000 allocated by main thread:
1:     #0 calloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:668 (libtsan.so.0+0x305aa)
1:     #1 kqueue ../src/common/kqueue.c:194 (libkqueue.so.2.4.0+0xab1a)
1:     #2 test_cleanup ../test/main.c:225 (libkqueue-test+0x5e63)
1:     #3 test_harness ../test/main.c:292 (libkqueue-test+0x64ab)
1:     #4 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Mutex M4 (0x7fee4f11b320) created at:
1:     #0 pthread_mutex_lock ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:4165 (libtsan.so.0+0x526fc)
1:     #1 kqueue ../src/common/kqueue.c:189 (libkqueue.so.2.4.0+0xaaec)
1:     #2 test_kqueue ../test/main.c:150 (libkqueue-test+0x5817)
1:     #3 test_harness ../test/main.c:288 (libkqueue-test+0x6409)
1:     #4 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Mutex M172680036309133384 is already destroyed.
1: 
1:   Thread T3 (tid=16280, running) created by main thread at:
1:     #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:962 (libtsan.so.0+0x5ea79)
1:     #1 linux_kqueue_start_thread ../src/linux/platform.c:267 (libkqueue.so.2.4.0+0xcc71)
1:     #2 pthread_once ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1442 (libtsan.so.0+0x41964)
1:     #3 linux_kqueue_init ../src/linux/platform.c:369 (libkqueue.so.2.4.0+0xd73a)
1:     #4 kqueue ../src/common/kqueue.c:200 (libkqueue.so.2.4.0+0xab99)
1:     #5 test_cleanup ../test/main.c:225 (libkqueue-test+0x5e63)
1:     #6 test_harness ../test/main.c:292 (libkqueue-test+0x64ab)
1:     #7 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1: SUMMARY: ThreadSanitizer: data race ../src/common/kqueue.c:147 in kqueue_free
1: ==================
1: ==================
1: WARNING: ThreadSanitizer: data race (pid=16276)
1:   Atomic write of size 8 at 0x7fee4c600060 by thread T1 (mutexes: write M4):
1:     #0 __tsan_atomic64_compare_exchange_strong ../../../../src/libsanitizer/tsan/tsan_interface_atomic.cpp:786 (libtsan.so.0+0x83a41)
1:     #1 map_delete ../src/common/map.c:125 (libkqueue.so.2.4.0+0xb7f7)
1:     #2 kqueue_free ../src/common/kqueue.c:143 (libkqueue.so.2.4.0+0xa90f)
1:     #3 monitoring_thread_kq_cleanup ../src/linux/platform.c:126 (libkqueue.so.2.4.0+0xc0e3)
1:     #4 monitoring_thread_loop ../src/linux/platform.c:245 (libkqueue.so.2.4.0+0xcae9)
1: 
1:   Previous read of size 8 at 0x7fee4c600060 by main thread:
1:     #0 map_lookup ../src/common/map.c:110 (libkqueue.so.2.4.0+0xb6fc)
1:     #1 kqueue_lookup ../src/common/kqueue.c:170 (libkqueue.so.2.4.0+0xaab0)
1:     #2 kevent ../src/common/kevent.c:266 (libkqueue.so.2.4.0+0x6419)
1:     #3 test_transition_from_write_to_read ../test/read.c:451 (libkqueue-test+0x999d)
1:     #4 test_evfilt_read ../test/read.c:482 (libkqueue-test+0x9ee2)
1:     #5 run_iteration ../test/main.c:269 (libkqueue-test+0x62f2)
1:     #6 test_harness ../test/main.c:314 (libkqueue-test+0x6643)
1:     #7 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Mutex M4 (0x7fee4f11b320) created at:
1:     #0 pthread_mutex_lock ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:4165 (libtsan.so.0+0x526fc)
1:     #1 kqueue ../src/common/kqueue.c:189 (libkqueue.so.2.4.0+0xaaec)
1:     #2 test_kqueue ../test/main.c:150 (libkqueue-test+0x5817)
1:     #3 test_harness ../test/main.c:288 (libkqueue-test+0x6409)
1:     #4 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Thread T1 (tid=16346, running) created by main thread at:
1:     #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:962 (libtsan.so.0+0x5ea79)
1:     #1 linux_kqueue_start_thread ../src/linux/platform.c:267 (libkqueue.so.2.4.0+0xcc71)
1:     #2 pthread_once ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1442 (libtsan.so.0+0x41964)
1:     #3 linux_kqueue_init ../src/linux/platform.c:369 (libkqueue.so.2.4.0+0xd73a)
1:     #4 kqueue ../src/common/kqueue.c:200 (libkqueue.so.2.4.0+0xab99)
1:     #5 test_harness ../test/main.c:302 (libkqueue-test+0x6519)
1:     #6 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1: SUMMARY: ThreadSanitizer: data race ../src/common/map.c:125 in map_delete
1: ==================
1: ==================
1: WARNING: ThreadSanitizer: data race (pid=16276)
1:   Atomic write of size 8 at 0x7fee4c600048 by thread T1 (mutexes: write M4):
1:     #0 __tsan_atomic64_compare_exchange_strong ../../../../src/libsanitizer/tsan/tsan_interface_atomic.cpp:786 (libtsan.so.0+0x83a41)
1:     #1 map_delete ../src/common/map.c:125 (libkqueue.so.2.4.0+0xb7f7)
1:     #2 kqueue_free ../src/common/kqueue.c:143 (libkqueue.so.2.4.0+0xa90f)
1:     #3 monitoring_thread_kq_cleanup ../src/linux/platform.c:126 (libkqueue.so.2.4.0+0xc0e3)
1:     #4 monitoring_thread_loop ../src/linux/platform.c:245 (libkqueue.so.2.4.0+0xcae9)
1: 
1:   Previous read of size 8 at 0x7fee4c600048 by main thread:
1:     #0 map_lookup ../src/common/map.c:110 (libkqueue.so.2.4.0+0xb6fc)
1:     #1 kqueue_lookup ../src/common/kqueue.c:170 (libkqueue.so.2.4.0+0xaab0)
1:     #2 kevent ../src/common/kevent.c:266 (libkqueue.so.2.4.0+0x6419)
1:     #3 _test_no_kevents ../test/kevent.c:30 (libkqueue-test+0x3acf)
1:     #4 test_kevent_user_dispatch ../test/user.c:178 (libkqueue-test+0xc009)
1:     #5 test_evfilt_user ../test/user.c:192 (libkqueue-test+0xc270)
1:     #6 run_iteration ../test/main.c:269 (libkqueue-test+0x62f2)
1:     #7 test_harness ../test/main.c:314 (libkqueue-test+0x6643)
1:     #8 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Mutex M4 (0x7fee4f11b320) created at:
1:     #0 pthread_mutex_lock ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:4165 (libtsan.so.0+0x526fc)
1:     #1 kqueue ../src/common/kqueue.c:189 (libkqueue.so.2.4.0+0xaaec)
1:     #2 test_kqueue ../test/main.c:150 (libkqueue-test+0x5817)
1:     #3 test_harness ../test/main.c:288 (libkqueue-test+0x6409)
1:     #4 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Thread T1 (tid=16346, running) created by main thread at:
1:     #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:962 (libtsan.so.0+0x5ea79)
1:     #1 linux_kqueue_start_thread ../src/linux/platform.c:267 (libkqueue.so.2.4.0+0xcc71)
1:     #2 pthread_once ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1442 (libtsan.so.0+0x41964)
1:     #3 linux_kqueue_init ../src/linux/platform.c:369 (libkqueue.so.2.4.0+0xd73a)
1:     #4 kqueue ../src/common/kqueue.c:200 (libkqueue.so.2.4.0+0xab99)
1:     #5 test_harness ../test/main.c:302 (libkqueue-test+0x6519)
1:     #6 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1: SUMMARY: ThreadSanitizer: data race ../src/common/map.c:125 in map_delete
1: ==================
1: ==================
1: WARNING: ThreadSanitizer: data race (pid=16276)
1:   Write of size 8 at 0x7b94000196c8 by thread T1 (mutexes: write M4):
1:     #0 memset ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:795 (libtsan.so.0+0x3790f)
1:     #1 memset ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:793 (libtsan.so.0+0x3790f)
1:     #2 filter_unregister_all ../src/common/filter.c:125 (libkqueue.so.2.4.0+0x43db)
1:     #3 kqueue_free ../src/common/kqueue.c:145 (libkqueue.so.2.4.0+0xa91b)
1:     #4 monitoring_thread_kq_cleanup ../src/linux/platform.c:126 (libkqueue.so.2.4.0+0xc0e3)
1:     #5 monitoring_thread_loop ../src/linux/platform.c:245 (libkqueue.so.2.4.0+0xcae9)
1: 
1:   Previous read of size 8 at 0x7b94000196c8 by main thread (mutexes: write M1207):
1:     #0 knote_delete ../src/common/knote.c:137 (libkqueue.so.2.4.0+0x9fe1)
1:     #1 kevent_copyin_one ../src/common/kevent.c:203 (libkqueue.so.2.4.0+0x5f56)
1:     #2 kevent_copyin ../src/common/kevent.c:228 (libkqueue.so.2.4.0+0x6267)
1:     #3 kevent ../src/common/kevent.c:284 (libkqueue.so.2.4.0+0x66a9)
1:     #4 kevent_add ../test/kevent.c:217 (libkqueue-test+0x4e85)
1:     #5 test_kevent_user_dispatch ../test/user.c:177 (libkqueue-test+0xbfd6)
1:     #6 test_evfilt_user ../test/user.c:192 (libkqueue-test+0xc270)
1:     #7 run_iteration ../test/main.c:269 (libkqueue-test+0x62f2)
1:     #8 test_harness ../test/main.c:314 (libkqueue-test+0x6643)
1:     #9 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Location is heap block of size 8376 at 0x7b9400019000 allocated by main thread:
1:     #0 calloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:668 (libtsan.so.0+0x305aa)
1:     #1 kqueue ../src/common/kqueue.c:194 (libkqueue.so.2.4.0+0xab1a)
1:     #2 test_harness ../test/main.c:302 (libkqueue-test+0x6519)
1:     #3 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Mutex M4 (0x7fee4f11b320) created at:
1:     #0 pthread_mutex_lock ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:4165 (libtsan.so.0+0x526fc)
1:     #1 kqueue ../src/common/kqueue.c:189 (libkqueue.so.2.4.0+0xaaec)
1:     #2 test_kqueue ../test/main.c:150 (libkqueue-test+0x5817)
1:     #3 test_harness ../test/main.c:288 (libkqueue-test+0x6409)
1:     #4 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Mutex M1207 (0x7b9400019848) created at:
1:     #0 pthread_mutex_init ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1220 (libtsan.so.0+0x4a616)
1:     #1 kqueue ../src/common/kqueue.c:198 (libkqueue.so.2.4.0+0xab46)
1:     #2 test_harness ../test/main.c:302 (libkqueue-test+0x6519)
1:     #3 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Thread T1 (tid=16346, running) created by main thread at:
1:     #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:962 (libtsan.so.0+0x5ea79)
1:     #1 linux_kqueue_start_thread ../src/linux/platform.c:267 (libkqueue.so.2.4.0+0xcc71)
1:     #2 pthread_once ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1442 (libtsan.so.0+0x41964)
1:     #3 linux_kqueue_init ../src/linux/platform.c:369 (libkqueue.so.2.4.0+0xd73a)
1:     #4 kqueue ../src/common/kqueue.c:200 (libkqueue.so.2.4.0+0xab99)
1:     #5 test_harness ../test/main.c:302 (libkqueue-test+0x6519)
1:     #6 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1: SUMMARY: ThreadSanitizer: data race ../src/common/filter.c:125 in filter_unregister_all
1: ==================
1: ==================
1: WARNING: ThreadSanitizer: data race (pid=16276)
1:   Write of size 8 at 0x7b9400019738 by thread T1 (mutexes: write M4):
1:     #0 memset ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:795 (libtsan.so.0+0x3790f)
1:     #1 memset ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:793 (libtsan.so.0+0x3790f)
1:     #2 filter_unregister_all ../src/common/filter.c:125 (libkqueue.so.2.4.0+0x43db)
1:     #3 kqueue_free ../src/common/kqueue.c:145 (libkqueue.so.2.4.0+0xa91b)
1:     #4 monitoring_thread_kq_cleanup ../src/linux/platform.c:126 (libkqueue.so.2.4.0+0xc0e3)
1:     #5 monitoring_thread_loop ../src/linux/platform.c:245 (libkqueue.so.2.4.0+0xcae9)
1: 
1:   Previous read of size 8 at 0x7b9400019738 by main thread (mutexes: write M1207):
1:     #0 linux_evfilt_user_knote_delete ../src/linux/user.c:196 (libkqueue.so.2.4.0+0x192e4)
1:     #1 knote_delete ../src/common/knote.c:137 (libkqueue.so.2.4.0+0xa002)
1:     #2 kevent_copyin_one ../src/common/kevent.c:203 (libkqueue.so.2.4.0+0x5f56)
1:     #3 kevent_copyin ../src/common/kevent.c:228 (libkqueue.so.2.4.0+0x6267)
1:     #4 kevent ../src/common/kevent.c:284 (libkqueue.so.2.4.0+0x66a9)
1:     #5 kevent_add ../test/kevent.c:217 (libkqueue-test+0x4e85)
1:     #6 test_kevent_user_dispatch ../test/user.c:177 (libkqueue-test+0xbfd6)
1:     #7 test_evfilt_user ../test/user.c:192 (libkqueue-test+0xc270)
1:     #8 run_iteration ../test/main.c:269 (libkqueue-test+0x62f2)
1:     #9 test_harness ../test/main.c:314 (libkqueue-test+0x6643)
1:     #10 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Location is heap block of size 8376 at 0x7b9400019000 allocated by main thread:
1:     #0 calloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:668 (libtsan.so.0+0x305aa)
1:     #1 kqueue ../src/common/kqueue.c:194 (libkqueue.so.2.4.0+0xab1a)
1:     #2 test_harness ../test/main.c:302 (libkqueue-test+0x6519)
1:     #3 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Mutex M4 (0x7fee4f11b320) created at:
1:     #0 pthread_mutex_lock ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:4165 (libtsan.so.0+0x526fc)
1:     #1 kqueue ../src/common/kqueue.c:189 (libkqueue.so.2.4.0+0xaaec)
1:     #2 test_kqueue ../test/main.c:150 (libkqueue-test+0x5817)
1:     #3 test_harness ../test/main.c:288 (libkqueue-test+0x6409)
1:     #4 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Mutex M1207 (0x7b9400019848) created at:
1:     #0 pthread_mutex_init ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1220 (libtsan.so.0+0x4a616)
1:     #1 kqueue ../src/common/kqueue.c:198 (libkqueue.so.2.4.0+0xab46)
1:     #2 test_harness ../test/main.c:302 (libkqueue-test+0x6519)
1:     #3 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Thread T1 (tid=16346, running) created by main thread at:
1:     #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:962 (libtsan.so.0+0x5ea79)
1:     #1 linux_kqueue_start_thread ../src/linux/platform.c:267 (libkqueue.so.2.4.0+0xcc71)
1:     #2 pthread_once ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1442 (libtsan.so.0+0x41964)
1:     #3 linux_kqueue_init ../src/linux/platform.c:369 (libkqueue.so.2.4.0+0xd73a)
1:     #4 kqueue ../src/common/kqueue.c:200 (libkqueue.so.2.4.0+0xab99)
1:     #5 test_harness ../test/main.c:302 (libkqueue-test+0x6519)
1:     #6 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1: SUMMARY: ThreadSanitizer: data race ../src/common/filter.c:125 in filter_unregister_all
1: ==================
1: ==================
1: WARNING: ThreadSanitizer: data race (pid=16276)
1:   Write of size 8 at 0x7ba000000060 by thread T1 (mutexes: write M4):
1:     #0 close ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1675 (libtsan.so.0+0x3458a)
1:     #1 linux_kqueue_cleanup ../src/linux/platform.c:415 (libkqueue.so.2.4.0+0xdbfd)
1:     #2 linux_kqueue_free ../src/linux/platform.c:465 (libkqueue.so.2.4.0+0xe253)
1:     #3 kqueue_free ../src/common/kqueue.c:146 (libkqueue.so.2.4.0+0xa92f)
1:     #4 monitoring_thread_kq_cleanup ../src/linux/platform.c:126 (libkqueue.so.2.4.0+0xc0e3)
1:     #5 monitoring_thread_loop ../src/linux/platform.c:245 (libkqueue.so.2.4.0+0xcae9)
1: 
1:   Previous read of size 8 at 0x7ba000000060 by main thread:
1:     #0 epoll_wait ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1821 (libtsan.so.0+0x625db)
1:     #1 linux_kevent_wait ../src/linux/platform.c:532 (libkqueue.so.2.4.0+0xe864)
1:     #2 kevent ../src/common/kevent.c:303 (libkqueue.so.2.4.0+0x687b)
1:     #3 _test_no_kevents ../test/kevent.c:30 (libkqueue-test+0x3acf)
1:     #4 test_kevent_user_dispatch ../test/user.c:178 (libkqueue-test+0xc009)
1:     #5 test_evfilt_user ../test/user.c:192 (libkqueue-test+0xc270)
1:     #6 run_iteration ../test/main.c:269 (libkqueue-test+0x62f2)
1:     #7 test_harness ../test/main.c:314 (libkqueue-test+0x6643)
1:     #8 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Location is file descriptor 6 created by main thread at:
1:     #0 epoll_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1793 (libtsan.so.0+0x32d7a)
1:     #1 linux_kqueue_init ../src/linux/platform.c:282 (libkqueue.so.2.4.0+0xcdaa)
1:     #2 kqueue ../src/common/kqueue.c:200 (libkqueue.so.2.4.0+0xab99)
1:     #3 test_harness ../test/main.c:302 (libkqueue-test+0x6519)
1:     #4 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Mutex M4 (0x7fee4f11b320) created at:
1:     #0 pthread_mutex_lock ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:4165 (libtsan.so.0+0x526fc)
1:     #1 kqueue ../src/common/kqueue.c:189 (libkqueue.so.2.4.0+0xaaec)
1:     #2 test_kqueue ../test/main.c:150 (libkqueue-test+0x5817)
1:     #3 test_harness ../test/main.c:288 (libkqueue-test+0x6409)
1:     #4 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Thread T1 (tid=16346, running) created by main thread at:
1:     #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:962 (libtsan.so.0+0x5ea79)
1:     #1 linux_kqueue_start_thread ../src/linux/platform.c:267 (libkqueue.so.2.4.0+0xcc71)
1:     #2 pthread_once ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1442 (libtsan.so.0+0x41964)
1:     #3 linux_kqueue_init ../src/linux/platform.c:369 (libkqueue.so.2.4.0+0xd73a)
1:     #4 kqueue ../src/common/kqueue.c:200 (libkqueue.so.2.4.0+0xab99)
1:     #5 test_harness ../test/main.c:302 (libkqueue-test+0x6519)
1:     #6 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1: SUMMARY: ThreadSanitizer: data race ../src/linux/platform.c:415 in linux_kqueue_cleanup
1: ==================
1: ==================
1: WARNING: ThreadSanitizer: data race (pid=16276)
1:   Write of size 4 at 0x7b940001987c by thread T1 (mutexes: write M4):
1:     #0 linux_kqueue_cleanup ../src/linux/platform.c:417 (libkqueue.so.2.4.0+0xdcd8)
1:     #1 linux_kqueue_free ../src/linux/platform.c:465 (libkqueue.so.2.4.0+0xe253)
1:     #2 kqueue_free ../src/common/kqueue.c:146 (libkqueue.so.2.4.0+0xa92f)
1:     #3 monitoring_thread_kq_cleanup ../src/linux/platform.c:126 (libkqueue.so.2.4.0+0xc0e3)
1:     #4 monitoring_thread_loop ../src/linux/platform.c:245 (libkqueue.so.2.4.0+0xcae9)
1: 
1:   Previous read of size 4 at 0x7b940001987c by main thread:
1:     #0 linux_kevent_wait ../src/linux/platform.c:532 (libkqueue.so.2.4.0+0xe832)
1:     #1 kevent ../src/common/kevent.c:303 (libkqueue.so.2.4.0+0x687b)
1:     #2 _test_no_kevents ../test/kevent.c:30 (libkqueue-test+0x3acf)
1:     #3 test_kevent_user_dispatch ../test/user.c:178 (libkqueue-test+0xc009)
1:     #4 test_evfilt_user ../test/user.c:192 (libkqueue-test+0xc270)
1:     #5 run_iteration ../test/main.c:269 (libkqueue-test+0x62f2)
1:     #6 test_harness ../test/main.c:314 (libkqueue-test+0x6643)
1:     #7 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Location is heap block of size 8376 at 0x7b9400019000 allocated by main thread:
1:     #0 calloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:668 (libtsan.so.0+0x305aa)
1:     #1 kqueue ../src/common/kqueue.c:194 (libkqueue.so.2.4.0+0xab1a)
1:     #2 test_harness ../test/main.c:302 (libkqueue-test+0x6519)
1:     #3 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Mutex M4 (0x7fee4f11b320) created at:
1:     #0 pthread_mutex_lock ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:4165 (libtsan.so.0+0x526fc)
1:     #1 kqueue ../src/common/kqueue.c:189 (libkqueue.so.2.4.0+0xaaec)
1:     #2 test_kqueue ../test/main.c:150 (libkqueue-test+0x5817)
1:     #3 test_harness ../test/main.c:288 (libkqueue-test+0x6409)
1:     #4 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1:   Thread T1 (tid=16346, running) created by main thread at:
1:     #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:962 (libtsan.so.0+0x5ea79)
1:     #1 linux_kqueue_start_thread ../src/linux/platform.c:267 (libkqueue.so.2.4.0+0xcc71)
1:     #2 pthread_once ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1442 (libtsan.so.0+0x41964)
1:     #3 linux_kqueue_init ../src/linux/platform.c:369 (libkqueue.so.2.4.0+0xd73a)
1:     #4 kqueue ../src/common/kqueue.c:200 (libkqueue.so.2.4.0+0xab99)
1:     #5 test_harness ../test/main.c:302 (libkqueue-test+0x6519)
1:     #6 main ../test/main.c:409 (libkqueue-test+0x6b25)
1: 
1: SUMMARY: ThreadSanitizer: data race ../src/linux/platform.c:417 in linux_kqueue_cleanup
1: ==================
1: ThreadSanitizer: reported 8 warnings
1: Running 1 iterations
1: 1: test_peer_close_detection()   
1: 2: test_kqueue() 
1: 3: test_kevent() 
1: 4: test_cleanup()    
1: 5: test_ev_receipt() 
1: 6: test_kevent_socket_add()  
1: 7: test_kevent_socket_del()  
1: 8: test_kevent_socket_add_without_ev_add()   
1: 9: test_kevent_socket_get()  
1: 10: test_kevent_socket_disable_and_enable()  
1: 11: test_kevent_socket_oneshot() 
1: 12: test_kevent_socket_clear()   
1: 13: test_kevent_socket_dispatch()    
1: 14: test_kevent_socket_listen_backlog()  
1: 15: test_kevent_socket_eof() 
1: 16: test_kevent_regular_file()   
1: 17: test_transition_from_write_to_read() 
1: 18: test_kevent_signal_add() 
1: 19: test_kevent_signal_del() 
1: 20: test_kevent_signal_get() 
1: 21: test_kevent_signal_disable() 
1: 22: test_kevent_signal_enable()  
1: 23: test_kevent_signal_oneshot() 
1: 24: test_kevent_signal_modify()  
1: 25: test_kevent_signal_dispatch()    
1: 26: test_kevent_timer_add()  
1: 27: test_kevent_timer_del()  
1: 28: test_kevent_timer_get()  
1: 29: test_kevent_timer_oneshot()  
1: 30: test_kevent_timer_periodic() 
1: 31: test_kevent_timer_disable_and_enable()   
1: 32: test_kevent_timer_dispatch() 
1: 33: test_kevent_vnode_add()  
1: 34: test_kevent_vnode_del()  
1: 35: test_kevent_vnode_disable_and_enable()   
1: 36: test_kevent_vnode_dispatch() 
1: 37: test_kevent_vnode_note_write()   
1: 38: test_kevent_vnode_note_attrib()  
1: 39: test_kevent_vnode_note_rename()  
1: 40: test_kevent_vnode_note_delete()  
1: 41: test_kevent_user_add_and_delete()    
1: 42: test_kevent_user_get()   
1: 43: test_kevent_user_get_hires() 
1: 44: test_kevent_user_disable_and_enable()    
1: 45: test_kevent_user_oneshot()   
1: 46: test_kevent_user_multi_trigger_merged()  
1: 47: test_kevent_user_dispatch()  
1: 
1: ---
1: +OK All 47 tests completed.
1/1 Test #1: libkqueue-test ...................***Failed   18.34 sec

0% tests passed, 1 tests failed out of 1

Total Test time (real) =  18.34 sec

The following tests FAILED:
      1 - libkqueue-test (Failed)
Errors while running CTest

[timwoj]

I like the comment above the line reported in the first one:

/* Hopefully we aren't racing with another thread, but you never know.. */

[arr2036]

Hmm fun. OK, could you send a PR to run with TSAN under CI seeing as you appear to have a working invocation already :)

[timwoj]

Sure. Pretty much all I did was set CFLAGS=-fsanitize=thread prior to calling CMake.

[timwoj]

Opened a PR, but I'm not sure what magic I need to get Travis to kick off a build from it. I thought it was originally that I didn't open it from a master branch, but that wasn't it.

[timwoj]

Opened a PR, but I'm not sure what magic I need to get Travis to kick off a build from it. I thought it was originally that I didn't open it from a master branch, but that wasn't it.

Oh, it appears you guys are still on travis-ci.org, which is in read-only mode and won't let any new builds start. It needs to be migrated over to travis-ci.com (https://docs.travis-ci.com/user/migrate/open-source-repository-migration/#frequently-asked-questions). It looks like the last successful CI build was 10 months ago: https://travis-ci.org/github/mheily/libkqueue/requests.

[arr2036]

sigh that's not something I can fix unfortunately as I don't have admin rights for the repo. @mheily could you take a look at redoing the travis integration?

[mheily]

@arr2036 I have migrated the Travis CI repositories to travis-ci.com, and updated the build status image in README.md. It currently shows several failing jobs where TSAN is enabled; example: https://travis-ci.com/github/mheily/libkqueue/builds/235346115

[timwoj]

Those look like they match the errors I was seeing as well. Thanks @mheily !

[mheily]

I posted a PR in #107 that seem to fix the races in map.c, but am still seeing other data races in linux_kqueue_cleanup() and kqueue_free()

[arr2036]

f3eab262fcf961cfd465c0f52fa6896a4782aca2 fixes the map issues. The others do appear to be genuine races. We need to hold the global lock before attempting to resolve the fd to a struct kqueue *, and continue to hold it until we lock the struct kqueue specific mutex. We also need to hold both the global mutex and the struct kqueue specific mutex in kqueue_free().

I've started working on fixes. I'm planning to add the following extensions to control whether the mutexes are used (unless you have an objection @mheily?), as these extra mutexes are not required in applications which do not share kqueues between threads.

The reason why I wanted to avoid rwlocks in map.c is because in our application there's actually a fair amount of churn in that table. All threads in our application create and destroy temporary event loops, and so even with an r/w lock, any time the table needed to be updated (during the creation of a temporary event loop), we'd block other threads trying to resolve file descriptors to kqueues.

#define EVFILT_LIBKQUEUE     (-12)     //!< libkqueue specific extensions.

/** @name libkqueue extension flags
 *
 * Unless otherwise specified EV_ADD/EV_ENABLE toggle the configuration option to true.
 * Unless otherwise specified EV_DELETE/EV_DISABLE toggle the configuration option to false.
 *
 * @{
 */
#define NOTE_LIBKQUEUE_THREADSAFE_MAP   0x0001
                                          //!< Synchronise concurrent accesses to the
                                          ///< fd <-> kqueue mapping table.
                                          ///< Only needed if kqueues are created by
                                          ///< one thread and closed in another.
                                          ///< default - enabled.
#define NOTE_LIBKQUEUE_THREADSAFE 0x0001
                                          ///< Synchronise concurrent operations on a
                                          ///< given kqueue.
                                          ///< May be disabled if a given kqueue is only
                                          ///< used/closed by the thread that created it.
                                          ///< default - enabled.
#define NOTE_LIBKQUEUE_LOG_FUNC   0x0002  //!< For debug builds, call the logging function
                                          ///< provided in data.
/** @} */

[timwoj]

Any luck with the rest of these? I'm looking at upgrading our local libkqueue submodule pointer, but I'd love to have this stuff in a release before I update it.

[gerickson]

I can confirm I've noted these same TSAN issues in opencflite and opencfnetwork:

make[2]: Entering directory '${HOME}/src/git/github.com/gerickson/opencflite/tmp-build-fsanitize-thread/examples/CFRunLoopTimerExample'
make  CFRunLoopTimerExample
make[3]: Entering directory '${HOME}/src/git/github.com/gerickson/opencflite/tmp-build-fsanitize-thread/examples/CFRunLoopTimerExample'
make[3]: 'CFRunLoopTimerExample' is up to date.
make[3]: Leaving directory '${HOME}/src/git/github.com/gerickson/opencflite/tmp-build-fsanitize-thread/examples/CFRunLoopTimerExample'
/bin/bash ../../libtool --mode execute ./CFRunLoopTimerExample 0.25  5.00
Will fire a total of 1 timer for 5 seconds.
Will fire timer 0 every 0.25 seconds for 5 seconds, up to 20 times (with a tolerance of 1 time).
2021-12-01 07:13:04.175, timer: 0, delta: 0.250, variance: +0.000, iteration: 0
2021-12-01 07:13:04.425, timer: 0, delta: 0.250, variance: +0.000, iteration: 1
2021-12-01 07:13:04.675, timer: 0, delta: 0.250, variance: -0.000, iteration: 2
2021-12-01 07:13:04.925, timer: 0, delta: 0.250, variance: +0.000, iteration: 3
2021-12-01 07:13:05.175, timer: 0, delta: 0.250, variance: -0.000, iteration: 4
2021-12-01 07:13:05.426, timer: 0, delta: 0.251, variance: +0.001, iteration: 5
2021-12-01 07:13:05.675, timer: 0, delta: 0.249, variance: -0.001, iteration: 6
2021-12-01 07:13:05.925, timer: 0, delta: 0.250, variance: -0.000, iteration: 7
2021-12-01 07:13:06.176, timer: 0, delta: 0.251, variance: +0.001, iteration: 8
2021-12-01 07:13:06.425, timer: 0, delta: 0.250, variance: -0.000, iteration: 9
2021-12-01 07:13:06.676, timer: 0, delta: 0.250, variance: +0.000, iteration: 10
2021-12-01 07:13:06.926, timer: 0, delta: 0.250, variance: +0.000, iteration: 11
2021-12-01 07:13:07.175, timer: 0, delta: 0.249, variance: -0.001, iteration: 12
2021-12-01 07:13:07.425, timer: 0, delta: 0.250, variance: +0.000, iteration: 13
2021-12-01 07:13:07.675, timer: 0, delta: 0.250, variance: -0.000, iteration: 14
2021-12-01 07:13:07.925, timer: 0, delta: 0.250, variance: +0.000, iteration: 15
2021-12-01 07:13:08.175, timer: 0, delta: 0.250, variance: -0.000, iteration: 16
2021-12-01 07:13:08.426, timer: 0, delta: 0.251, variance: +0.001, iteration: 17
2021-12-01 07:13:08.675, timer: 0, delta: 0.249, variance: -0.001, iteration: 18
2021-12-01 07:13:08.926, timer: 0, delta: 0.250, variance: +0.000, iteration: 19
==================
WARNING: ThreadSanitizer: data race (pid=193538)
  Write of size 8 at 0x7ba000000030 by thread T1 (mutexes: write M14):
    #0 close ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1675 (libtsan.so.0+0x3458a)
    #1 linux_kqueue_cleanup ${HOME}/src/git/github.com/gerickson/libkqueue/src/linux/platform.c:396 (libkqueue.so.0+0x88cd)

  Previous read of size 8 at 0x7ba000000030 by main thread (mutexes: write M12, write M3794586426146832, write M16):
    #0 epoll_ctl ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1809 (libtsan.so.0+0x544bc)
    #1 linux_evfilt_user_knote_delete ${HOME}/src/git/github.com/gerickson/libkqueue/src/linux/user.c:183 (libkqueue.so.0+0x100a9)
    #2 __CFRunLoopDeallocate ../CFRunLoop.c:2028 (libCoreFoundation.so.635+0xe4b99)
    #3 __CFRunLoopDestroyRunLoop ../CFRunLoop.c:2161 (libCoreFoundation.so.635+0xe51c3)
    #4 CFDictionaryApplyFunction ../CFDictionary.c:736 (libCoreFoundation.so.635+0x9f863)
    #5 __CFRunLoopDestroyRunLoops ../CFRunLoop.c:2178 (libCoreFoundation.so.635+0xe525b)
    #6 __CFRunLoopDestroy ../CFRunLoop.c:2199 (libCoreFoundation.so.635+0xe52e7)
    #7 <null> <null> (ld-linux-x86-64.so.2+0x11f5a)

  Location is file descriptor 3 created by main thread at:
    #0 epoll_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1793 (libtsan.so.0+0x32d7a)
    #1 linux_kqueue_init ${HOME}/src/git/github.com/gerickson/libkqueue/src/linux/platform.c:271 (libkqueue.so.0+0x813a)
    #2 _CFRunLoopGet0 ../CFRunLoop.c:2213 (libCoreFoundation.so.635+0xe53fc)
    #3 CFRunLoopGetCurrent ../CFRunLoop.c:2341 (libCoreFoundation.so.635+0xe5b1f)
    #4 main ../../../examples/CFRunLoopTimerExample/CFRunLoopTimerExample.c:414 (CFRunLoopTimerExample+0x1eeb)

  Mutex M14 (0x7fc9776cd2e0) created at:
    #0 pthread_mutex_lock ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:4165 (libtsan.so.0+0x526fc)
    #1 kqueue ${HOME}/src/git/github.com/gerickson/libkqueue/src/common/kqueue.c:173 (libkqueue.so.0+0x6e1e)
    #2 _CFRunLoopGet0 ../CFRunLoop.c:2213 (libCoreFoundation.so.635+0xe53fc)
    #3 CFRunLoopGetCurrent ../CFRunLoop.c:2341 (libCoreFoundation.so.635+0xe5b1f)
    #4 main ../../../examples/CFRunLoopTimerExample/CFRunLoopTimerExample.c:414 (CFRunLoopTimerExample+0x1eeb)

  Mutex M12 (0x7fc977c1408c) created at:
    #0 pthread_spin_init ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1268 (libtsan.so.0+0x36aef)
    #1 __CFSpinLock ../CFInternal.h:480 (libCoreFoundation.so.635+0xe0736)
    #2 _CFRunLoopGet0 ../CFRunLoop.c:2209 (libCoreFoundation.so.635+0xe5389)
    #3 CFRunLoopGetCurrent ../CFRunLoop.c:2341 (libCoreFoundation.so.635+0xe5b1f)
    #4 main ../../../examples/CFRunLoopTimerExample/CFRunLoopTimerExample.c:414 (CFRunLoopTimerExample+0x1eeb)

  Mutex M3794586426146832 is already destroyed.

  Mutex M16 (0x7b94000007f0) created at:
    #0 pthread_mutex_init ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1220 (libtsan.so.0+0x4a616)
    #1 kqueue ${HOME}/src/git/github.com/gerickson/libkqueue/src/common/kqueue.c:182 (libkqueue.so.0+0x6e78)
    #2 _CFRunLoopGet0 ../CFRunLoop.c:2213 (libCoreFoundation.so.635+0xe53fc)
    #3 CFRunLoopGetCurrent ../CFRunLoop.c:2341 (libCoreFoundation.so.635+0xe5b1f)
    #4 main ../../../examples/CFRunLoopTimerExample/CFRunLoopTimerExample.c:414 (CFRunLoopTimerExample+0x1eeb)

  Thread T1 'libkqueue_mon' (tid=193565, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:962 (libtsan.so.0+0x5ea79)
    #1 linux_kqueue_start_thread ${HOME}/src/git/github.com/gerickson/libkqueue/src/linux/platform.c:257 (libkqueue.so.0+0x806e)
    #2 linux_kqueue_init ${HOME}/src/git/github.com/gerickson/libkqueue/src/linux/platform.c:350 (libkqueue.so.0+0x8619)
    #3 _CFRunLoopGet0 ../CFRunLoop.c:2213 (libCoreFoundation.so.635+0xe53fc)
    #4 CFRunLoopGetCurrent ../CFRunLoop.c:2341 (libCoreFoundation.so.635+0xe5b1f)
    #5 main ../../../examples/CFRunLoopTimerExample/CFRunLoopTimerExample.c:414 (CFRunLoopTimerExample+0x1eeb)

SUMMARY: ThreadSanitizer: data race ${HOME}/src/git/github.com/gerickson/libkqueue/src/linux/platform.c:396 in linux_kqueue_cleanup
==================
==================
WARNING: ThreadSanitizer: data race (pid=193538)
  Write of size 1 at 0x7b94000007f0 by thread T1 (mutexes: write M14):
    #0 pthread_mutex_destroy ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1237 (libtsan.so.0+0x3725c)
    #1 kqueue_free ${HOME}/src/git/github.com/gerickson/libkqueue/src/common/kqueue.c:132 (libkqueue.so.0+0x6d7c)

  Previous atomic read of size 1 at 0x7b94000007f0 by main thread (mutexes: write M12, write M3794586426146832, write M4639475212748784):
    #0 pthread_mutex_unlock ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:4179 (libtsan.so.0+0x3aadc)
    #1 kevent ${HOME}/src/git/github.com/gerickson/libkqueue/src/common/kevent.c:321 (libkqueue.so.0+0x49f5)
    #2 __CFRunLoopDeallocate ../CFRunLoop.c:2028 (libCoreFoundation.so.635+0xe4b99)
    #3 __CFRunLoopDestroyRunLoop ../CFRunLoop.c:2161 (libCoreFoundation.so.635+0xe51c3)
    #4 CFDictionaryApplyFunction ../CFDictionary.c:736 (libCoreFoundation.so.635+0x9f863)
    #5 __CFRunLoopDestroyRunLoops ../CFRunLoop.c:2178 (libCoreFoundation.so.635+0xe525b)
    #6 __CFRunLoopDestroy ../CFRunLoop.c:2199 (libCoreFoundation.so.635+0xe52e7)
    #7 <null> <null> (ld-linux-x86-64.so.2+0x11f5a)

  Location is heap block of size 8256 at 0x7b9400000000 allocated by main thread:
    #0 calloc ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:668 (libtsan.so.0+0x305aa)
    #1 kqueue ${HOME}/src/git/github.com/gerickson/libkqueue/src/common/kqueue.c:178 (libkqueue.so.0+0x6e4c)
    #2 _CFRunLoopGet0 ../CFRunLoop.c:2213 (libCoreFoundation.so.635+0xe53fc)
    #3 CFRunLoopGetCurrent ../CFRunLoop.c:2341 (libCoreFoundation.so.635+0xe5b1f)
    #4 main ../../../examples/CFRunLoopTimerExample/CFRunLoopTimerExample.c:414 (CFRunLoopTimerExample+0x1eeb)

  Mutex M14 (0x7fc9776cd2e0) created at:
    #0 pthread_mutex_lock ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:4165 (libtsan.so.0+0x526fc)
    #1 kqueue ${HOME}/src/git/github.com/gerickson/libkqueue/src/common/kqueue.c:173 (libkqueue.so.0+0x6e1e)
    #2 _CFRunLoopGet0 ../CFRunLoop.c:2213 (libCoreFoundation.so.635+0xe53fc)
    #3 CFRunLoopGetCurrent ../CFRunLoop.c:2341 (libCoreFoundation.so.635+0xe5b1f)
    #4 main ../../../examples/CFRunLoopTimerExample/CFRunLoopTimerExample.c:414 (CFRunLoopTimerExample+0x1eeb)

  Mutex M12 (0x7fc977c1408c) created at:
    #0 pthread_spin_init ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1268 (libtsan.so.0+0x36aef)
    #1 __CFSpinLock ../CFInternal.h:480 (libCoreFoundation.so.635+0xe0736)
    #2 _CFRunLoopGet0 ../CFRunLoop.c:2209 (libCoreFoundation.so.635+0xe5389)
    #3 CFRunLoopGetCurrent ../CFRunLoop.c:2341 (libCoreFoundation.so.635+0xe5b1f)
    #4 main ../../../examples/CFRunLoopTimerExample/CFRunLoopTimerExample.c:414 (CFRunLoopTimerExample+0x1eeb)

  Mutex M3794586426146832 is already destroyed.

  Mutex M4639475212748784 is already destroyed.

  Thread T1 'libkqueue_mon' (tid=193565, running) created by main thread at:
    #0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:962 (libtsan.so.0+0x5ea79)
    #1 linux_kqueue_start_thread ${HOME}/src/git/github.com/gerickson/libkqueue/src/linux/platform.c:257 (libkqueue.so.0+0x806e)
    #2 linux_kqueue_init ${HOME}/src/git/github.com/gerickson/libkqueue/src/linux/platform.c:350 (libkqueue.so.0+0x8619)
    #3 _CFRunLoopGet0 ../CFRunLoop.c:2213 (libCoreFoundation.so.635+0xe53fc)
    #4 CFRunLoopGetCurrent ../CFRunLoop.c:2341 (libCoreFoundation.so.635+0xe5b1f)
    #5 main ../../../examples/CFRunLoopTimerExample/CFRunLoopTimerExample.c:414 (CFRunLoopTimerExample+0x1eeb)

SUMMARY: ThreadSanitizer: data race ${HOME}/src/git/github.com/gerickson/libkqueue/src/common/kqueue.c:132 in kqueue_free
==================
ThreadSanitizer: reported 2 warnings
make[2]: *** [Makefile:634: check] Error 66
make[2]: Leaving directory '${HOME}/src/git/github.com/gerickson/opencflite/tmp-build-fsanitize-thread/examples/CFRunLoopTimerExample'

The dependent project was configured with:

% ./configure -C --with-icu=/usr --with-tz-includes=${HOME}/tmp/tzcode2021a CFLAGS="-O0 -g -fsanitize=thread" LDFLAGS="-O0 -g -fsanitize=thread" CPPFLAGS="-O0 -g -fsanitize=thread"

and built and tested with:

% make -j3 && make check

[arr2036]

All errors reported by TSAN have now been fixed and locking has been simplified substantially.

For common code we now have two types of mutex, the global mutex which is used to synchronise the fd map, the linked list of active kqueues, and the global kqueue counter, and kq specific mutexes which are used to prevent conflicting operations on the same kq.

I've removed the filter specific mutexes, there was never much of a reason to use them. This means only a single thread can operate on a kq at a given time.

If you're still experiencing issues with current HEAD, send over a PR with a test case which reproduces the issue and I'll take a look.

[timwoj]

Great, thanks for getting this sorted out! I'll pull the new version over into our builds this week and let you know if we run into anything else.

[gerickson]

Awesome, thank you! I'll pull from top of tree and get this verified in my dependent projects.

[timwoj]

Looks solid from my end. Huge thanks again from the Zeek team for fixing all of these.

[arr2036]

@timwoj great, thanks for confirming. Just checking a hang-on-exit issue is fixed in another project and I'll cut a new release.

[arr2036]

OK, it's fixed. No more hanging CI jobs. v2.6.0 has been tagged.

[gerickson]

Confirmed with opencflite that all thread sanitizer issues have been resolved. Thanks!

[arr2036]

@gerickson awesome, thanks for confirming!