search

theRockLiu / thread-sanitizer

Automatically exported from code.google.com/p/thread-sanitizer
0 stars 0 forks source link

<empty stack>s for deadlock detector v2 in Ubuntu 14.04 #63

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
Built tip of tree standalone deadlock detector in llvm (with cmake/ninja using 
CC=clang, CXX=clang++, clang version 3.5.0 (trunk 206824)).

Built regular webrtc (tip of tree) in debug mode:

gyp: (unset CC; unset CXX; cd ~/webrtc/trunk/ && ./webrtc/build/gyp_webrtc 
-Dclang=1 -Ddebug_extra_cflags="-fcolor-diagnostics")
ninja: (cd ~/webrtc/trunk/ && ninja -C out/Debug)

$LD_PRELOAD=~/llvm/build/lib/libclang_rt.dyndd-x86_64.so 
~/webrtc/trunk/out/Debug/libjingle_media_unittest --gtest_filter=*2*.*

With and without DSAN_OPTIONS=second_deadlock_stack=1 I don't get a stack for 
the lock-order inversion that is present.

...
[ RUN      ] WebRtcVideoChannel2BaseTest.SetSend
[000:010] WebRtcVideoEngine2::WebRtcVideoEngine2
[000:011] WebRtcVideoEngine2::Init
[000:057] CreateChannel: Without voice channel.
[000:091] SetRenderer: ssrc:0 (ptr)
[000:091] SetRecvCodecs: {VideoCodec[100:VP8:640:400:30:0], 
VideoCodec[116:red:0:0:0:0], VideoCodec[117:ulpfec:0:0:0:0], 
VideoCodec[96:rtx:0:0:0:0]}
[000:092] AddSendStream: {ssrcs:[1234];ssrc_groups:;}
[000:092] SetSendCodecs: {VideoCodec[100:VP8:640:400:30:0], 
VideoCodec[116:red:0:0:0:0], VideoCodec[117:ulpfec:0:0:0:0], 
VideoCodec[96:rtx:0:0:0:0]}
[000:093] Using codec: VideoCodec[100:VP8:640:400:30:0]
[000:196]  Capture Requested I420 640x480x30
[000:197]  Supported I420 1280x720x30 distance 171845877760
[000:197]  Supported I420 640x480x30 distance 0
[000:197]  Supported I420 320x240x30 distance 257698037760
[000:197]  Supported I420 160x120x30 distance 386547056640
[000:197]  Best I420 640x480x30 Interval 33333333 distance 0
[000:197] VAdapt input interval changed from 0 to 33333333
[000:198] SetCapturer: 1234 -> (capturer)
[000:199] SetCapturer: 1234 -> (capturer)
[000:201]  Capture Requested I420 640x400x30
[000:201]  Supported I420 1280x720x30 distance 171814420480
[000:201]  Supported I420 640x480x30 distance 5242880
[000:201]  Supported I420 320x240x30 distance 257700659200
[000:201]  Supported I420 160x120x30 distance 386548367360
[000:202]  Best I420 640x480x30 Interval 33333333 distance 5242880
[000:205] SetSendCodecs: {VideoCodec[100:VP8:640:400:30:0]}
[000:206] Using codec: VideoCodec[100:VP8:640:400:30:0]
==============================
WARNING: lock-order-inversion (potential deadlock)
Thread 0 locks mutex 186 while holding mutex 185:
    <empty stack>

Thread 0 locks mutex 185 while holding mutex 186:
    <empty stack>

==============================
==============================
WARNING: lock-order-inversion (potential deadlock)
Thread 0 locks mutex 19 while holding mutex 18:
    <empty stack>

Thread 0 locks mutex 18 while holding mutex 19:
    <empty stack>

==============================
[000:665] WebRtcVideoEngine2::Terminate
[000:674] WebRtcVideoEngine2::~WebRtcVideoEngine2
[       OK ] WebRtcVideoChannel2BaseTest.SetSend (667 ms)
...

Original issue reported on code.google.com by pbos@google.com on 16 May 2014 at 9:37

GoogleCodeExporter commented 9 years ago 
kcc, please see if you can repro on your 14.04.

Original comment by pbos@google.com on 16 May 2014 at 9:38

GoogleCodeExporter commented 9 years ago 
Does tsan's dd (TSAN_OPTIONS=detect_deadlocks=1:second_deadlock_stack=1) work 
for you?

Original comment by konstant...@gmail.com on 16 May 2014 at 9:42

GoogleCodeExporter commented 9 years ago 
Does both with and without second_deadlock_stack=1. Building webrtc with tsan 
from clang version 3.5.0 (trunk 206824), I'll try building with tip of tree 
clang as well.

Original comment by pbos@google.com on 16 May 2014 at 10:18

GoogleCodeExporter commented 9 years ago 
The standalone DD is in Dmitry's hands. 
In fact, I really dislike the idea of having this standalone dd as a separate 
entity -- too much additional work. 
We need to try to incorporate this in to one run-time (tsan), but *probably* 
allow it to work in standalone mode (LD_PRELOAD)

Original comment by konstant...@gmail.com on 16 May 2014 at 10:37

GoogleCodeExporter commented 9 years ago 
Issue in ToT clang (synced today), but only without second_deadlock_stack=1:

Built gyp:
(export GYP_GENERATOR_OUTPUT=out-tsan; export 
CC=/usr/local/google/home/pbos/llvm/build/bin/clang; export 
CXX=/usr/local/google/home/pbos/llvm/build/bin/clang++; cd ~/webrtc/trunk/ && 
./webrtc/build/^Cp_webrtc -D tsan=1 -D clang_use_chrome_plugins=0 
-Ddebug_extra_cflags="-fcolor-diagnostics")

ninja as above.

Output:

$ TSAN_OPTIONS=detect_deadlocks=1 
~/webrtc/trunk/out-tsan/out/Debug/libjingle_media_unittest 
--gtest_filter=*2*.SetSend

[ RUN      ] WebRtcVideoChannel2BaseTest.SetSend
[000:006] WebRtcVideoEngine2::WebRtcVideoEngine2
[000:010] WebRtcVideoEngine2::Init
[000:138] CreateChannel: Without voice channel.
[000:224] SetRenderer: ssrc:0 (ptr)
[000:225] SetRecvCodecs: {VideoCodec[100:VP8:640:400:30:0], 
VideoCodec[116:red:0:0:0:0], VideoCodec[117:ulpfec:0:0:0:0], 
VideoCodec[96:rtx:0:0:0:0]}
[000:226] AddSendStream: {ssrcs:[1234];ssrc_groups:;}
[000:226] SetSendCodecs: {VideoCodec[100:VP8:640:400:30:0], 
VideoCodec[116:red:0:0:0:0], VideoCodec[117:ulpfec:0:0:0:0], 
VideoCodec[96:rtx:0:0:0:0]}
[000:228] Using codec: VideoCodec[100:VP8:640:400:30:0]
[000:388]  Capture Requested I420 640x480x30
[000:388]  Supported I420 1280x720x30 distance 171845877760
[000:389]  Supported I420 640x480x30 distance 0
[000:389]  Supported I420 320x240x30 distance 257698037760
[000:389]  Supported I420 160x120x30 distance 386547056640
[000:390]  Best I420 640x480x30 Interval 33333333 distance 0
[000:390] VAdapt input interval changed from 0 to 33333333
[000:390] SetCapturer: 1234 -> (capturer)
[000:391] SetCapturer: 1234 -> (capturer)
[000:391]  Capture Requested I420 640x400x30
[000:391]  Supported I420 1280x720x30 distance 171814420480
[000:392]  Supported I420 640x480x30 distance 5242880
[000:392]  Supported I420 320x240x30 distance 257700659200
[000:392]  Supported I420 160x120x30 distance 386548367360
[000:393]  Best I420 640x480x30 Interval 33333333 distance 5242880
[000:395] SetSendCodecs: {VideoCodec[100:VP8:640:400:30:0]}
[000:396] Using codec: VideoCodec[100:VP8:640:400:30:0]
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=368)
  Cycle in lock order graph: M170 (0x000000000000) => M171 (0x000000000000) => M170

  Mutex M171 acquired here while holding mutex M170 in main thread:
    #0 <null> <null>:0 (0x000000000042)

    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message

  Mutex M170 acquired here while holding mutex M171 in main thread:
    #0 <null> <null>:0 (0x000000000042)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) ??:0 ??
==================
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=368)
  Cycle in lock order graph: M3 (0x000000000000) => M4 (0x000000000000) => M3

  Mutex M4 acquired here while holding mutex M3 in main thread:
    #0 <null> <null>:0 (0x000000000042)

    Hint: use TSAN_OPTIONS=second_deadlock_stack=1 to get more informative warning message

  Mutex M3 acquired here while holding mutex M4 in main thread:
    #0 <null> <null>:0 (0x000000000042)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) ??:0 ??
==================
[001:426] WebRtcVideoEngine2::Terminate
[001:427] WebRtcVideoEngine2::~WebRtcVideoEngine2
[       OK ] WebRtcVideoChannel2BaseTest.SetSend (1422 ms)

$ TSAN_OPTIONS=detect_deadlocks=1:second_deadlock_stack=1 
~/webrtc/trunk/out-tsan/out/Debug/libjingle_media_unittest 
--gtest_filter=*2*.SetSend

[ RUN      ] WebRtcVideoChannel2BaseTest.SetSend
[000:000] WebRtcVideoEngine2::WebRtcVideoEngine2
[000:005] WebRtcVideoEngine2::Init
[000:126] CreateChannel: Without voice channel.
[000:217] SetRenderer: ssrc:0 (ptr)
[000:217] SetRecvCodecs: {VideoCodec[100:VP8:640:400:30:0], 
VideoCodec[116:red:0:0:0:0], VideoCodec[117:ulpfec:0:0:0:0], 
VideoCodec[96:rtx:0:0:0:0]}
[000:218] AddSendStream: {ssrcs:[1234];ssrc_groups:;}
[000:219] SetSendCodecs: {VideoCodec[100:VP8:640:400:30:0], 
VideoCodec[116:red:0:0:0:0], VideoCodec[117:ulpfec:0:0:0:0], 
VideoCodec[96:rtx:0:0:0:0]}
[000:220] Using codec: VideoCodec[100:VP8:640:400:30:0]
[000:371]  Capture Requested I420 640x480x30
[000:372]  Supported I420 1280x720x30 distance 171845877760
[000:372]  Supported I420 640x480x30 distance 0
[000:373]  Supported I420 320x240x30 distance 257698037760
[000:373]  Supported I420 160x120x30 distance 386547056640
[000:373]  Best I420 640x480x30 Interval 33333333 distance 0
[000:374] VAdapt input interval changed from 0 to 33333333
[000:374] SetCapturer: 1234 -> (capturer)
[000:374] SetCapturer: 1234 -> (capturer)
[000:374]  Capture Requested I420 640x400x30
[000:375]  Supported I420 1280x720x30 distance 171814420480
[000:375]  Supported I420 640x480x30 distance 5242880
[000:375]  Supported I420 320x240x30 distance 257700659200
[000:376]  Supported I420 160x120x30 distance 386548367360
[000:376]  Best I420 640x480x30 Interval 33333333 distance 5242880
[000:379] SetSendCodecs: {VideoCodec[100:VP8:640:400:30:0]}
[000:379] Using codec: VideoCodec[100:VP8:640:400:30:0]
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=744)
  Cycle in lock order graph: M170 (0x000000000000) => M171 (0x000000000000) => M170

  Mutex M171 acquired here while holding mutex M170 in main thread:
    #0 pthread_mutex_lock /usr/local/google/home/pbos/llvm/build/../projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:2688 (libjingle_media_unittest+0x0000000e3ff0)
...
    #19 main /usr/local/google/home/pbos/webrtc/trunk/out-tsan/out/Debug/../../../talk/base/unittest_main.cc:82:13 (libjingle_media_unittest+0x000000bcd835)

  Mutex M170 acquired here while holding mutex M171 in main thread:
    #0 pthread_mutex_lock /usr/local/google/home/pbos/llvm/build/../projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:2688 (libjingle_media_unittest+0x0000000e3ff0)
...
    #22 main /usr/local/google/home/pbos/webrtc/trunk/out-tsan/out/Debug/../../../talk/base/unittest_main.cc:82:13 (libjingle_media_unittest+0x000000bcd835)

  Mutex M171 previously acquired by the same thread here:
    #0 pthread_mutex_lock /usr/local/google/home/pbos/llvm/build/../projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:2688 (libjingle_media_unittest+0x0000000e3ff0)
...
    #24 main /usr/local/google/home/pbos/webrtc/trunk/out-tsan/out/Debug/../../../talk/base/unittest_main.cc:82:13 (libjingle_media_unittest+0x000000bcd835)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) 
/usr/local/google/home/pbos/webrtc/trunk/out-tsan/out/Debug/../../../talk/base/c
riticalsection.h:105 talk_base::CriticalSection::Enter()
==================
[001:304] AddRecvStream: {ssrcs:[1234];ssrc_groups:;}
==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=744)
  Cycle in lock order graph: M3 (0x000000000000) => M4 (0x000000000000) => M3

  Mutex M4 acquired here while holding mutex M3 in main thread:
    #0 pthread_mutex_lock /usr/local/google/home/pbos/llvm/build/../projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:2688 (libjingle_media_unittest+0x0000000e3ff0)
...
    #18 main /usr/local/google/home/pbos/webrtc/trunk/out-tsan/out/Debug/../../../talk/base/unittest_main.cc:82:13 (libjingle_media_unittest+0x000000bcd835)

  Mutex M3 previously acquired by the same thread here:
    #0 pthread_mutex_lock /usr/local/google/home/pbos/llvm/build/../projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:2688 (libjingle_media_unittest+0x0000000e3ff0)
...
    #21 main /usr/local/google/home/pbos/webrtc/trunk/out-tsan/out/Debug/../../../talk/base/unittest_main.cc:82:13 (libjingle_media_unittest+0x000000bcd835)

  Mutex M3 acquired here while holding mutex M4 in main thread:
    #0 pthread_mutex_lock /usr/local/google/home/pbos/llvm/build/../projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:2688 (libjingle_media_unittest+0x0000000e3ff0)
...
    #22 main /usr/local/google/home/pbos/webrtc/trunk/out-tsan/out/Debug/../../../talk/base/unittest_main.cc:82:13 (libjingle_media_unittest+0x000000bcd835)

  Mutex M4 previously acquired by the same thread here:
    #0 pthread_mutex_lock /usr/local/google/home/pbos/llvm/build/../projects/compiler-rt/lib/tsan/../sanitizer_common/sanitizer_common_interceptors.inc:2688 (libjingle_media_unittest+0x0000000e3ff0)
...
    #23 main /usr/local/google/home/pbos/webrtc/trunk/out-tsan/out/Debug/../../../talk/base/unittest_main.cc:82:13 (libjingle_media_unittest+0x000000bcd835)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) 
/usr/local/google/home/pbos/webrtc/trunk/out-tsan/out/Debug/../../../talk/base/c
riticalsection.h:105 talk_base::CriticalSection::Enter()
==================
[001:866] WebRtcVideoEngine2::Terminate
[001:867] WebRtcVideoEngine2::~WebRtcVideoEngine2
[       OK ] WebRtcVideoChannel2BaseTest.SetSend (1868 ms)

Original comment by pbos@google.com on 16 May 2014 at 10:45

GoogleCodeExporter commented 9 years ago 
So, what's the issue? 
second_deadlock_stack=1 enables the second stack

Original comment by konstant...@gmail.com on 16 May 2014 at 10:57

GoogleCodeExporter commented 9 years ago 
I built this with:

# define SANITIZER_DEADLOCK_DETECTOR_VERSION 2
projects/compiler-rt/lib/sanitizer_common/sanitizer_deadlock_detector_interface.
h

which I use to get a standalone DD, I'm rebuilding with 
SANITIZER_DEADLOCK_DETECTOR_VERSION=1 now to see if this is only present in 
SANITIZER_DEADLOCK_DETECTOR_VERSION=2. Without second_deadlock_stack=1 there 
should still be at least one valid stack trace, right?

Original comment by pbos@google.com on 16 May 2014 at 11:11

GoogleCodeExporter commented 9 years ago 
Yep, this works with and without second_deadlock_stack=1 in ToT for 
SANITIZER_DEADLOCK_DETECTOR_VERSION=1. This is only an issue in 
SANITIZER_DEADLOCK_DETECTOR_VERSION=2.

Original comment by pbos@google.com on 16 May 2014 at 11:18