search

theRockLiu / thread-sanitizer

Automatically exported from code.google.com/p/thread-sanitizer
0 stars 0 forks source link

CHECK failure in sanitizer_deadlock_detector1.cc #49

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Ran TSAN_OPTIONS=detect_deadlocks=1 gdb -ex 'set disable-randomization off' -ex 
'set print thread-events off' out/Debug/video_engine_tests

where video_engine_tests is built from a code.webrtc.org checkout (in trunk):

$ ./webrtc/build/gyp_webrtc -D tsan=1 -D clang_use_chrome_plugins=0
$ ninja -C out/Debug video_engine_tests

What is the expected output? What do you see instead?

A lot of mutex inversion error reports, but no internal tsan crashes.

After a lot of error reports I instead get a CHECK failure:

FATAL: ThreadSanitizer CHECK failed: 
../projects/compiler-rt/lib/sanitizer_common/sanitizer_deadlock_detector1.cc:102
 "((!dd.isHeld(&lt->dd, m->id))) != (0)" (0x0, 0x0)
    #0 __tsan::PrintCurrentStackSlow() /usr/local/google/home/pbos/llvm/build/../projects/compiler-rt/lib/tsan/rtl/tsan_rtl_report.cc:724 (video_engine_tests+0x0000000bc30c)
    #1 __tsan::TsanCheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /usr/local/google/home/pbos/llvm/build/../projects/compiler-rt/lib/tsan/rtl/tsan_rtl_report.cc:44 (video_engine_tests+0x0000000bc3e3)
    #2 __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) /usr/local/google/home/pbos/llvm/build/../projects/compiler-rt/lib/sanitizer_common/sanitizer_common.cc:76 (video_engine_tests+0x0000000c1de3)
    #3 __sanitizer::DDetectorImpl::MutexLock(__sanitizer::DDPhysicalThread*, __sanitizer::DDLogicalThread*, __sanitizer::DDMutex*, bool, bool) /usr/local/google/home/pbos/llvm/build/../projects/compiler-rt/lib/sanitizer_common/sanitizer_deadlock_detector1.cc:102 (video_engine_tests+0x0000000c3f30)
    #4 __tsan::MutexReadLock(__tsan::ThreadState*, unsigned long, unsigned long, bool) /usr/local/google/home/pbos/llvm/build/../projects/compiler-rt/lib/tsan/rtl/tsan_rtl_mutex.cc:184 (video_engine_tests+0x0000000b91b8)
    #5 pthread_rwlock_rdlock /usr/local/google/home/pbos/llvm/build/../projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:1041 (video_engine_tests+0x00000006a112)
    #6 webrtc::RWLockPosix::AcquireLockShared() /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/system_wrappers/source/rw_lock_posix.cc:44 (video_engine_tests+0x0000002bbc09)
    #7 webrtc::ViEManagerBase::ReadLockManager() const /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/video_engine/vie_manager_base.cc:27 (video_engine_tests+0x0000004d865a)
    #8 webrtc::ViEManagerScopedBase::ViEManagerScopedBase(webrtc::ViEManagerBase const&) /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/video_engine/vie_manager_base.cc:45 (video_engine_tests+0x0000004d889f)
    #9 webrtc::ViEChannelManagerScoped::ViEChannelManagerScoped(webrtc::ViEChannelManager const&) /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/video_engine/vie_channel_manager.cc:532 (video_engine_tests+0x000000492503)
    #10 webrtc::ViECodecImpl::GetSendSideDelay(int, int*, int*) const /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/video_engine/vie_codec_impl.cc:743 (video_engine_tests+0x00000044692f)
    #11 webrtc::internal::VideoSendStream::GetSendSideDelay(webrtc::VideoSendStream::Stats*) /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/video/video_send_stream.cc:270 (video_engine_tests+0x0000002b5304)
    #12 non-virtual thunk to webrtc::internal::VideoSendStream::GetSendSideDelay(webrtc::VideoSendStream::Stats*) /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/video/video_send_stream.cc:272 (video_engine_tests+0x0000002b5397)
    #13 webrtc::SendStatisticsProxy::GetStats() const /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/video/send_statistics_proxy.cc:47 (video_engine_tests+0x0000002a4344)
    #14 webrtc::internal::VideoSendStream::GetStats() const /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/video/video_send_stream.cc:266 (video_engine_tests+0x0000002b5200)
    #15 webrtc::VideoSendStreamTest_ProducesStats_Test::TestBody()::StatsObserver::OnSendRtcp(unsigned char const*, unsigned long) /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/video/video_send_stream_tests.cc:1092 (video_engine_tests+0x000000152175)
    #16 webrtc::test::RtpRtcpObserver::PacketTransport::SendRtcp(unsigned char const*, unsigned long) /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/test/rtp_rtcp_observer.h:149 (video_engine_tests+0x00000012e34e)
    #17 webrtc::internal::TransportAdapter::SendRTCPPacket(int, void const*, int) /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/video/transport_adapter.cc:36 (video_engine_tests+0x0000002a8aae)
    #18 webrtc::ViESender::SendRTCPPacket(int, void const*, int) /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/video_engine/vie_sender.cc:129 (video_engine_tests+0x0000004f0bf4)
    #19 webrtc::RTCPSender::SendToNetwork(unsigned char const*, unsigned short) /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/modules/rtp_rtcp/source/rtcp_sender.cc:2108 (video_engine_tests+0x00000022e154)
    #20 webrtc::RTCPSender::SendRTCP(webrtc::RTCPSender::FeedbackState const&, unsigned int, int, unsigned short const*, bool, unsigned long) /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/modules/rtp_rtcp/source/rtcp_sender.cc:1736 (video_engine_tests+0x00000021c6d4)
    #21 webrtc::RTCPSender::SetSendingStatus(webrtc::RTCPSender::FeedbackState const&, bool) /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/modules/rtp_rtcp/source/rtcp_sender.cc:310 (video_engine_tests+0x00000021c225)
    #22 webrtc::ModuleRtpRtcpImpl::SetSendingStatus(bool) /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/modules/rtp_rtcp/source/rtp_rtcp_impl.cc:501 (video_engine_tests+0x00000076630f)
    #23 webrtc::ViEChannel::StopSend() /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/video_engine/vie_channel.cc:1577 (video_engine_tests+0x00000047d1ac)
    #24 webrtc::ViEBaseImpl::StopSend(int) /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/video_engine/vie_base_impl.cc:323 (video_engine_tests+0x000000436b01)
    #25 webrtc::internal::VideoSendStream::StopSending() /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/video/video_send_stream.cc:215 (video_engine_tests+0x0000002b482b)
    #26 webrtc::VideoSendStreamTest_ProducesStats_Test::TestBody() /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/video/video_send_stream_tests.cc:1141 (video_engine_tests+0x000000151888)
    #27 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../testing/gtest/src/gtest.cc:1989 (video_engine_tests+0x0000001d85f0)
    #28 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../testing/gtest/src/gtest.cc:2042 (video_engine_tests+0x0000001a1c19)
    #29 testing::Test::Run() /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../testing/gtest/src/gtest.cc:2061 (video_engine_tests+0x00000018d836)
    #30 testing::TestInfo::Run() /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../testing/gtest/src/gtest.cc:2237 (video_engine_tests+0x00000018e737)
    #31 testing::TestCase::Run() /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../testing/gtest/src/gtest.cc:2344 (video_engine_tests+0x00000018f350)
    #32 testing::internal::UnitTestImpl::RunAllTests() /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../testing/gtest/src/gtest.cc:4065 (video_engine_tests+0x000000197447)
    #33 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../testing/gtest/src/gtest.cc:1989 (video_engine_tests+0x0000001c1f40)
    #34 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../testing/gtest/src/gtest.cc:2042 (video_engine_tests+0x0000001a6b5f)
    #35 testing::UnitTest::Run() /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../testing/gtest/src/gtest.cc:3697 (video_engine_tests+0x000000196d1c)
    #36 RUN_ALL_TESTS() /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../testing/gtest/include/gtest/gtest.h:2231 (video_engine_tests+0x00000028787d)
    #37 webrtc::test::RunAllTests() /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/test/run_tests.cc:16:10 (video_engine_tests+0x0000002876b5)
    #38 main /usr/local/google/home/pbos/webrtc/trunk/out/Debug/../../webrtc/test/test_main.cc:22 (video_engine_tests+0x00000017f42d)
    #39 __libc_start_main /build/buildd/eglibc-2.15/csu/libc-start.c:226 (libc.so.6+0x00000002176c)
    #40 _start <null>:0 (video_engine_tests+0x0000000d0260)                      

What version of the product are you using? On what operating system?

Tip of tree built yesterday (2014-03-03), Ubuntu 12.04.

Original issue reported on code.google.com by pbos@google.com on 4 Mar 2014 at 9:27

GoogleCodeExporter commented 9 years ago

Original comment by dvyu...@google.com on 4 Mar 2014 at 9:28

GoogleCodeExporter commented 9 years ago 
Yea, that's a recursive reader lock (right?).
Haven't implemented that yet...
(see test/tsan/deadlock_detector_stress_test.cc:
// LockTest().Test15();  // FIXME: this is broken for PthreadRWLock

Original comment by konstant...@gmail.com on 11 Mar 2014 at 1:28

GoogleCodeExporter commented 9 years ago 
It should be, thanks Kostya.

Original comment by pbos@google.com on 11 Mar 2014 at 7:51

GoogleCodeExporter commented 9 years ago 
I hope this should be fixed by
http://llvm.org/viewvc/llvm-project?view=revision&revision=203779
Let me check on webrtc

Original comment by konstant...@gmail.com on 13 Mar 2014 at 10:34

GoogleCodeExporter commented 9 years ago 
I used the following sequence of commands to verify that there is not crash 
left:

gclient config http://webrtc.googlecode.com/svn/trunk
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk-amd64
gclient sync --force
cd trunk
CC=clang CXX=clang++ ./webrtc/build/gyp_webrtc -D tsan=1 -D 
clang_use_chrome_plugins=0
ninja -C out/Debug video_engine_tests
TSAN_OPTIONS=detect_deadlocks=1  out/Debug/video_engine_tests

Original comment by konstant...@gmail.com on 13 Mar 2014 at 12:04