search

yugabyte / cassandra-cpp-driver

YugabyteDB C++ Driver for YCQL, based on the DataStax Driver
https://docs.yugabyte.com/latest/develop/client-drivers/cpp/
Apache License 2.0
8 stars 11 forks source link

Fix TSAN lock-order-inversion (potential deadlock) #18

Open amitanandaiyer opened 1 year ago

amitanandaiyer commented 1 year ago

================== WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=17173) Cycle in lock order graph: M0 (0x7b7c00001200) => M1 (0x7b7c000013c0) => M0

Mutex M1 acquired here while holding mutex M0 in main thread:

5 cass::Session::clear(cass::Config const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/cassandra-cpp-driver-2.9.0-yb- 13/src/session.cpp:181:17 (libcassandra.so.2+0x20848c)

 #6 cass::Session::connect_async(cass::Config const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, cass::SharedRefPtr<cass::Future>  const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/cassandra-cpp-driver-2.9.0-yb-13/src/session.cpp:351:3 (libcassandra.so.  2+0x2054c4)

Mutex M0 previously acquired by the same thread here:

7 cass::Session::connect_async(cass::Config const&, std::1::basic_string<char, std::__1::char_traits, std::1::allocator> const&, cass::SharedRefPtr const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/cassandra-cpp-driver-2.9.0-yb-13/src/session.cpp:343:15 (libcassandra.so. 2+0x205494)

 #6 cass_session_connect_keyspace_n /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/cassandra-cpp-driver-2.9.0-yb-13/src/      session.cpp:68:12 (libcassandra.so.2+0x2052d3)
 #7 cass_session_connect_keyspace /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/cassandra-cpp-driver-2.9.0-yb-13/src/        session.cpp:57:10 (libcassandra.so.2+0x205079)
 #8 cass_session_connect /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/cassandra-cpp-driver-2.9.0-yb-13/src/session.cpp:51:  10 (libcassandra.so.2+0x205079)
 #9 yb::CassandraSession::Connect(CassCluster_*) ${BUILD_ROOT}/../../src/yb/integration-tests/cql_test_util.cc:351:26 (libcql_test_util.so+0x16921)
 #10 yb::CassandraSession::Create(CassCluster_*) ${BUILD_ROOT}/../../src/yb/integration-tests/cql_test_util.cc:357:3 (libcql_test_util.so+0x16a89)
 #11 yb::CppCassandraDriver::CreateSession() ${BUILD_ROOT}/../../src/yb/integration-tests/cql_test_util.cc:500:10 (libcql_test_util.so+0x184c8)
 #12 yb::tools::AdminTestBase::CqlConnect(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) ${BUILD_ROOT}/../../src/yb/tools/admin-test-  base.cc:77:17 (libadmin-test-base.so+0xc57a)
 #13 yb::tools::AdminCliTest_AddTransactionStatusTablet_Test::TestBody() ${BUILD_ROOT}/../../src/yb/tools/yb-admin-test.cc:1023:18 (yb-admin-test+0x15f7d6)
 #14 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-  db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/googletest-1.12.1/googletest/src/gtest.cc:2599:10 (libgtest.so.1.12.1+0x8c2b9)
 #15 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-  thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/googletest-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so.1.12.1+0x8c2b9)
 #16 testing::Test::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/googletest-1.12.1/googletest/src/gtest.cc:2674:5     (libgtest.so.1.12.1+0x6358f)
 #17 testing::TestInfo::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/googletest-1.12.1/googletest/src/gtest.cc:2853:  11 (libgtest.so.1.12.1+0x64d69)
 #18 testing::TestSuite::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/googletest-1.12.1/googletest/src/gtest.cc:3012: 30 (libgtest.so.1.12.1+0x65fc3)
 #19 testing::internal::UnitTestImpl::RunAllTests() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/googletest-1.12.1/         googletest/src/gtest.cc:5870:44 (libgtest.so.1.12.1+0x7ebd2)
 #20 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::               UnitTestImpl::*)(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/googletest-1.12.1/googletest/src/gtest.cc:2599:  10 (libgtest.so.1.12.1+0x8d7a9)
 #21 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*   )(), char const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/googletest-1.12.1/googletest/src/gtest.cc:2635:14 (libgtest.so. 1.12.1+0x8d7a9)
 #22 testing::UnitTest::Run() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/googletest-1.12.1/googletest/src/gtest.cc:5444:  10 (libgtest.so.1.12.1+0x7e3ed)
 #23 RUN_ALL_TESTS() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/installed/tsan/include/gtest/gtest.h:2293:73                  (libyb_test_main.so+0x49e3)
 #24 main ${BUILD_ROOT}/../../src/yb/util/test_main.cc:109:13 (libyb_test_main.so+0x49e3)

Mutex M0 acquired here while holding mutex M1 in thread T68:

5 cass::Session::refresh_metadatacallback(CassFuture, void) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/cassandra- cpp-driver-2.9.0-yb-13/src/session.cpp:482:17 (libcassandra.so.2+0x20c14c)

Mutex M1 previously acquired by the same thread here:

5 cass::Session::refresh_metadatacallback(CassFuture, void) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/cassandra- cpp-driver-2.9.0-yb-13/src/session.cpp:478:15 (libcassandra.so.2+0x20c0d1)

 #6 cass::Future::internal_set(cass::ScopedLock<cass::Mutex>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/cassandra-cpp-  driver-2.9.0-yb-13/src/future.cpp:190:5 (libcassandra.so.2+0x181c10)
 #7 cass::ResponseFuture::set_response(cass::Address, cass::SharedRefPtr<cass::Response> const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7- x86_64-clang15/src/cassandra-cpp-driver-2.9.0-yb-13/src/request_handler.hpp:64:7 (libcassandra.so.2+0x1f7297)
 #8 cass::RequestHandler::set_response(cass::SharedRefPtr<cass::Host> const&, cass::SharedRefPtr<cass::Response> const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-            v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/cassandra-cpp-driver-2.9.0-yb-13/src/request_handler.cpp:142:16 (libcassandra.so.2+0x1f5621)
 #9 cass::RequestExecution::set_response(cass::SharedRefPtr<cass::Response> const&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-       clang15/src/cassandra-cpp-driver-2.9.0-yb-13/src/request_handler.cpp:538:21 (libcassandra.so.2+0x1f5621)
 #10 cass::RequestExecution::on_result_response(cass::Connection*, cass::ResponseMessage*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-       x86_64-clang15/src/cassandra-cpp-driver-2.9.0-yb-13/src/request_handler.cpp:369:7 (libcassandra.so.2+0x1f5621)
 #11 cass::RequestExecution::on_set(cass::ResponseMessage*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/cassandra-cpp-     driver-2.9.0-yb-13/src/request_handler.cpp:252:7 (libcassandra.so.2+0x1f4e2f)
 #12 cass::Connection::consume(char*, unsigned long) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/cassandra-cpp-driver-2.9. 0-yb-13/src/connection.cpp:472:25 (libcassandra.so.2+0x14607a)
 #13 cass::Connection::on_read(uv_stream_s*, long, uv_buf_t const*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/cassandra- cpp-driver-2.9.0-yb-13/src/connection.cpp:670:15 (libcassandra.so.2+0x146cbd)
 #14 uv__read /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/libuv-1.23.0/src/unix/stream.c:1257:7 (libuv.so+0x29dce)
 #15 uv__stream_io /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/libuv-1.23.0/src/unix/stream.c:1324:5 (libuv.so+0x29dce)
 #16 uv__io_poll /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/libuv-1.23.0/src/unix/linux-core.c:401:11 (libuv.so+0x31e0c)      #17 uv_run /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/libuv-1.23.0/src/unix/core.c:370:5 (libuv.so+0x19ecc)
 #18 cass::LoopThread::on_run_internal(void*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20221019013514-1aec6dda08-centos7-x86_64-clang15/src/cassandra-cpp-driver-2.9.0-yb-   13/src/loop_thread.hpp:128:5 (libcassandra.so.2+0x21453d)
bmatican commented 1 year ago

@amitanandaiyer were you able to commit the fix above to our fork of the cpp driver? do we need to actually update our thirdparty to pick it up?