search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.99k stars 1.07k forks source link

[DocDB] Tablet splitting: TSAN race detected in SnapshotTxnTest.TruncateDuringShutdown #19393

Open arybochkin opened 1 year ago

arybochkin commented 1 year ago

Jira Link: DB-8187

Description

Rare failures are happening for the TSAN build for SnapshotTxnTest.TruncateDuringShutdown, source.

Log example:

WARNING: ThreadSanitizer: data race (pid=8551)
  Write of size 8 at 0x7b7000073a60 by thread T413 (mutexes: write M0):
    #0 yb::ByteBuffer<64ul>::operator=(yb::ByteBuffer<64ul>&&) ${BUILD_ROOT}/../../src/yb/util/byte_buffer.h:84:11 (libtablet.so+0x1857d1)
    #1 yb::dockv::KeyBytes::operator=(yb::dockv::KeyBytes&&) ${BUILD_ROOT}/../../src/yb/dockv/key_bytes.h:38:47 (libtablet.so+0x1857d1)
    #2 yb::docdb::KeyBounds::operator=(yb::docdb::KeyBounds&&) ${BUILD_ROOT}/../../src/yb/docdb/key_bounds.h:31:8 (libtablet.so+0x1857d1)
    #3 yb::tablet::Tablet::CompleteShutdownRocksDBs(yb::tablet::TabletScopedRWOperationPauses const&) ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:1207:15 (libtablet.so+0x1857d1)
    #4 yb::tablet::Tablet::Truncate(yb::tablet::TruncateOperation*) ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:3155:27 (libtablet.so+0x1a2340)
...
  Previous read of size 8 at 0x7b7000073a60 by thread T390:
    #0 yb::ByteBuffer<64ul>::empty() const ${BUILD_ROOT}/../../src/yb/util/byte_buffer.h:103:12 (libtablet.so+0x1ab14b)
    #1 yb::dockv::KeyBytes::empty() const ${BUILD_ROOT}/../../src/yb/dockv/key_bytes.h:55:18 (libtablet.so+0x1ab14b)
    #2 yb::docdb::KeyBounds::IsInitialized() const ${BUILD_ROOT}/../../src/yb/docdb/key_bounds.h:46:19 (libtablet.so+0x1ab14b)
    #3 yb::tablet::Tablet::StillHasOrphanedPostSplitDataAbortable() ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:3417:23 (libtablet.so+0x1ab14b)
    #4 yb::tablet::Tablet::TriggerPostSplitCompactionIfNeeded() ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:3963:8 (libtablet.so+0x1ab14b)
    #5 yb::tserver::TSTabletManager::OpenTablet(scoped_refptr<yb::tablet::RaftGroupMetadata> const&, scoped_refptr<yb::tserver::TransitionInProgressDeleter> const&)

Warning: Please confirm that this issue does not contain any sensitive information

arybochkin commented 1 year ago

Also deadlock was detected during a jenkins run for another issue revision:

==================
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=16837)
  Cycle in lock order graph: M0 (0x7b74002f0c80) => M1 (0x7b74002f08b8) => M0

  Mutex M1 acquired here while holding mutex M0 in thread T382:
    #0 pthread_mutex_lock /opt/yb-build/llvm/yb-llvm-v16.0.6-yb-2-1687901233-bdefb807-centos7-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1367:3 (snapshot-txn-test+0x9c0eb)
    #1 std::mutex::lock() <null> (libc++.so.1+0xa2049)
    #2 std::lock_guard<std::mutex>::lock_guard[abi:v160006](std::mutex&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__mutex_base:94:27 (libtablet.so+0x27e13d)
    #3 yb::tablet::TransactionParticipant::Impl::LoadTransaction(yb::TransactionMetadata&&, yb::tablet::TransactionalBatchData&&, yb::OneWayBitmap&&, yb::tablet::ApplyStateWithCommitHt const*) ${BUILD_ROOT}/../../src/yb/tablet/transaction_participant.cc:1600:21 (libtablet.so+0x27e13d)
    #4 non-virtual thunk to yb::tablet::TransactionParticipant::Impl::LoadTransaction(yb::TransactionMetadata&&, yb::tablet::TransactionalBatchData&&, yb::OneWayBitmap&&, yb::tablet::ApplyStateWithCommitHt const*) ${BUILD_ROOT}/../../src/yb/tablet/transaction_participant.cc (libtablet.so+0x27e970)
    #5 yb::tablet::TransactionLoader::Executor::LoadTransaction(yb::StronglyTypedUuid<yb::TransactionId_Tag> const&) ${BUILD_ROOT}/../../src/yb/tablet/transaction_loader.cc:252:17 (libtablet.so+0x266178)
    #6 yb::tablet::TransactionLoader::Executor::LoadTransactions() ${BUILD_ROOT}/../../src/yb/tablet/transaction_loader.cc:126:9 (libtablet.so+0x263d22)
    #7 yb::tablet::TransactionLoader::Executor::Execute() ${BUILD_ROOT}/../../src/yb/tablet/transaction_loader.cc:99:14 (libtablet.so+0x262226)
    #8 decltype(*std::declval<yb::tablet::TransactionLoader::Executor*&>().*std::declval<void (yb::tablet::TransactionLoader::Executor::*&)()>()()) std::__invoke[abi:v160006]<void (yb::tablet::TransactionLoader::Executor::*&)(), yb::tablet::TransactionLoader::Executor*&, void>(void (yb::tablet::TransactionLoader::Executor::*&)(), yb::tablet::TransactionLoader::Executor*&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:359:23 (libtablet.so+0x2624cb)
    #9 std::__bind_return<void (yb::tablet::TransactionLoader::Executor::*)(), std::tuple<yb::tablet::TransactionLoader::Executor*>, std::tuple<>, __is_valid_bind_return<void (yb::tablet::TransactionLoader::Executor::*)(), std::tuple<yb::tablet::TransactionLoader::Executor*>, std::tuple<>>::value>::type std::__apply_functor[abi:v160006]<void (yb::tablet::TransactionLoader::Executor::*)(), std::tuple<yb::tablet::TransactionLoader::Executor*>, 0ul, std::tuple<>>(void (yb::tablet::TransactionLoader::Executor::*&)(), std::tuple<yb::tablet::TransactionLoader::Executor*>&, std::__tuple_indices<0ul>, std::tuple<>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:263:12 (libtablet.so+0x2624cb)
    #10 std::__bind_return<void (yb::tablet::TransactionLoader::Executor::*)(), std::tuple<yb::tablet::TransactionLoader::Executor*>, std::tuple<>, __is_valid_bind_return<void (yb::tablet::TransactionLoader::Executor::*)(), std::tuple<yb::tablet::TransactionLoader::Executor*>, std::tuple<>>::value>::type std::__bind<void (yb::tablet::TransactionLoader::Executor::* const&)(), yb::tablet::TransactionLoader::Executor* const&>::operator()[abi:v160006]<>() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:295:20 (libtablet.so+0x2624cb)
    #11 decltype(std::declval<std::__bind<void (yb::tablet::TransactionLoader::Executor::* const&)(), yb::tablet::TransactionLoader::Executor* const&>&>()()) std::__invoke[abi:v160006]<std::__bind<void (yb::tablet::TransactionLoader::Executor::* const&)(), yb::tablet::TransactionLoader::Executor* const&>&>(std::__bind<void (yb::tablet::TransactionLoader::Executor::* const&)(), yb::tablet::TransactionLoader::Executor* const&>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:394:23 (libtablet.so+0x2624cb)
    #12 void std::__invoke_void_return_wrapper<void, true>::__call<std::__bind<void (yb::tablet::TransactionLoader::Executor::* const&)(), yb::tablet::TransactionLoader::Executor* const&>&>(std::__bind<void (yb::tablet::TransactionLoader::Executor::* const&)(), yb::tablet::TransactionLoader::Executor* const&>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:487:9 (libtablet.so+0x2624cb)
    #13 std::__function::__alloc_func<std::__bind<void (yb::tablet::TransactionLoader::Executor::* const&)(), yb::tablet::TransactionLoader::Executor* const&>, std::allocator<std::__bind<void (yb::tablet::TransactionLoader::Executor::* const&)(), yb::tablet::TransactionLoader::Executor* const&>>, void ()>::operator()[abi:v160006]() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:185:16 (libtablet.so+0x2624cb)
    #14 std::__function::__func<std::__bind<void (yb::tablet::TransactionLoader::Executor::* const&)(), yb::tablet::TransactionLoader::Executor* const&>, std::allocator<std::__bind<void (yb::tablet::TransactionLoader::Executor::* const&)(), yb::tablet::TransactionLoader::Executor* const&>>, void ()>::operator()() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:356:12 (libtablet.so+0x2624cb)
    #15 std::__function::__value_func<void ()>::operator()[abi:v160006]() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:510:16 (libyb_util.so+0x3b37f2)
    #16 std::function<void ()>::operator()() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:1156:12 (libyb_util.so+0x3b37f2)
    #17 yb::Thread::SuperviseThread(void*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:853:3 (libyb_util.so+0x3b37f2)

  Mutex M0 previously acquired by the same thread here:
    #0 pthread_mutex_lock /opt/yb-build/llvm/yb-llvm-v16.0.6-yb-2-1687901233-bdefb807-centos7-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1367:3 (snapshot-txn-test+0x9c0eb)
    #1 std::mutex::lock() <null> (libc++.so.1+0xa2049)
    #2 std::lock_guard<std::mutex>::lock_guard[abi:v160006](std::mutex&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__mutex_base:94:27 (libtablet.so+0x265f7c)
    #3 yb::tablet::TransactionLoader::Executor::LoadTransaction(yb::StronglyTypedUuid<yb::TransactionId_Tag> const&) ${BUILD_ROOT}/../../src/yb/tablet/transaction_loader.cc:250:23 (libtablet.so+0x265f7c)
    #4 yb::tablet::TransactionLoader::Executor::LoadTransactions() ${BUILD_ROOT}/../../src/yb/tablet/transaction_loader.cc:126:9 (libtablet.so+0x263d22)
    #5 yb::tablet::TransactionLoader::Executor::Execute() ${BUILD_ROOT}/../../src/yb/tablet/transaction_loader.cc:99:14 (libtablet.so+0x262226)
    #6 decltype(*std::declval<yb::tablet::TransactionLoader::Executor*&>().*std::declval<void (yb::tablet::TransactionLoader::Executor::*&)()>()()) std::__invoke[abi:v160006]<void (yb::tablet::TransactionLoader::Executor::*&)(), yb::tablet::TransactionLoader::Executor*&, void>(void (yb::tablet::TransactionLoader::Executor::*&)(), yb::tablet::TransactionLoader::Executor*&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:359:23 (libtablet.so+0x2624cb)
    #7 std::__bind_return<void (yb::tablet::TransactionLoader::Executor::*)(), std::tuple<yb::tablet::TransactionLoader::Executor*>, std::tuple<>, __is_valid_bind_return<void (yb::tablet::TransactionLoader::Executor::*)(), std::tuple<yb::tablet::TransactionLoader::Executor*>, std::tuple<>>::value>::type std::__apply_functor[abi:v160006]<void (yb::tablet::TransactionLoader::Executor::*)(), std::tuple<yb::tablet::TransactionLoader::Executor*>, 0ul, std::tuple<>>(void (yb::tablet::TransactionLoader::Executor::*&)(), std::tuple<yb::tablet::TransactionLoader::Executor*>&, std::__tuple_indices<0ul>, std::tuple<>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:263:12 (libtablet.so+0x2624cb)
    #8 std::__bind_return<void (yb::tablet::TransactionLoader::Executor::*)(), std::tuple<yb::tablet::TransactionLoader::Executor*>, std::tuple<>, __is_valid_bind_return<void (yb::tablet::TransactionLoader::Executor::*)(), std::tuple<yb::tablet::TransactionLoader::Executor*>, std::tuple<>>::value>::type std::__bind<void (yb::tablet::TransactionLoader::Executor::* const&)(), yb::tablet::TransactionLoader::Executor* const&>::operator()[abi:v160006]<>() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:295:20 (libtablet.so+0x2624cb)
    #9 decltype(std::declval<std::__bind<void (yb::tablet::TransactionLoader::Executor::* const&)(), yb::tablet::TransactionLoader::Executor* const&>&>()()) std::__invoke[abi:v160006]<std::__bind<void (yb::tablet::TransactionLoader::Executor::* const&)(), yb::tablet::TransactionLoader::Executor* const&>&>(std::__bind<void (yb::tablet::TransactionLoader::Executor::* const&)(), yb::tablet::TransactionLoader::Executor* const&>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:394:23 (libtablet.so+0x2624cb)
    #10 void std::__invoke_void_return_wrapper<void, true>::__call<std::__bind<void (yb::tablet::TransactionLoader::Executor::* const&)(), yb::tablet::TransactionLoader::Executor* const&>&>(std::__bind<void (yb::tablet::TransactionLoader::Executor::* const&)(), yb::tablet::TransactionLoader::Executor* const&>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:487:9 (libtablet.so+0x2624cb)
    #11 std::__function::__alloc_func<std::__bind<void (yb::tablet::TransactionLoader::Executor::* const&)(), yb::tablet::TransactionLoader::Executor* const&>, std::allocator<std::__bind<void (yb::tablet::TransactionLoader::Executor::* const&)(), yb::tablet::TransactionLoader::Executor* const&>>, void ()>::operator()[abi:v160006]() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:185:16 (libtablet.so+0x2624cb)
    #12 std::__function::__func<std::__bind<void (yb::tablet::TransactionLoader::Executor::* const&)(), yb::tablet::TransactionLoader::Executor* const&>, std::allocator<std::__bind<void (yb::tablet::TransactionLoader::Executor::* const&)(), yb::tablet::TransactionLoader::Executor* const&>>, void ()>::operator()() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:356:12 (libtablet.so+0x2624cb)
    #13 std::__function::__value_func<void ()>::operator()[abi:v160006]() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:510:16 (libyb_util.so+0x3b37f2)
    #14 std::function<void ()>::operator()() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:1156:12 (libyb_util.so+0x3b37f2)
    #15 yb::Thread::SuperviseThread(void*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:853:3 (libyb_util.so+0x3b37f2)

  Mutex M0 acquired here while holding mutex M1 in thread T374:
    #0 pthread_mutex_lock /opt/yb-build/llvm/yb-llvm-v16.0.6-yb-2-1687901233-bdefb807-centos7-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1367:3 (snapshot-txn-test+0x9c0eb)
    #1 std::mutex::lock() <null> (libc++.so.1+0xa2049)
    #2 std::lock_guard<std::mutex>::lock_guard[abi:v160006](std::mutex&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__mutex_base:94:27 (libtablet.so+0x26157a)
    #3 yb::tablet::TransactionLoader::GetPendingApply(yb::StronglyTypedUuid<yb::TransactionId_Tag> const&) const ${BUILD_ROOT}/../../src/yb/tablet/transaction_loader.cc:421:19 (libtablet.so+0x26157a)
    #4 yb::tablet::TransactionParticipant::Impl::Add(yb::TransactionMetadata const&) ${BUILD_ROOT}/../../src/yb/tablet/transaction_participant.cc:298:36 (libtablet.so+0x270290)
    #5 yb::tablet::TransactionParticipant::Add(yb::TransactionMetadata const&) ${BUILD_ROOT}/../../src/yb/tablet/transaction_participant.cc:2029:17 (libtablet.so+0x26cd83)
    #6 yb::tablet::Tablet::WriteTransactionalBatch(long, yb::docdb::LWKeyValueWriteBatchPB const&, yb::HybridTime, rocksdb::UserFrontiers const*) ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:1355:50 (libtablet.so+0x188bc8)
    #7 yb::tablet::Tablet::ApplyKeyValueRowOperations(long, yb::docdb::LWKeyValueWriteBatchPB const&, rocksdb::UserFrontiersBase<yb::docdb::ConsensusFrontier>*, yb::HybridTime, yb::HybridTime, yb::StronglyTypedBool<yb::tablet::AlreadyAppliedToRegularDB_Tag>) ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:1411:5 (libtablet.so+0x18861c)
    #8 yb::tablet::Tablet::ApplyOperation(yb::tablet::Operation const&, long, yb::docdb::LWKeyValueWriteBatchPB const&, yb::StronglyTypedBool<yb::tablet::AlreadyAppliedToRegularDB_Tag>) ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:1338:10 (libtablet.so+0x18810d)
    #9 yb::tablet::Tablet::ApplyRowOperations(yb::tablet::WriteOperation*, yb::StronglyTypedBool<yb::tablet::AlreadyAppliedToRegularDB_Tag>) ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:1305:10 (libtablet.so+0x187adb)
    #10 yb::tablet::WriteOperation::DoReplicated(long, yb::Status*) ${BUILD_ROOT}/../../src/yb/tablet/operations/write_operation.cc:88:52 (libtablet.so+0x2e07de)
    #11 yb::tablet::Operation::Replicated(long, yb::StronglyTypedBool<yb::tablet::WasPending_Tag>) ${BUILD_ROOT}/../../src/yb/tablet/operations/operation.cc:82:3 (libtablet.so+0x2a86a4)
    #12 yb::tablet::OperationDriver::ApplyTask(long, std::vector<yb::OpId, std::allocator<yb::OpId>>*) ${BUILD_ROOT}/../../src/yb/tablet/operations/operation_driver.cc:404:31 (libtablet.so+0x2c0a25)
    #13 yb::tablet::OperationDriver::ReplicationFinished(yb::Status const&, long, std::vector<yb::OpId, std::allocator<yb::OpId>>*) ${BUILD_ROOT}/../../src/yb/tablet/operations/operation_driver.cc:362:5 (libtablet.so+0x2c03a3)
    #14 yb::consensus::ConsensusRound::NotifyReplicationFinished(yb::Status const&, long, std::vector<yb::OpId, std::allocator<yb::OpId>>*) ${BUILD_ROOT}/../../src/yb/consensus/consensus_round.cc:47:14 (libconsensus.so+0xcbebf)
    #15 yb::consensus::ReplicaState::NotifyReplicationFinishedUnlocked(scoped_refptr<yb::consensus::ConsensusRound> const&, yb::Status const&, long, std::vector<yb::OpId, std::allocator<yb::OpId>>*) ${BUILD_ROOT}/../../src/yb/consensus/replica_state.cc:1462:10 (libconsensus.so+0x138a29)
    #16 yb::consensus::ReplicaState::ApplyPendingOperationsUnlocked(yb::OpId const&, yb::StronglyTypedBool<yb::consensus::CouldStop_Tag>) ${BUILD_ROOT}/../../src/yb/consensus/replica_state.cc:959:5 (libconsensus.so+0x138a29)
    #17 yb::consensus::ReplicaState::AdvanceCommittedOpIdUnlocked(yb::OpId const&, yb::StronglyTypedBool<yb::consensus::CouldStop_Tag>) ${BUILD_ROOT}/../../src/yb/consensus/replica_state.cc:872:17 (libconsensus.so+0x1378c2)
    #18 yb::consensus::RaftConsensus::MarkOperationsAsCommittedUnlocked(yb::consensus::LWConsensusRequestPB const&, yb::consensus::RaftConsensus::LeaderRequest const&, yb::OpId) ${BUILD_ROOT}/../../src/yb/consensus/raft_consensus.cc:2247:33 (libconsensus.so+0x109871)
    #19 yb::consensus::RaftConsensus::UpdateReplica(std::shared_ptr<yb::consensus::LWConsensusRequestPB> const&, yb::consensus::LWConsensusResponsePB*) ${BUILD_ROOT}/../../src/yb/consensus/raft_consensus.cc:2004:3 (libconsensus.so+0x1045ab)
    #20 yb::consensus::RaftConsensus::Update(std::shared_ptr<yb::consensus::LWConsensusRequestPB> const&, yb::consensus::LWConsensusResponsePB*, std::chrono::time_point<yb::CoarseMonoClock, std::chrono::duration<long long, std::ratio<1l, 1000000000l>>>) ${BUILD_ROOT}/../../src/yb/consensus/raft_consensus.cc:1570:14 (libconsensus.so+0x102c91)
    #21 yb::tserver::ConsensusServiceImpl::UpdateConsensus(yb::consensus::LWConsensusRequestPB const*, yb::consensus::LWConsensusResponsePB*, yb::rpc::RpcContext) ${BUILD_ROOT}/../../src/yb/tserver/tablet_service.cc:2199:25 (libtserver.so+0x2f8f75)
    #22 yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0::operator()(std::shared_ptr<yb::rpc::InboundCall>) const::'lambda'(yb::consensus::LWConsensusRequestPB const*, yb::consensus::LWConsensusResponsePB*, yb::rpc::RpcContext)::operator()(yb::consensus::LWConsensusRequestPB const*, yb::consensus::LWConsensusResponsePB*, yb::rpc::RpcContext) const ${BUILD_ROOT}/src/yb/consensus/consensus.service.cc:288:9 (libconsensus_proto.so+0xe08d6)
    #23 auto yb::rpc::HandleCall<yb::rpc::RpcCallLWParamsImpl<yb::consensus::LWConsensusRequestPB, yb::consensus::LWConsensusResponsePB>, yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0::operator()(std::shared_ptr<yb::rpc::InboundCall>) const::'lambda'(yb::consensus::LWConsensusRequestPB const*, yb::consensus::LWConsensusResponsePB*, yb::rpc::RpcContext)>(std::shared_ptr<yb::rpc::InboundCall>, yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0::operator()(std::shared_ptr<yb::rpc::InboundCall>) const::'lambda'(yb::consensus::LWConsensusRequestPB const*, yb::consensus::LWConsensusResponsePB*, yb::rpc::RpcContext)) ${BUILD_ROOT}/../../src/yb/rpc/local_call.h:122:7 (libconsensus_proto.so+0xe08d6)
    #24 yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0::operator()(std::shared_ptr<yb::rpc::InboundCall>) const ${BUILD_ROOT}/src/yb/consensus/consensus.service.cc:286:7 (libconsensus_proto.so+0xe08d6)
    #25 decltype(std::declval<yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0&>()(std::declval<std::shared_ptr<yb::rpc::InboundCall>>())) std::__invoke[abi:v160006]<yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0&, std::shared_ptr<yb::rpc::InboundCall>>(yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0&, std::shared_ptr<yb::rpc::InboundCall>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:394:23 (libconsensus_proto.so+0xe08d6)
    #26 void std::__invoke_void_return_wrapper<void, true>::__call<yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0&, std::shared_ptr<yb::rpc::InboundCall>>(yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0&, std::shared_ptr<yb::rpc::InboundCall>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:487:9 (libconsensus_proto.so+0xe08d6)
    #27 std::__function::__alloc_func<yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0, std::allocator<yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0>, void (std::shared_ptr<yb::rpc::InboundCall>)>::operator()[abi:v160006](std::shared_ptr<yb::rpc::InboundCall>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:185:16 (libconsensus_proto.so+0xe08d6)
    #28 std::__function::__func<yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0, std::allocator<yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0>, void (std::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::shared_ptr<yb::rpc::InboundCall>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:356:12 (libconsensus_proto.so+0xe08d6)
    #29 std::__function::__value_func<void (std::shared_ptr<yb::rpc::InboundCall>)>::operator()[abi:v160006](std::shared_ptr<yb::rpc::InboundCall>&&) const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:510:16 (libconsensus_proto.so+0xe0252)
    #30 std::function<void (std::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::shared_ptr<yb::rpc::InboundCall>) const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:1156:12 (libconsensus_proto.so+0xe0252)
    #31 yb::consensus::ConsensusServiceIf::Handle(std::shared_ptr<yb::rpc::InboundCall>) ${BUILD_ROOT}/src/yb/consensus/consensus.service.cc:271:3 (libconsensus_proto.so+0xe0252)
    #32 yb::rpc::ServicePoolImpl::Handle(std::shared_ptr<yb::rpc::InboundCall>) ${BUILD_ROOT}/../../src/yb/rpc/service_pool.cc:261:19 (libyrpc.so+0x197685)
    #33 yb::rpc::InboundCall::InboundCallTask::Run() ${BUILD_ROOT}/../../src/yb/rpc/inbound_call.cc:244:13 (libyrpc.so+0xf9039)
    #34 yb::rpc::(anonymous namespace)::Worker::Execute() ${BUILD_ROOT}/../../src/yb/rpc/thread_pool.cc:104:15 (libyrpc.so+0x1adcd4)
    #35 decltype(*std::declval<yb::rpc::(anonymous namespace)::Worker*&>().*std::declval<void (yb::rpc::(anonymous namespace)::Worker::*&)()>()()) std::__invoke[abi:v160006]<void (yb::rpc::(anonymous namespace)::Worker::*&)(), yb::rpc::(anonymous namespace)::Worker*&, void>(void (yb::rpc::(anonymous namespace)::Worker::*&)(), yb::rpc::(anonymous namespace)::Worker*&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:359:23 (libyrpc.so+0x1ae12b)
    #36 std::__bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::tuple<>, __is_valid_bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::tuple<>>::value>::type std::__apply_functor[abi:v160006]<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>, 0ul, std::tuple<>>(void (yb::rpc::(anonymous namespace)::Worker::*&)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>&, std::__tuple_indices<0ul>, std::tuple<>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:263:12 (libyrpc.so+0x1ae12b)
    #37 std::__bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::tuple<>, __is_valid_bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::tuple<>>::value>::type std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>::operator()[abi:v160006]<>() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:295:20 (libyrpc.so+0x1ae12b)
    #38 decltype(std::declval<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&>()()) std::__invoke[abi:v160006]<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&>(std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:394:23 (libyrpc.so+0x1ae12b)
    #39 void std::__invoke_void_return_wrapper<void, true>::__call<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&>(std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:487:9 (libyrpc.so+0x1ae12b)
    #40 std::__function::__alloc_func<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>, std::allocator<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>>, void ()>::operator()[abi:v160006]() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:185:16 (libyrpc.so+0x1ae12b)
    #41 std::__function::__func<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>, std::allocator<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>>, void ()>::operator()() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:356:12 (libyrpc.so+0x1ae12b)
    #42 std::__function::__value_func<void ()>::operator()[abi:v160006]() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:510:16 (libyb_util.so+0x3b37f2)
    #43 std::function<void ()>::operator()() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:1156:12 (libyb_util.so+0x3b37f2)
    #44 yb::Thread::SuperviseThread(void*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:853:3 (libyb_util.so+0x3b37f2)

  Mutex M1 previously acquired by the same thread here:
    #0 pthread_mutex_lock /opt/yb-build/llvm/yb-llvm-v16.0.6-yb-2-1687901233-bdefb807-centos7-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1367:3 (snapshot-txn-test+0x9c0eb)
    #1 std::mutex::lock() <null> (libc++.so.1+0xa2049)
    #2 std::lock_guard<std::mutex>::lock_guard[abi:v160006](std::mutex&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__mutex_base:94:27 (libtablet.so+0x26fb38)
    #3 yb::tablet::TransactionParticipant::Impl::Add(yb::TransactionMetadata const&) ${BUILD_ROOT}/../../src/yb/tablet/transaction_participant.cc:274:21 (libtablet.so+0x26fb38)
    #4 yb::tablet::TransactionParticipant::Add(yb::TransactionMetadata const&) ${BUILD_ROOT}/../../src/yb/tablet/transaction_participant.cc:2029:17 (libtablet.so+0x26cd83)
    #5 yb::tablet::Tablet::WriteTransactionalBatch(long, yb::docdb::LWKeyValueWriteBatchPB const&, yb::HybridTime, rocksdb::UserFrontiers const*) ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:1355:50 (libtablet.so+0x188bc8)
    #6 yb::tablet::Tablet::ApplyKeyValueRowOperations(long, yb::docdb::LWKeyValueWriteBatchPB const&, rocksdb::UserFrontiersBase<yb::docdb::ConsensusFrontier>*, yb::HybridTime, yb::HybridTime, yb::StronglyTypedBool<yb::tablet::AlreadyAppliedToRegularDB_Tag>) ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:1411:5 (libtablet.so+0x18861c)
    #7 yb::tablet::Tablet::ApplyOperation(yb::tablet::Operation const&, long, yb::docdb::LWKeyValueWriteBatchPB const&, yb::StronglyTypedBool<yb::tablet::AlreadyAppliedToRegularDB_Tag>) ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:1338:10 (libtablet.so+0x18810d)
    #8 yb::tablet::Tablet::ApplyRowOperations(yb::tablet::WriteOperation*, yb::StronglyTypedBool<yb::tablet::AlreadyAppliedToRegularDB_Tag>) ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:1305:10 (libtablet.so+0x187adb)
    #9 yb::tablet::WriteOperation::DoReplicated(long, yb::Status*) ${BUILD_ROOT}/../../src/yb/tablet/operations/write_operation.cc:88:52 (libtablet.so+0x2e07de)
    #10 yb::tablet::Operation::Replicated(long, yb::StronglyTypedBool<yb::tablet::WasPending_Tag>) ${BUILD_ROOT}/../../src/yb/tablet/operations/operation.cc:82:3 (libtablet.so+0x2a86a4)
    #11 yb::tablet::OperationDriver::ApplyTask(long, std::vector<yb::OpId, std::allocator<yb::OpId>>*) ${BUILD_ROOT}/../../src/yb/tablet/operations/operation_driver.cc:404:31 (libtablet.so+0x2c0a25)
    #12 yb::tablet::OperationDriver::ReplicationFinished(yb::Status const&, long, std::vector<yb::OpId, std::allocator<yb::OpId>>*) ${BUILD_ROOT}/../../src/yb/tablet/operations/operation_driver.cc:362:5 (libtablet.so+0x2c03a3)
    #13 yb::consensus::ConsensusRound::NotifyReplicationFinished(yb::Status const&, long, std::vector<yb::OpId, std::allocator<yb::OpId>>*) ${BUILD_ROOT}/../../src/yb/consensus/consensus_round.cc:47:14 (libconsensus.so+0xcbebf)
    #14 yb::consensus::ReplicaState::NotifyReplicationFinishedUnlocked(scoped_refptr<yb::consensus::ConsensusRound> const&, yb::Status const&, long, std::vector<yb::OpId, std::allocator<yb::OpId>>*) ${BUILD_ROOT}/../../src/yb/consensus/replica_state.cc:1462:10 (libconsensus.so+0x138a29)
    #15 yb::consensus::ReplicaState::ApplyPendingOperationsUnlocked(yb::OpId const&, yb::StronglyTypedBool<yb::consensus::CouldStop_Tag>) ${BUILD_ROOT}/../../src/yb/consensus/replica_state.cc:959:5 (libconsensus.so+0x138a29)
    #16 yb::consensus::ReplicaState::AdvanceCommittedOpIdUnlocked(yb::OpId const&, yb::StronglyTypedBool<yb::consensus::CouldStop_Tag>) ${BUILD_ROOT}/../../src/yb/consensus/replica_state.cc:872:17 (libconsensus.so+0x1378c2)
    #17 yb::consensus::RaftConsensus::MarkOperationsAsCommittedUnlocked(yb::consensus::LWConsensusRequestPB const&, yb::consensus::RaftConsensus::LeaderRequest const&, yb::OpId) ${BUILD_ROOT}/../../src/yb/consensus/raft_consensus.cc:2247:33 (libconsensus.so+0x109871)
    #18 yb::consensus::RaftConsensus::UpdateReplica(std::shared_ptr<yb::consensus::LWConsensusRequestPB> const&, yb::consensus::LWConsensusResponsePB*) ${BUILD_ROOT}/../../src/yb/consensus/raft_consensus.cc:2004:3 (libconsensus.so+0x1045ab)
    #19 yb::consensus::RaftConsensus::Update(std::shared_ptr<yb::consensus::LWConsensusRequestPB> const&, yb::consensus::LWConsensusResponsePB*, std::chrono::time_point<yb::CoarseMonoClock, std::chrono::duration<long long, std::ratio<1l, 1000000000l>>>) ${BUILD_ROOT}/../../src/yb/consensus/raft_consensus.cc:1570:14 (libconsensus.so+0x102c91)
    #20 yb::tserver::ConsensusServiceImpl::UpdateConsensus(yb::consensus::LWConsensusRequestPB const*, yb::consensus::LWConsensusResponsePB*, yb::rpc::RpcContext) ${BUILD_ROOT}/../../src/yb/tserver/tablet_service.cc:2199:25 (libtserver.so+0x2f8f75)
    #21 yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0::operator()(std::shared_ptr<yb::rpc::InboundCall>) const::'lambda'(yb::consensus::LWConsensusRequestPB const*, yb::consensus::LWConsensusResponsePB*, yb::rpc::RpcContext)::operator()(yb::consensus::LWConsensusRequestPB const*, yb::consensus::LWConsensusResponsePB*, yb::rpc::RpcContext) const ${BUILD_ROOT}/src/yb/consensus/consensus.service.cc:288:9 (libconsensus_proto.so+0xe08d6)
    #22 auto yb::rpc::HandleCall<yb::rpc::RpcCallLWParamsImpl<yb::consensus::LWConsensusRequestPB, yb::consensus::LWConsensusResponsePB>, yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0::operator()(std::shared_ptr<yb::rpc::InboundCall>) const::'lambda'(yb::consensus::LWConsensusRequestPB const*, yb::consensus::LWConsensusResponsePB*, yb::rpc::RpcContext)>(std::shared_ptr<yb::rpc::InboundCall>, yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0::operator()(std::shared_ptr<yb::rpc::InboundCall>) const::'lambda'(yb::consensus::LWConsensusRequestPB const*, yb::consensus::LWConsensusResponsePB*, yb::rpc::RpcContext)) ${BUILD_ROOT}/../../src/yb/rpc/local_call.h:122:7 (libconsensus_proto.so+0xe08d6)
    #23 yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0::operator()(std::shared_ptr<yb::rpc::InboundCall>) const ${BUILD_ROOT}/src/yb/consensus/consensus.service.cc:286:7 (libconsensus_proto.so+0xe08d6)
    #24 decltype(std::declval<yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0&>()(std::declval<std::shared_ptr<yb::rpc::InboundCall>>())) std::__invoke[abi:v160006]<yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0&, std::shared_ptr<yb::rpc::InboundCall>>(yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0&, std::shared_ptr<yb::rpc::InboundCall>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:394:23 (libconsensus_proto.so+0xe08d6)
    #25 void std::__invoke_void_return_wrapper<void, true>::__call<yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0&, std::shared_ptr<yb::rpc::InboundCall>>(yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0&, std::shared_ptr<yb::rpc::InboundCall>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:487:9 (libconsensus_proto.so+0xe08d6)
    #26 std::__function::__alloc_func<yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0, std::allocator<yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0>, void (std::shared_ptr<yb::rpc::InboundCall>)>::operator()[abi:v160006](std::shared_ptr<yb::rpc::InboundCall>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:185:16 (libconsensus_proto.so+0xe08d6)
    #27 std::__function::__func<yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0, std::allocator<yb::consensus::ConsensusServiceIf::InitMethods(scoped_refptr<yb::MetricEntity> const&)::$_0>, void (std::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::shared_ptr<yb::rpc::InboundCall>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:356:12 (libconsensus_proto.so+0xe08d6)
    #28 std::__function::__value_func<void (std::shared_ptr<yb::rpc::InboundCall>)>::operator()[abi:v160006](std::shared_ptr<yb::rpc::InboundCall>&&) const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:510:16 (libconsensus_proto.so+0xe0252)
    #29 std::function<void (std::shared_ptr<yb::rpc::InboundCall>)>::operator()(std::shared_ptr<yb::rpc::InboundCall>) const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:1156:12 (libconsensus_proto.so+0xe0252)
    #30 yb::consensus::ConsensusServiceIf::Handle(std::shared_ptr<yb::rpc::InboundCall>) ${BUILD_ROOT}/src/yb/consensus/consensus.service.cc:271:3 (libconsensus_proto.so+0xe0252)
    #31 yb::rpc::ServicePoolImpl::Handle(std::shared_ptr<yb::rpc::InboundCall>) ${BUILD_ROOT}/../../src/yb/rpc/service_pool.cc:261:19 (libyrpc.so+0x197685)
    #32 yb::rpc::InboundCall::InboundCallTask::Run() ${BUILD_ROOT}/../../src/yb/rpc/inbound_call.cc:244:13 (libyrpc.so+0xf9039)
    #33 yb::rpc::(anonymous namespace)::Worker::Execute() ${BUILD_ROOT}/../../src/yb/rpc/thread_pool.cc:104:15 (libyrpc.so+0x1adcd4)
    #34 decltype(*std::declval<yb::rpc::(anonymous namespace)::Worker*&>().*std::declval<void (yb::rpc::(anonymous namespace)::Worker::*&)()>()()) std::__invoke[abi:v160006]<void (yb::rpc::(anonymous namespace)::Worker::*&)(), yb::rpc::(anonymous namespace)::Worker*&, void>(void (yb::rpc::(anonymous namespace)::Worker::*&)(), yb::rpc::(anonymous namespace)::Worker*&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:359:23 (libyrpc.so+0x1ae12b)
    #35 std::__bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::tuple<>, __is_valid_bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::tuple<>>::value>::type std::__apply_functor[abi:v160006]<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>, 0ul, std::tuple<>>(void (yb::rpc::(anonymous namespace)::Worker::*&)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>&, std::__tuple_indices<0ul>, std::tuple<>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:263:12 (libyrpc.so+0x1ae12b)
    #36 std::__bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::tuple<>, __is_valid_bind_return<void (yb::rpc::(anonymous namespace)::Worker::*)(), std::tuple<yb::rpc::(anonymous namespace)::Worker*>, std::tuple<>>::value>::type std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>::operator()[abi:v160006]<>() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:295:20 (libyrpc.so+0x1ae12b)
    #37 decltype(std::declval<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&>()()) std::__invoke[abi:v160006]<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&>(std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:394:23 (libyrpc.so+0x1ae12b)
    #38 void std::__invoke_void_return_wrapper<void, true>::__call<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&>(std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:487:9 (libyrpc.so+0x1ae12b)
    #39 std::__function::__alloc_func<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>, std::allocator<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>>, void ()>::operator()[abi:v160006]() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:185:16 (libyrpc.so+0x1ae12b)
    #40 std::__function::__func<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>, std::allocator<std::__bind<void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&>>, void ()>::operator()() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:356:12 (libyrpc.so+0x1ae12b)
    #41 std::__function::__value_func<void ()>::operator()[abi:v160006]() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:510:16 (libyb_util.so+0x3b37f2)
    #42 std::function<void ()>::operator()() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:1156:12 (libyb_util.so+0x3b37f2)
    #43 yb::Thread::SuperviseThread(void*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:853:3 (libyb_util.so+0x3b37f2)

  Thread T382 'TransactionLoad' (tid=17228, running) created by thread T358 at:
    #0 pthread_create /opt/yb-build/llvm/yb-llvm-v16.0.6-yb-2-1687901233-bdefb807-centos7-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1048:3 (snapshot-txn-test+0x9a45b)
    #1 yb::Thread::StartThread(string const&, string const&, std::function<void ()>, scoped_refptr<yb::Thread>*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:770:15 (libyb_util.so+0x3b1cff)
    #2 yb::Status yb::Thread::Create<void (yb::tablet::TransactionLoader::Executor::*)(), yb::tablet::TransactionLoader::Executor*>(string const&, string const&, void (yb::tablet::TransactionLoader::Executor::* const&)(), yb::tablet::TransactionLoader::Executor* const&, scoped_refptr<yb::Thread>*) ${BUILD_ROOT}/../../src/yb/util/thread.h:165:12 (libtablet.so+0x261e2b)
    #3 yb::tablet::TransactionLoader::Executor::Start(yb::docdb::DocDB const&) ${BUILD_ROOT}/../../src/yb/tablet/transaction_loader.cc:77:5 (libtablet.so+0x261e2b)
    #4 yb::tablet::TransactionLoader::Start(yb::RWOperationCounter*, yb::docdb::DocDB const&) ${BUILD_ROOT}/../../src/yb/tablet/transaction_loader.cc:373:19 (libtablet.so+0x2610b6)
    #5 yb::tablet::TransactionParticipant::Impl::SetDB(yb::docdb::DocDB const&, yb::docdb::KeyBounds const*, yb::RWOperationCounter*) ${BUILD_ROOT}/../../src/yb/tablet/transaction_participant.cc:947:15 (libtablet.so+0x276977)
    #6 yb::tablet::TransactionParticipant::SetDB(yb::docdb::DocDB const&, yb::docdb::KeyBounds const*, yb::RWOperationCounter*) ${BUILD_ROOT}/../../src/yb/tablet/transaction_participant.cc:2114:17 (libtablet.so+0x26d553)
    #7 yb::tablet::Tablet::OpenKeyValueTablet() ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:884:5 (libtablet.so+0x17f6ef)
    #8 yb::tablet::Tablet::Open() ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:627:7 (libtablet.so+0x17cff9)
    #9 yb::tablet::TabletBootstrap::OpenTablet() ${BUILD_ROOT}/../../src/yb/tablet/tablet_bootstrap.cc:611:7 (libtablet.so+0x1e7101)
    #10 yb::tablet::TabletBootstrap::Bootstrap(std::shared_ptr<yb::tablet::Tablet>*, scoped_refptr<yb::log::Log>*, yb::consensus::ConsensusBootstrapInfo*) ${BUILD_ROOT}/../../src/yb/tablet/tablet_bootstrap.cc:495:29 (libtablet.so+0x1e464a)
    #11 yb::tablet::BootstrapTabletImpl(yb::tablet::BootstrapTabletData const&, std::shared_ptr<yb::tablet::Tablet>*, scoped_refptr<yb::log::Log>*, yb::consensus::ConsensusBootstrapInfo*) ${BUILD_ROOT}/../../src/yb/tablet/tablet_bootstrap.cc:1921:44 (libtablet.so+0x1e35d2)
    #12 yb::tablet::BootstrapTablet(yb::tablet::BootstrapTabletData const&, std::shared_ptr<yb::tablet::Tablet>*, scoped_refptr<yb::log::Log>*, yb::consensus::ConsensusBootstrapInfo*) ${BUILD_ROOT}/../../src/yb/tablet/tablet_bootstrap_if.cc:109:3 (libtablet.so+0x2079df)
    #13 yb::tserver::TSTabletManager::OpenTablet(scoped_refptr<yb::tablet::RaftGroupMetadata> const&, scoped_refptr<yb::tserver::TransitionInProgressDeleter> const&) ${BUILD_ROOT}/../../src/yb/tserver/ts_tablet_manager.cc:1683:9 (libtserver.so+0x34a4b6)
    #14 decltype(*std::declval<yb::tserver::TSTabletManager*&>().*std::declval<void (yb::tserver::TSTabletManager::*&)(scoped_refptr<yb::tablet::RaftGroupMetadata> const&, scoped_refptr<yb::tserver::TransitionInProgressDeleter> const&)>()(std::declval<scoped_refptr<yb::tablet::RaftGroupMetadata>&>(), std::declval<scoped_refptr<yb::tserver::TransitionInProgressDeleter>&>())) std::__invoke[abi:v160006]<void (yb::tserver::TSTabletManager::*&)(scoped_refptr<yb::tablet::RaftGroupMetadata> const&, scoped_refptr<yb::tserver::TransitionInProgressDeleter> const&), yb::tserver::TSTabletManager*&, scoped_refptr<yb::tablet::RaftGroupMetadata>&, scoped_refptr<yb::tserver::TransitionInProgressDeleter>&, void>(void (yb::tserver::TSTabletManager::*&)(scoped_refptr<yb::tablet::RaftGroupMetadata> const&, scoped_refptr<yb::tserver::TransitionInProgressDeleter> const&), yb::tserver::TSTabletManager*&, scoped_refptr<yb::tablet::RaftGroupMetadata>&, scoped_refptr<yb::tserver::TransitionInProgressDeleter>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:359:23 (libtserver.so+0x373896)
    #15 std::__bind_return<void (yb::tserver::TSTabletManager::*)(scoped_refptr<yb::tablet::RaftGroupMetadata> const&, scoped_refptr<yb::tserver::TransitionInProgressDeleter> const&), std::tuple<yb::tserver::TSTabletManager*, scoped_refptr<yb::tablet::RaftGroupMetadata>, scoped_refptr<yb::tserver::TransitionInProgressDeleter>>, std::tuple<>, __is_valid_bind_return<void (yb::tserver::TSTabletManager::*)(scoped_refptr<yb::tablet::RaftGroupMetadata> const&, scoped_refptr<yb::tserver::TransitionInProgressDeleter> const&), std::tuple<yb::tserver::TSTabletManager*, scoped_refptr<yb::tablet::RaftGroupMetadata>, scoped_refptr<yb::tserver::TransitionInProgressDeleter>>, std::tuple<>>::value>::type std::__apply_functor[abi:v160006]<void (yb::tserver::TSTabletManager::*)(scoped_refptr<yb::tablet::RaftGroupMetadata> const&, scoped_refptr<yb::tserver::TransitionInProgressDeleter> const&), std::tuple<yb::tserver::TSTabletManager*, scoped_refptr<yb::tablet::RaftGroupMetadata>, scoped_refptr<yb::tserver::TransitionInProgressDeleter>>, 0ul, 1ul, 2ul, std::tuple<>>(void (yb::tserver::TSTabletManager::*&)(scoped_refptr<yb::tablet::RaftGroupMetadata> const&, scoped_refptr<yb::tserver::TransitionInProgressDeleter> const&), std::tuple<yb::tserver::TSTabletManager*, scoped_refptr<yb::tablet::RaftGroupMetadata>, scoped_refptr<yb::tserver::TransitionInProgressDeleter>>&, std::__tuple_indices<0ul, 1ul, 2ul>, std::tuple<>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:263:12 (libtserver.so+0x373896)
    #16 std::__bind_return<void (yb::tserver::TSTabletManager::*)(scoped_refptr<yb::tablet::RaftGroupMetadata> const&, scoped_refptr<yb::tserver::TransitionInProgressDeleter> const&), std::tuple<yb::tserver::TSTabletManager*, scoped_refptr<yb::tablet::RaftGroupMetadata>, scoped_refptr<yb::tserver::TransitionInProgressDeleter>>, std::tuple<>, __is_valid_bind_return<void (yb::tserver::TSTabletManager::*)(scoped_refptr<yb::tablet::RaftGroupMetadata> const&, scoped_refptr<yb::tserver::TransitionInProgressDeleter> const&), std::tuple<yb::tserver::TSTabletManager*, scoped_refptr<yb::tablet::RaftGroupMetadata>, scoped_refptr<yb::tserver::TransitionInProgressDeleter>>, std::tuple<>>::value>::type std::__bind<void (yb::tserver::TSTabletManager::*)(scoped_refptr<yb::tablet::RaftGroupMetadata> const&, scoped_refptr<yb::tserver::TransitionInProgressDeleter> const&), yb::tserver::TSTabletManager*, scoped_refptr<yb::tablet::RaftGroupMetadata> const&, scoped_refptr<yb::tserver::TransitionInProgressDeleter>&>::operator()[abi:v160006]<>() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:295:20 (libtserver.so+0x373896)
    #17 yb::RunnableImpl<std::__bind<void (yb::tserver::TSTabletManager::*)(scoped_refptr<yb::tablet::RaftGroupMetadata> const&, scoped_refptr<yb::tserver::TransitionInProgressDeleter> const&), yb::tserver::TSTabletManager*, scoped_refptr<yb::tablet::RaftGroupMetadata> const&, scoped_refptr<yb::tserver::TransitionInProgressDeleter>&>>::Run() ${BUILD_ROOT}/../../src/yb/util/threadpool.h:78:5 (libtserver.so+0x373896)
    #18 yb::ThreadPool::DispatchThread(bool) ${BUILD_ROOT}/../../src/yb/util/threadpool.cc:612:22 (libyb_util.so+0x3c61cb)
    #19 decltype(*std::declval<yb::ThreadPool*&>().*std::declval<void (yb::ThreadPool::*&)(bool)>()(std::declval<bool&>())) std::__invoke[abi:v160006]<void (yb::ThreadPool::*&)(bool), yb::ThreadPool*&, bool&, void>(void (yb::ThreadPool::*&)(bool), yb::ThreadPool*&, bool&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:359:23 (libyb_util.so+0x3ce1c8)
    #20 std::__bind_return<void (yb::ThreadPool::*)(bool), std::tuple<yb::ThreadPool*, bool>, std::tuple<>, __is_valid_bind_return<void (yb::ThreadPool::*)(bool), std::tuple<yb::ThreadPool*, bool>, std::tuple<>>::value>::type std::__apply_functor[abi:v160006]<void (yb::ThreadPool::*)(bool), std::tuple<yb::ThreadPool*, bool>, 0ul, 1ul, std::tuple<>>(void (yb::ThreadPool::*&)(bool), std::tuple<yb::ThreadPool*, bool>&, std::__tuple_indices<0ul, 1ul>, std::tuple<>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:263:12 (libyb_util.so+0x3ce1c8)
    #21 std::__bind_return<void (yb::ThreadPool::*)(bool), std::tuple<yb::ThreadPool*, bool>, std::tuple<>, __is_valid_bind_return<void (yb::ThreadPool::*)(bool), std::tuple<yb::ThreadPool*, bool>, std::tuple<>>::value>::type std::__bind<void (yb::ThreadPool::* const&)(bool), yb::ThreadPool* const&, bool const&>::operator()[abi:v160006]<>() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:295:20 (libyb_util.so+0x3ce1c8)
    #22 decltype(std::declval<std::__bind<void (yb::ThreadPool::* const&)(bool), yb::ThreadPool* const&, bool const&>&>()()) std::__invoke[abi:v160006]<std::__bind<void (yb::ThreadPool::* const&)(bool), yb::ThreadPool* const&, bool const&>&>(std::__bind<void (yb::ThreadPool::* const&)(bool), yb::ThreadPool* const&, bool const&>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:394:23 (libyb_util.so+0x3ce1c8)
    #23 void std::__invoke_void_return_wrapper<void, true>::__call<std::__bind<void (yb::ThreadPool::* const&)(bool), yb::ThreadPool* const&, bool const&>&>(std::__bind<void (yb::ThreadPool::* const&)(bool), yb::ThreadPool* const&, bool const&>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:487:9 (libyb_util.so+0x3ce1c8)
    #24 std::__function::__alloc_func<std::__bind<void (yb::ThreadPool::* const&)(bool), yb::ThreadPool* const&, bool const&>, std::allocator<std::__bind<void (yb::ThreadPool::* const&)(bool), yb::ThreadPool* const&, bool const&>>, void ()>::operator()[abi:v160006]() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:185:16 (libyb_util.so+0x3ce1c8)
    #25 std::__function::__func<std::__bind<void (yb::ThreadPool::* const&)(bool), yb::ThreadPool* const&, bool const&>, std::allocator<std::__bind<void (yb::ThreadPool::* const&)(bool), yb::ThreadPool* const&, bool const&>>, void ()>::operator()() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:356:12 (libyb_util.so+0x3ce1c8)
    #26 std::__function::__value_func<void ()>::operator()[abi:v160006]() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:510:16 (libyb_util.so+0x3b37f2)
    #27 std::function<void ()>::operator()() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:1156:12 (libyb_util.so+0x3b37f2)
    #28 yb::Thread::SuperviseThread(void*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:853:3 (libyb_util.so+0x3b37f2)

  Thread T374 'rpc_tp_TabletSe' (tid=17220, running) created by thread T345 at:
    #0 pthread_create /opt/yb-build/llvm/yb-llvm-v16.0.6-yb-2-1687901233-bdefb807-centos7-x86_64-build/src/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1048:3 (snapshot-txn-test+0x9a45b)
    #1 yb::Thread::StartThread(string const&, string const&, std::function<void ()>, scoped_refptr<yb::Thread>*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:770:15 (libyb_util.so+0x3b1cff)
    #2 yb::Status yb::Thread::Create<void (yb::rpc::(anonymous namespace)::Worker::*)(), yb::rpc::(anonymous namespace)::Worker*>(string const&, string const&, void (yb::rpc::(anonymous namespace)::Worker::* const&)(), yb::rpc::(anonymous namespace)::Worker* const&, scoped_refptr<yb::Thread>*) ${BUILD_ROOT}/../../src/yb/util/thread.h:165:12 (libyrpc.so+0x1aeeb3)
    #3 yb::rpc::(anonymous namespace)::Worker::Start(unsigned long) ${BUILD_ROOT}/../../src/yb/rpc/thread_pool.cc:61:12 (libyrpc.so+0x1aeeb3)
    #4 yb::rpc::ThreadPool::Impl::Enqueue(yb::rpc::ThreadPoolTask*) ${BUILD_ROOT}/../../src/yb/rpc/thread_pool.cc:201:35 (libyrpc.so+0x1aeeb3)
    #5 yb::rpc::ThreadPool::Enqueue(yb::rpc::ThreadPoolTask*) ${BUILD_ROOT}/../../src/yb/rpc/thread_pool.cc:285:17 (libyrpc.so+0x1abfe8)
    #6 yb::rpc::ServicePoolImpl::Enqueue(std::shared_ptr<yb::rpc::InboundCall> const&) ${BUILD_ROOT}/../../src/yb/rpc/service_pool.cc:196:18 (libyrpc.so+0x197193)
    #7 yb::rpc::ServicePool::QueueInboundCall(std::shared_ptr<yb::rpc::InboundCall>) ${BUILD_ROOT}/../../src/yb/rpc/service_pool.cc:477:10 (libyrpc.so+0x19617a)
    #8 yb::rpc::Messenger::Handle(std::shared_ptr<yb::rpc::InboundCall>, yb::StronglyTypedBool<yb::rpc::Queue_Tag>) ${BUILD_ROOT}/../../src/yb/rpc/messenger.cc:516:21 (libyrpc.so+0x1072c1)
    #9 yb::rpc::YBInboundConnectionContext::HandleCall(std::shared_ptr<yb::rpc::Connection> const&, yb::rpc::CallData*) ${BUILD_ROOT}/../../src/yb/rpc/yb_rpc.cc:189:38 (libyrpc.so+0x1b235f)
    #10 non-virtual thunk to yb::rpc::YBInboundConnectionContext::HandleCall(std::shared_ptr<yb::rpc::Connection> const&, yb::rpc::CallData*) ${BUILD_ROOT}/../../src/yb/rpc/yb_rpc.cc (libyrpc.so+0x1b2ae4)
    #11 yb::rpc::BinaryCallParser::Parse(std::shared_ptr<yb::rpc::Connection> const&, boost::container::small_vector<iovec, 4ul, void, void> const&, yb::StronglyTypedBool<yb::rpc::ReadBufferFull_Tag>, std::shared_ptr<yb::MemTracker> const*) ${BUILD_ROOT}/../../src/yb/rpc/binary_call_parser.cc:163:7 (libyrpc.so+0xcc3c7)
    #12 yb::rpc::YBInboundConnectionContext::ProcessCalls(std::shared_ptr<yb::rpc::Connection> const&, boost::container::small_vector<iovec, 4ul, void, void> const&, yb::StronglyTypedBool<yb::rpc::ReadBufferFull_Tag>) ${BUILD_ROOT}/../../src/yb/rpc/yb_rpc.cc:148:19 (libyrpc.so+0x1b1e50)
    #13 yb::rpc::Connection::ProcessReceived(yb::StronglyTypedBool<yb::rpc::ReadBufferFull_Tag>) ${BUILD_ROOT}/../../src/yb/rpc/connection.cc:423:27 (libyrpc.so+0xe0747)
    #14 yb::rpc::RefinedStream::ProcessReceived(yb::StronglyTypedBool<yb::rpc::ReadBufferFull_Tag>) ${BUILD_ROOT}/../../src/yb/rpc/refined_stream.cc:152:24 (libyrpc.so+0x15174d)
    #15 yb::rpc::RefinedStream::ProcessReceived(yb::StronglyTypedBool<yb::rpc::ReadBufferFull_Tag>) ${BUILD_ROOT}/../../src/yb/rpc/refined_stream.cc:148:14 (libyrpc.so+0x151713)
    #16 non-virtual thunk to yb::rpc::RefinedStream::ProcessReceived(yb::StronglyTypedBool<yb::rpc::ReadBufferFull_Tag>) ${BUILD_ROOT}/../../src/yb/rpc/refined_stream.cc (libyrpc.so+0x15265d)
    #17 yb::rpc::TcpStream::TryProcessReceived() ${BUILD_ROOT}/../../src/yb/rpc/tcp_stream.cc:408:17 (libyrpc.so+0x1a8516)
    #18 yb::rpc::TcpStream::ReadHandler() ${BUILD_ROOT}/../../src/yb/rpc/tcp_stream.cc:334:31 (libyrpc.so+0x1a6b75)
    #19 yb::rpc::TcpStream::Handler(ev::io&, int) ${BUILD_ROOT}/../../src/yb/rpc/tcp_stream.cc:276:14 (libyrpc.so+0x1a637a)
    #20 void ev::base<ev_io, ev::io>::method_thunk<yb::rpc::TcpStream, &yb::rpc::TcpStream::Handler(ev::io&, int)>(ev_loop*, ev_io*, int) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/common/include/ev++.h:479:7 (libyrpc.so+0x1aab08)
    #21 ev_invoke_pending <null> (libev.so.4+0x896a)
    #22 decltype(*std::declval<yb::rpc::Reactor*&>().*std::declval<void (yb::rpc::Reactor::*&)()>()()) std::__invoke[abi:v160006]<void (yb::rpc::Reactor::*&)(), yb::rpc::Reactor*&, void>(void (yb::rpc::Reactor::*&)(), yb::rpc::Reactor*&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:359:23 (libyrpc.so+0x1467cb)
    #23 std::__bind_return<void (yb::rpc::Reactor::*)(), std::tuple<yb::rpc::Reactor*>, std::tuple<>, __is_valid_bind_return<void (yb::rpc::Reactor::*)(), std::tuple<yb::rpc::Reactor*>, std::tuple<>>::value>::type std::__apply_functor[abi:v160006]<void (yb::rpc::Reactor::*)(), std::tuple<yb::rpc::Reactor*>, 0ul, std::tuple<>>(void (yb::rpc::Reactor::*&)(), std::tuple<yb::rpc::Reactor*>&, std::__tuple_indices<0ul>, std::tuple<>&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:263:12 (libyrpc.so+0x1467cb)
    #24 std::__bind_return<void (yb::rpc::Reactor::*)(), std::tuple<yb::rpc::Reactor*>, std::tuple<>, __is_valid_bind_return<void (yb::rpc::Reactor::*)(), std::tuple<yb::rpc::Reactor*>, std::tuple<>>::value>::type std::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>::operator()[abi:v160006]<>() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/bind.h:295:20 (libyrpc.so+0x1467cb)
    #25 decltype(std::declval<std::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>&>()()) std::__invoke[abi:v160006]<std::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>&>(std::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:394:23 (libyrpc.so+0x1467cb)
    #26 void std::__invoke_void_return_wrapper<void, true>::__call<std::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>&>(std::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/invoke.h:487:9 (libyrpc.so+0x1467cb)
    #27 std::__function::__alloc_func<std::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>, std::allocator<std::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>>, void ()>::operator()[abi:v160006]() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:185:16 (libyrpc.so+0x1467cb)
    #28 std::__function::__func<std::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>, std::allocator<std::__bind<void (yb::rpc::Reactor::* const&)(), yb::rpc::Reactor* const&>>, void ()>::operator()() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:356:12 (libyrpc.so+0x1467cb)
    #29 std::__function::__value_func<void ()>::operator()[abi:v160006]() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:510:16 (libyb_util.so+0x3b37f2)
    #30 std::function<void ()>::operator()() const /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/include/c++/v1/__functional/function.h:1156:12 (libyb_util.so+0x3b37f2)
    #31 yb::Thread::SuperviseThread(void*) ${BUILD_ROOT}/../../src/yb/util/thread.cc:853:3 (libyb_util.so+0x3b37f2)

SUMMARY: ThreadSanitizer: lock-order-inversion (potential deadlock) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20230920205330-09031a51d0-centos7-x86_64-clang16/installed/tsan/libcxx/lib/libc++.so.1+0xa2049) in std::mutex::lock()
==================

Logs are attached to corresponding JIRA ticket.

arybochkin commented 2 weeks ago

TSAN failure as of 2024/10/03:

[ts-1] I1003 21:11:00.676199 300843 ts_tablet_manager.cc:2071] T {ql_client_test_table_tablet_id2} P {ts1_peer_id}: Time spent starting tablet: real 0.048s user 0.020s sys 0.004s
[ts-1] I1003 21:11:00.676311 300843 ts_tablet_manager.cc:3492] Deleted transition in progress opening tablet for tablet {ql_client_test_table_tablet_id2}
...
[ts-1] I1003 21:11:00.744583 300857 raft_consensus.cc:3634] T {ql_client_test_table_tablet_id2} P {ts1_peer_id} [term 1 FOLLOWER]: Advancing to term 2
[ts-1] I1003 21:11:00.744719 300857 consensus_meta.cc:361] T {ql_client_test_table_tablet_id2} P {ts1_peer_id}: Updating active role from FOLLOWER to FOLLOWER. Consensus state: current_term: 2 leader_uuid: "" config { opid_index: -1 peers { permanent_uuid: "{ts2_peer_id}" member_type: VOTER last_known_private_addr { host: "127.0.0.4" port: 26295 } last_known_broadcast_addr { host: "127.0.0.5.ip.yugabyte" port: 26295 } cloud_info { placement_cloud: "cloud1" placement_region: "rack2" placement_zone: "zone" } } peers { permanent_uuid: "{ts1_peer_id}" member_type: VOTER last_known_private_addr { host: "127.0.0.2" port: 25738 } last_known_broadcast_addr { host: "127.0.0.3.ip.yugabyte" port: 25738 } cloud_info { placement_cloud: "cloud1" placement_region: "rack1" placement_zone: "zone" } } peers { permanent_uuid: "{ts3_peer_id}" member_type: VOTER last_known_private_addr { host: "127.0.0.6" port: 24154 } last_known_broadcast_addr { host: "127.0.0.7.ip.yugabyte" port: 24154 } cloud_info { placement_cloud: "cloud2" placement_region: "rack3" placement_zone: "zone" } } }, has_pending_config = 0
[ts-1] I1003 21:11:00.745471 300493 messenger.cc:339] TEST: Restore incoming/outgoing connectivity with: 127.0.0.37
[ts-1] I1003 21:11:00.745548 300493 server_base.cc:791] Break 1 => 127.0.0.38
[ts-1] I1003 21:11:00.745599 300493 messenger.cc:281] TEST: Break incoming/outgoing connectivity with: 127.0.0.38
[ts-1] I1003 21:11:00.745618 300857 raft_consensus.cc:1754] T {ql_client_test_table_tablet_id2} P {ts1_peer_id} [term 2 FOLLOWER]: Refusing update from remote peer {ts2_peer_id}: Log matching property violated. Preceding OpId in replica: 1.10. Preceding OpId from leader: 2.14. (index mismatch)
[ts-1] I1003 21:11:00.745678 300857 raft_consensus.cc:1970] T {ql_client_test_table_tablet_id2} P {ts1_peer_id} [term 2 FOLLOWER]: Returning from UpdateConsensus because of error: error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: 1.10. Preceding OpId from leader: 2.14. (index mismatch)" source_file: "../../src/yb/consensus/raft_consensus.cc" source_line: 1752 errors: "00" } }
[m-1] W1003 21:11:00.742408 300706 master_heartbeat_service.cc:915] Stale heartbeat for Tablet {ql_client_test_table_tablet_id2} (table ql_client_test_table [id=8f4e93bc6ccc401ebe3881d0ddce7986]) on TS {ts1_peer_id} cstate=current_term: 1 config { opid_index: -1 peers  ...
...
[ts-2] I1003 21:11:00.750475 300708 consensus_queue.cc:1398] T {ql_client_test_table_tablet_id2} P {ts2_peer_id} [LEADER]: Connected to new peer: { peer: {ts1_peer_id} is_new: 0 last_received: 1.10 next_index: 11 last_known_committed_idx: 10 is_last_exchange_successful: 0 needs_remote_bootstrap: 0 member_type: VOTER num_sst_files: 0 last_applied: 1.10 last_successful_communication_time: 0ms ago}
[ts-1] I1003 21:11:00.751847 300857 consensus_meta.cc:361] T {ql_client_test_table_tablet_id2} P {ts1_peer_id}: Updating active role from FOLLOWER to FOLLOWER. Consensus state: current_term: 2 leader_uuid: "{ts2_peer_id}" config { opid_index: -1 peers ...
[ts-1] I1003 21:11:00.752099 300857 raft_consensus.cc:3519] T {ql_client_test_table_tablet_id2} P {ts1_peer_id} [term 2 FOLLOWER]: Calling mark dirty synchronously for reason code NEW_LEADER_ELECTED
[ts-1] I1003 21:11:00.754289 300857 replica_state.cc:1176] T {ql_client_test_table_tablet_id2} P {ts1_peer_id} [term 2 FOLLOWER]: Reset our lease: 43152.311s
[ts-1] I1003 21:11:00.764902 300857 raft_consensus.cc:3519] T {ql_client_test_table_tablet_id2} P {ts1_peer_id} [term 2 FOLLOWER]: Calling mark dirty synchronously for reason code FOLLOWER_NO_OP_COMPLETE
...
[ts-1] I1003 21:11:00.801199 300869 transaction_loader.cc:182] T {ql_client_test_table_tablet_id2} P {ts1_peer_id}: LoadTransactions done: loaded 4 transactions
[ts-1] I1003 21:11:00.801318 300869 transaction_participant.cc:1556] T {ql_client_test_table_tablet_id2} P {ts1_peer_id}: LoadFinished: starting transaction status resolution
[ts-1] I1003 21:11:00.801849 300857 db_impl.cc:805] T {ql_client_test_table_tablet_id2} P {ts1_peer_id} [R]: Shutting down RocksDB at: ${TEST_TMPDIR}/snapshot-txn-test.SnapshotTxnTest.TruncateDuringShutdown.1727989844071471-300493/minicluster-data/ts-1-root/yb-data/tserver/data/rocksdb/table-8f4e93bc6ccc401ebe3881d0ddce7986/tablet-{ql_client_test_table_tablet_id2}
[ts-1] I1003 21:11:00.801930 300857 db_impl.cc:805] T {ql_client_test_table_tablet_id2} P {ts1_peer_id} [I]: Shutting down RocksDB at: ${TEST_TMPDIR}/snapshot-txn-test.SnapshotTxnTest.TruncateDuringShutdown.1727989844071471-300493/minicluster-data/ts-1-root/yb-data/tserver/data/rocksdb/table-8f4e93bc6ccc401ebe3881d0ddce7986/tablet-{ql_client_test_table_tablet_id2}.intents
[ts-1] W1003 21:11:00.803030 300857 operation_counter.cc:176] Waiting for 1 pending operations to complete now for 0.000s
[ts-1] I1003 21:11:00.804442 300857 db_impl.cc:833] T {ql_client_test_table_tablet_id2} P {ts1_peer_id} [I]: Skipping mem table flush - disable_flush_on_shutdown_ is set
[ts-1] I1003 21:11:00.805970 300857 db_impl.cc:928] T {ql_client_test_table_tablet_id2} P {ts1_peer_id} [I]: Shutdown done
[ts-1] F1003 21:11:00.806439 300857 column_family.cc:464] Check failed: dummy_versions_->TEST_Next() == dummy_versions_ (0x7b7800160800 vs. 0x7b780015fc00) 
Fatal failure details written to ${BUILD_ROOT}/yb-test-logs/tests-client__snapshot-txn-test/SnapshotTxnTest_TruncateDuringShutdown.fatal_failure_details.2024-10-03T21_11_00.pid300493.txt
F20241003 21:11:00 ../../src/yb/rocksdb/db/column_family.cc:464] Check failed: dummy_versions_->TEST_Next() == dummy_versions_ (0x7b7800160800 vs. 0x7b780015fc00) 
${YB_SRC_ROOT}/src/yb/util/logging.cc:464:     @     0x7ff056f6b7e4  yb::LogFatalHandlerSink::send(int, char const*, char const*, int, tm const*, char const*, unsigned long)
    @     0x7ff056b99dc7 
    @     0x7ff056b9b0a2 
    @     0x7ff056b9f059 
${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/ostream:0:     @     0x7ff05aee6741  rocksdb::ColumnFamilyData::~ColumnFamilyData()
${YB_SRC_ROOT}/src/yb/rocksdb/db/column_family.cc:909:     @     0x7ff05aee841a  rocksdb::ColumnFamilySet::~ColumnFamilySet()
${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__memory/unique_ptr.h:68:     @     0x7ff05b01e37b  std::default_delete<rocksdb::ColumnFamilySet>::operator()[abi:ue170006](rocksdb::ColumnFamilySet*) const
${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__memory/unique_ptr.h:300:     @     0x7ff05b01e37b  unique_ptr<rocksdb::ColumnFamilySet, std::default_delete<rocksdb::ColumnFamilySet>>::reset[abi:ue170006](rocksdb::ColumnFamilySet*)
${YB_SRC_ROOT}/src/yb/rocksdb/db/version_set.cc:2245:     @     0x7ff05b01e37b  rocksdb::VersionSet::~VersionSet()
${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__memory/unique_ptr.h:68:     @     0x7ff05af3f7b7  std::default_delete<rocksdb::VersionSet>::operator()[abi:ue170006](rocksdb::VersionSet*) const
${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__memory/unique_ptr.h:300:     @     0x7ff05af3f7b7  unique_ptr<rocksdb::VersionSet, std::default_delete<rocksdb::VersionSet>>::reset[abi:ue170006](rocksdb::VersionSet*)
${YB_SRC_ROOT}/src/yb/rocksdb/db/db_impl.cc:920:     @     0x7ff05af3f7b7  rocksdb::DBImpl::~DBImpl()
${YB_SRC_ROOT}/src/yb/rocksdb/db/db_impl.cc:820:     @     0x7ff05af44e59  rocksdb::DBImpl::~DBImpl()
${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__memory/unique_ptr.h:68:     @     0x7ff05ba521d2  std::default_delete<rocksdb::DB>::operator()[abi:ue170006](rocksdb::DB*) const
${YB_THIRDPARTY_DIR}/installed/tsan/libcxx/include/c++/v1/__memory/unique_ptr.h:300:     @     0x7ff05ba521d2  unique_ptr<rocksdb::DB, std::default_delete<rocksdb::DB>>::reset[abi:ue170006](rocksdb::DB*)
${YB_SRC_ROOT}/src/yb/tablet/tablet.cc:1411:     @     0x7ff05ba521d2  yb::tablet::Tablet::CompleteShutdownRocksDBs(yb::tablet::TabletScopedRWOperationPauses const&)
${YB_SRC_ROOT}/src/yb/tablet/tablet.cc:3455:     @     0x7ff05ba6ff4a  yb::tablet::Tablet::Truncate(yb::tablet::TruncateOperation*)
${YB_SRC_ROOT}/src/yb/tablet/operations/truncate_operation.cc:55:     @     0x7ff05bba0175  yb::tablet::TruncateOperation::DoReplicated(long, yb::Status*)
${YB_SRC_ROOT}/src/yb/tablet/operations/operation.cc:82:     @     0x7ff05bb82a04  yb::tablet::Operation::Replicated(long, yb::StronglyTypedBool<yb::tablet::WasPending_Tag>)
...