search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.99k stars 1.07k forks source link

[DocDB] Core dump StopActiveTxnsPriorTo after restore with many tables #23948

Closed pilshchikov closed 1 month ago

pilshchikov commented 1 month ago

Jira Link: DB-12844

Description

Case cycle:

  1. Run cluster, 3 nodes, RF=3, m7g.large, 2 CPU, 8 GB RAM, 2.23.1.0-b146
  2. Run cycle 2.1. Create N tables 2.2. Run nemesis in background:
    • restart VM
    • restart master/tserver processes
    • partition network 2.3. Load data in all tables for 30 min 2.4. PITR restore 10 minute ago 2.5. Drop all tables

Cycle 1 - 800 tables Cycle 2 - 900 tables

On second cycle core dump happen

* thread #1, name = 'yb-tserver', stop reason = signal SIGSEGV
  * frame #0: 0x000055b7694b3cd7 yb-server`std::__1::__function::__func<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>), std::__1::allocator<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>)>, void (yb::Result<yb::TransactionStatusResult>)>::operator()(yb::Result<yb::TransactionStatusResult>&&) [inlined] unsigned long std::__1::__cxx_atomic_fetch_sub[abi:ue170006]<unsigned long>(__a=0x00000000000bd7c6, __delta=1, __order=acq_rel) at cxx_atomic_impl.h:464:10
    frame #1: 0x000055b7694b3cd7 yb-server`std::__1::__function::__func<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>), std::__1::allocator<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>)>, void (yb::Result<yb::TransactionStatusResult>)>::operator()(yb::Result<yb::TransactionStatusResult>&&) [inlined] std::__1::__atomic_base<unsigned long, true>::fetch_sub[abi:ue170006](this=0x00000000000bd7c6, __op=1, __m=acq_rel) at atomic_base.h:171:14
    frame #2: 0x000055b7694b3cd7 yb-server`std::__1::__function::__func<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>), std::__1::allocator<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>)>, void (yb::Result<yb::TransactionStatusResult>)>::operator()(yb::Result<yb::TransactionStatusResult>&&) [inlined] yb::intrusive_ptr_release(state=0x00000000000bd7c6) at status.cc:637:22
    frame #3: 0x000055b7694b3cd7 yb-server`std::__1::__function::__func<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>), std::__1::allocator<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>)>, void (yb::Result<yb::TransactionStatusResult>)>::operator()(yb::Result<yb::TransactionStatusResult>&&) [inlined] boost::intrusive_ptr<yb::Status::State>::~intrusive_ptr(this=<unavailable>) at intrusive_ptr.hpp:98:23
    frame #4: 0x000055b7694b3cd2 yb-server`std::__1::__function::__func<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>), std::__1::allocator<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>)>, void (yb::Result<yb::TransactionStatusResult>)>::operator()(yb::Result<yb::TransactionStatusResult>&&) [inlined] boost::intrusive_ptr<yb::Status::State>::operator=(this=0x00007f1ca38aead0, rhs=0x00007f1f7f667d18) at intrusive_ptr.hpp:154:9
    frame #5: 0x000055b7694b3cbc yb-server`std::__1::__function::__func<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>), std::__1::allocator<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>)>, void (yb::Result<yb::TransactionStatusResult>)>::operator()(yb::Result<yb::TransactionStatusResult>&&) [inlined] yb::Status::operator=(this=0x00007f1ca38aead0, (null)=0x00007f1f7f667d18) at status.h:98:7
    frame #6: 0x000055b7694b3cbc yb-server`std::__1::__function::__func<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>), std::__1::allocator<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>)>, void (yb::Result<yb::TransactionStatusResult>)>::operator()(yb::Result<yb::TransactionStatusResult>&&) at transaction_participant.cc:1322:33
    frame #7: 0x000055b7694b39ec yb-server`std::__1::__function::__func<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>), std::__1::allocator<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>)>, void (yb::Result<yb::TransactionStatusResult>)>::operator()(yb::Result<yb::TransactionStatusResult>&&) [inlined] decltype(__f=<unavailable>, __args=<unavailable>)::'lambda'(yb::Result<yb::TransactionStatusResult>)&>()(std::declval<yb::Result<yb::TransactionStatusResult>>())) std::__1::__invoke[abi:ue170006]<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>)&, yb::Result<yb::TransactionStatusResult>>(yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>)&, yb::Result<yb::TransactionStatusResult>&&) at invoke.h:340:25
    frame #8: 0x000055b7694b3954 yb-server`std::__1::__function::__func<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>), std::__1::allocator<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>)>, void (yb::Result<yb::TransactionStatusResult>)>::operator()(yb::Result<yb::TransactionStatusResult>&&) [inlined] void std::__1::__invoke_void_return_wrapper<void, true>::__call[abi:ue170006]<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(__args=<unavailable>, __args=<unavailable>)::'lambda'(yb::Result<yb::TransactionStatusResult>)&, yb::Result<yb::TransactionStatusResult>>(yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>)&, yb::Result<yb::TransactionStatusResult>&&) at invoke.h:415:5
    frame #9: 0x000055b7694b3954 yb-server`std::__1::__function::__func<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>), std::__1::allocator<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>)>, void (yb::Result<yb::TransactionStatusResult>)>::operator()(yb::Result<yb::TransactionStatusResult>&&) [inlined] std::__1::__function::__alloc_func<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>), std::__1::allocator<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>)>, void (yb::Result<yb::TransactionStatusResult>)>::operator(this=<unavailable>, __arg=<unavailable>)[abi:ue170006](yb::Result<yb::TransactionStatusResult>&&) at function.h:192:16
    frame #10: 0x000055b7694b3954 yb-server`std::__1::__function::__func<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>), std::__1::allocator<yb::tablet::TransactionParticipant::Impl::StopActiveTxnsPriorTo(yb::HybridTime, std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>, yb::StronglyTypedUuid<yb::TransactionId_Tag>*)::'lambda'(yb::Result<yb::TransactionStatusResult>)>, void (yb::Result<yb::TransactionStatusResult>)>::operator(this=0x000014bae4d0dac0, __arg=<unavailable>)(yb::Result<yb::TransactionStatusResult>&&) at function.h:363:12
    frame #11: 0x000055b7693d9f41 yb-server`yb::tablet::RunningTransaction::AbortReceived(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, yb::Status const&, yb::tserver::AbortTransactionResponsePB const&, std::__1::shared_ptr<yb::tablet::RunningTransaction> const&) [inlined] std::__1::__function::__value_func<void (yb::Result<yb::TransactionStatusResult>)>::operator(this=0x000014baf9e70940, __args=0x00007f1f7f667de0)[abi:ue170006](yb::Result<yb::TransactionStatusResult>&&) const at function.h:517:16
    frame #12: 0x000055b7693d9f27 yb-server`yb::tablet::RunningTransaction::AbortReceived(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, yb::Status const&, yb::tserver::AbortTransactionResponsePB const&, std::__1::shared_ptr<yb::tablet::RunningTransaction> const&) [inlined] std::__1::function<void (yb::Result<yb::TransactionStatusResult>)>::operator(this=0x000014baf9e70940, __arg=Result<yb::TransactionStatusResult> @ 0x00007f1f7f667de0)(yb::Result<yb::TransactionStatusResult>) const at function.h:1168:12
    frame #13: 0x000055b7693d9f27 yb-server`yb::tablet::RunningTransaction::AbortReceived(this=<unavailable>, status_tablet=<unavailable>, status=<unavailable>, response=<unavailable>, shared_self=<unavailable>) at running_transaction.cc:564:5
    frame #14: 0x000055b768272f4f yb-server`yb::client::(anonymous namespace)::TransactionRpcBase::Finished(this=0x000014bb6a694b60, status=<unavailable>) at transaction_rpc.cc:67:7
    frame #15: 0x000055b768273330 yb-server`std::__1::__function::__func<std::__1::__bind<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>, std::__1::allocator<std::__1::__bind<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>>, void ()>::operator()() [inlined] decltype(__f=<unavailable>, __a0=<unavailable>, __args=<unavailable>)::TransactionRpcBase*&>().*std::declval<void (yb::client::(anonymous namespace)::TransactionRpcBase::*&)(yb::Status const&)>()(std::declval<yb::Status::OK&>())) std::__1::__invoke[abi:ue170006]<void (yb::client::(anonymous namespace)::TransactionRpcBase::*&)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*&, yb::Status::OK&, void>(void (yb::client::(anonymous namespace)::TransactionRpcBase::*&)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*&, yb::Status::OK&) at invoke.h:308:25
    frame #16: 0x000055b768273311 yb-server`std::__1::__function::__func<std::__1::__bind<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>, std::__1::allocator<std::__1::__bind<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>>, void ()>::operator()() [inlined] std::__1::__bind_return<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), std::__1::tuple<yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>, std::__1::tuple<>, __is_valid_bind_return<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), std::__1::tuple<yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>, std::__1::tuple<>>::value>::type std::__1::__apply_functor[abi:ue170006]<void (__f=<unavailable>, __bound_args=<unavailable>, (null)=<unavailable>, __args=<unavailable>)::TransactionRpcBase::*)(yb::Status const&), std::__1::tuple<yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>, 0ul, 1ul, std::__1::tuple<>>(void (yb::client::(anonymous namespace)::TransactionRpcBase::*&)(yb::Status const&), std::__1::tuple<yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>&, std::__1::__tuple_indices<0ul, 1ul>, std::__1::tuple<>&&) at bind.h:260:12
    frame #17: 0x000055b768273311 yb-server`std::__1::__function::__func<std::__1::__bind<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>, std::__1::allocator<std::__1::__bind<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>>, void ()>::operator()() [inlined] std::__1::__bind_return<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), std::__1::tuple<yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>, std::__1::tuple<>, __is_valid_bind_return<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), std::__1::tuple<yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>, std::__1::tuple<>>::value>::type std::__1::__bind<void (this=<unavailable>)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>::operator()[abi:ue170006]<>() at bind.h:292:20
    frame #18: 0x000055b768273309 yb-server`std::__1::__function::__func<std::__1::__bind<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>, std::__1::allocator<std::__1::__bind<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>>, void ()>::operator()() [inlined] decltype(__f=<unavailable>)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>&>()()) std::__1::__invoke[abi:ue170006]<std::__1::__bind<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>&>(std::__1::__bind<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>&) at invoke.h:340:25
    frame #19: 0x000055b768273309 yb-server`std::__1::__function::__func<std::__1::__bind<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>, std::__1::allocator<std::__1::__bind<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>>, void ()>::operator()() [inlined] void std::__1::__invoke_void_return_wrapper<void, true>::__call[abi:ue170006]<std::__1::__bind<void (__args=<unavailable>)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>&>(std::__1::__bind<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>&) at invoke.h:415:5
    frame #20: 0x000055b768273309 yb-server`std::__1::__function::__func<std::__1::__bind<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>, std::__1::allocator<std::__1::__bind<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>>, void ()>::operator()() [inlined] std::__1::__function::__alloc_func<std::__1::__bind<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>, std::__1::allocator<std::__1::__bind<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>>, void ()>::operator(this=<unavailable>)[abi:ue170006]() at function.h:192:16
    frame #21: 0x000055b768273309 yb-server`std::__1::__function::__func<std::__1::__bind<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>, std::__1::allocator<std::__1::__bind<void (yb::client::(anonymous namespace)::TransactionRpcBase::*)(yb::Status const&), yb::client::(anonymous namespace)::TransactionRpcBase*, yb::Status::OK>>, void ()>::operator(this=<unavailable>)() at function.h:363:12
    frame #22: 0x000055b769243b4a yb-server`yb::rpc::OutboundCall::InvokeCallbackSync(std::__1::optional<std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>>) [inlined] std::__1::__function::__value_func<void ()>::operator(this=0x000014bb780b1890)[abi:ue170006]() const at function.h:517:16
    frame #23: 0x000055b769243b44 yb-server`yb::rpc::OutboundCall::InvokeCallbackSync(std::__1::optional<std::__1::chrono::time_point<yb::CoarseMonoClock, std::__1::chrono::duration<long long, std::__1::ratio<1l, 1000000000l>>>>) [inlined] std::__1::function<void ()>::operator(this=0x000014bb780b1890)() const at function.h:1168:12
    frame #24: 0x000055b769243b44 yb-server`yb::rpc::OutboundCall::InvokeCallbackSync(this=0x000014bb780b1800, now_optional=<unavailable>) at outbound_call.cc:468:3
    frame #25: 0x000055b769320ce3 yb-server`yb::rpc::(anonymous namespace)::Worker::Execute(this=0x000014bae66687e0) at thread_pool.cc:115:15
    frame #26: 0x000055b769b904b3 yb-server`yb::Thread::SuperviseThread(void*) [inlined] std::__1::__function::__value_func<void ()>::operator(this=0x000014bae4481a40)[abi:ue170006]() const at function.h:517:16
    frame #27: 0x000055b769b9049d yb-server`yb::Thread::SuperviseThread(void*) [inlined] std::__1::function<void ()>::operator(this=0x000014bae4481a40)() const at function.h:1168:12
    frame #28: 0x000055b769b9049d yb-server`yb::Thread::SuperviseThread(arg=0x000014bae44819e0) at thread.cc:866:3
    frame #29: 0x00007f278ab461ca libpthread.so.0`start_thread + 234
    frame #30: 0x00007f278ad97e73 libc.so.6`__clone + 67
  thread #2, stop reason = signal 0
    frame #0: 0x00007f278ab4c45c libpthread.so.0`pthread_cond_wait@@GLIBC_2.3.2 + 508
    frame #1: 0x000055b769320ba5 yb-server`yb::rpc::(anonymous namespace)::Worker::Execute() [inlined] yb::rpc::(anonymous namespace)::Worker::PopTask(this=0x000014bb04986770, task=0x00007f1e2bbc1068) at thread_pool.cc:144:13
    frame #2: 0x000055b76932073d yb-server`yb::rpc::(anonymous namespace)::Worker::Execute(this=0x000014bb04986770) at thread_pool.cc:114:11
    frame #3: 0x000055b769b904b3 yb-server`yb::Thread::SuperviseThread(void*) [inlined] std::__1::__function::__value_func<void ()>::operator(this=0x000014bae1929c80)[abi:ue170006]() const at function.h:517:16
    frame #4: 0x000055b769b9049d yb-server`yb::Thread::SuperviseThread(void*) [inlined] std::__1::function<void ()>::operator(this=0x000014bae1929c80)() const at function.h:1168:12
    frame #5: 0x000055b769b9049d yb-server`yb::Thread::SuperviseThread(arg=0x000014bae1929c20) at thread.cc:866:3
    frame #6: 0x00007f278ab461ca libpthread.so.0`start_thread + 234
    frame #7: 0x00007f278ad97e73 libc.so.6`__clone + 67
  thread #3, stop reason = signal 0
    frame #0: 0x00007f278ab4c45c libpthread.so.0`pthread_cond_wait@@GLIBC_2.3.2 + 508
    frame #1: 0x000055b7682bd60b yb-server`yb::TerminationMonitor::WaitForTermination() [inlined] void std::__1::condition_variable::wait<yb::TerminationMonitor::WaitForTermination()::$_0>(this=0x000014baffc07f40, __lk=<unavailable>, __pred=(unnamed class) @ rbx)::$_0) at condition_variable.h:148:5
    frame #2: 0x000055b7682bd5f3 yb-server`yb::TerminationMonitor::WaitForTermination(this=0x000014baffc07f10) at termination_monitor.cc:68:16
    frame #3: 0x000055b76983bfaf yb-server`yb::tserver::TabletServerMain(argc=<unavailable>, argv=<unavailable>) at tablet_server_main_impl.cc:377:24
    frame #4: 0x000055b767eabe54 yb-server`main(argc=3, argv=0x00007ffe27410f38) at master_main.cc:184:12
    frame #5: 0x00007f278ad98d85 libc.so.6`__libc_start_main + 229
    frame #6: 0x000055b767d6f02e yb-server`_start + 46
  thread #4, stop reason = signal 0

All logs in first comment in JIRA ticket

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information

rthallamko3 commented 1 month ago

Looks like a regression caused by fix for #23399 ?

amitanandaiyer commented 1 month ago

2.23.1.0-b146 does not contain D37883 / fix for https://github.com/yugabyte/yugabyte-db/issues/23399

I think it is worth testing on the latest master to see if D37883 fixes the issue automatically, as the stack trace seems to suggest some kind of an issue/race on the un-protected status prior to D37883.