search

baidu / braft

An industrial-grade C++ implementation of RAFT consensus algorithm based on brpc, widely used inside Baidu to build highly-available distributed systems.
Apache License 2.0
3.95k stars 881 forks source link

使用braft过程中遇到braft deadlock 线程堆栈如下 #309

Open cl079074282 opened 3 years ago

cl079074282 commented 3 years ago

Thread 26 (Thread 0x7fa51effd700 (LWP 6893)):

0 futex_abstimed_wait_cancelable (private=, abstime=0x7fa51effb6a0, clockid=, expected=0, futex_word=0x7fa524001048) at ../sysdeps/nptl/futex-internal.h:320

1 __pthread_cond_wait_common (abstime=0x7fa51effb6a0, clockid=, mutex=0x7fa524000ff8, cond=0x7fa524001020) at pthread_cond_wait.c:520

2 __pthread_cond_timedwait (cond=0x7fa524001020, mutex=0x7fa524000ff8, abstime=0x7fa51effb6a0) at pthread_cond_wait.c:656

3 0x00007fa584d00bf2 in gthread_cond_timedwait (cond=0x7fa524001020, mutex=0x7fa524000ff8, abs_timeout=0x7fa51effb6a0) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:872

4 0x00007fa584d0333d in std::condition_variable::wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fa524001020, lock=..., __atime=...) at /usr/include/c++/9/condition_variable:188

5 0x00007fa584d027e1 in std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fa524001020, lock=..., atime=...) at /usr/include/c++/9/condition_variable:121

6 0x00007fa584d01eac in std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x7fa524001020, lock=..., rtime=...) at /usr/include/c++/9/condition_variable:152

7 0x00007fa584cfe916 in lse::chunkserver::PageAlloter::<lambda()>::operator()(void) const (__closure=0x7fa524c5a0b8) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/page_allocator.cpp:45

8 0x00007fa584d007e8 in std::invoke_impl<void, lse::chunkserver::PageAlloter::Start()::<lambda()> >(std::__invoke_other, lse::chunkserver::PageAlloter::<lambda()> &&) (f=...) at /usr/include/c++/9/bits/invoke.h:60

9 0x00007fa584d0073b in std::invoke<lse::chunkserver::PageAlloter::Start()::<lambda()> >(lse::chunkserver::PageAlloter::<lambda()> &&) (fn=...) at /usr/include/c++/9/bits/invoke.h:95

10 0x00007fa584d00699 in std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x7fa524c5a0b8) at /usr/include/c++/9/thread:244

11 0x00007fa584d0061e in std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > >::operator()(void) (this=0x7fa524c5a0b8) at /usr/include/c++/9/thread:251

12 0x00007fa584d005e1 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > > >::_M_run(void) (this=0x7fa524c5a0b0) at /usr/include/c++/9/thread:195

13 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

14 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477

15 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 25 (Thread 0x7fa51f7fe700 (LWP 6892)):

0 0x00007fa5834253bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa51f7fc740, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa58342b047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa51f7fc740, remaining=remaining@entry=0x0) at nanosleep.c:27

2 0x00007fa58345d9bf in usleep (useconds=) at ../sysdeps/posix/usleep.c:32

3 0x00007fa58507c516 in lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher (this=0x7fa524c42250) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/l2pmap_mgr.h:734

4 0x00007fa5850ad5a3 in std::__invoke_impl<int, int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> (f=@0x7fa524c7af70: (int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(class lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey> const)) 0x7fa58507c3dc <lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher()>, t=@0x7fa524c7af68: 0x7fa524c42250) at /usr/include/c++/9/bits/invoke.h:73

5 0x00007fa5850a8211 in std::invoke<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> (fn=@0x7fa524c7af70: (int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(class lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey> const)) 0x7fa58507c3dc <lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher()>) at /usr/include/c++/9/bits/invoke.h:95

6 0x00007fa5850a344c in std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> >::_M_invoke<0ul, 1ul> (this=0x7fa524c7af68) at /usr/include/c++/9/thread:244

7 0x00007fa58509a098 in std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> >::operator() (this=0x7fa524c7af68) at /usr/include/c++/9/thread:251

8 0x00007fa585098d8f in std::thread::_State_impl<std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> > >::_M_run (this=0x7fa524c7af60) at /usr/include/c++/9/thread:195

9 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

10 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477

11 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 24 (Thread 0x7fa538ff9700 (LWP 6890)):

0 0x0000559a49f333ad in spdk_nvme_qpair_process_completions (qpair=qpair@entry=0x20000eca4d18, max_completions=max_completions@entry=0) at nvme_qpair.c:684

1 0x0000559a49f2e1cc in nvme_pcie_poll_group_process_completions (tgroup=0x559a4b01b390, completions_per_qpair=0, disconnected_qpair_cb=0x7fa584d048a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair, void)>) at nvme_pcie.c:2536

2 0x0000559a49f36cc9 in nvme_transport_poll_group_process_completions (tgroup=tgroup@entry=0x559a4b01b390, completions_per_qpair=completions_per_qpair@entry=0, disconnected_qpair_cb=disconnected_qpair_cb@entry=0x7fa584d048a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair, void)>) at nvme_transport.c:499

3 0x0000559a49f40c79 in spdk_nvme_poll_group_process_completions (group=, completions_per_qpair=0, disconnected_qpair_cb=0x7fa584d048a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair, void)>) at nvme_poll_group.c:127

4 0x00007fa584d07dfb in lse::chunkserver::SpdkNamespace::Complete (this=0x559a4b19d610) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/spdk.cpp:408

5 0x00007fa585079ddc in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::Running (this=0x559a4afcfcb0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1207

6 0x00007fa585074d24 in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}::operator()() const (this=0x559a4afcfcb0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1152

7 0x00007fa5850ad6be in std::invoke_impl<void, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}>(std::__invoke_other, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}&&) (f=...) at /usr/include/c++/9/bits/invoke.h:60

8 0x00007fa5850a83c8 in std::invoke<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}>(std::invoke_result&&, (lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}&&)...) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95

9 0x00007fa5850a34f9 in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7fa524c41a28) at /usr/include/c++/9/thread:244

10 0x00007fa58509a16e in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> >::operator()() (this=0x7fa524c41a28) at /usr/include/c++/9/thread:251

11 0x00007fa585098dfb in std::thread::_State_impl<std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> > >::_M_run() (this=0x7fa524c41a20) at /usr/include/c++/9/thread:195

12 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

13 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477

14 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 23 (Thread 0x7fa5397fa700 (LWP 6887)):

0 0x00007fa5834253bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa5397f86f0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa58342b047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa5397f86f0, remaining=remaining@entry=0x0) at nanosleep.c:27

2 0x00007fa58345d9bf in usleep (useconds=useconds@entry=1000000) at ../sysdeps/posix/usleep.c:32

3 0x00007fa583a4d178 in bthread_usleep (microseconds=microseconds@entry=1000000) at /home/leo/code/thirdparty/brpc/src/bthread/bthread.cpp:363

4 0x00007fa583b40862 in brpc::Server::RunUntilAskedToQuit (this=0x7fa524c5b280) at /home/leo/code/thirdparty/brpc/src/brpc/server.cpp:1693

5 0x00007fa585068779 in lse::chunkserver::Container::execute (this=0x559a4b01b750) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/container.cpp:182

6 0x00007fa5850858a0 in std::__invoke_impl<void, void (lse::chunkserver::Container::)(), lse::chunkserver::Container> (f=@0x559a4afbd7d0: (void (lse::chunkserver::Container::)(class lse::chunkserver::Container const)) 0x7fa585067818 <lse::chunkserver::Container::execute()>, t=@0x559a4afbd7c8: 0x559a4b01b750) at /usr/include/c++/9/bits/invoke.h:73

7 0x00007fa58507f616 in std::invoke<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> (fn=@0x559a4afbd7d0: (void (lse::chunkserver::Container::)(class lse::chunkserver::Container const)) 0x7fa585067818 <lse::chunkserver::Container::execute()>) at /usr/include/c++/9/bits/invoke.h:95

8 0x00007fa585077e9e in std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> >::_M_invoke<0ul, 1ul> (this=0x559a4afbd7c8) at /usr/include/c++/9/thread:244

9 0x00007fa58507430e in std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> >::operator() (this=0x559a4afbd7c8) at /usr/include/c++/9/thread:251

10 0x00007fa58506f2cf in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> > >::_M_run (this=0x559a4afbd7c0) at /usr/include/c++/9/thread:195

11 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

12 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477

13 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 22 (Thread 0x7fa539ffb700 (LWP 6886)):

0 futex_abstimed_wait_cancelable (private=, abstime=0x7fa539ff96a0, clockid=, expected=0, futex_word=0x7fa530001048) at ../sysdeps/nptl/futex-internal.h:320

1 __pthread_cond_wait_common (abstime=0x7fa539ff96a0, clockid=, mutex=0x7fa530000ff8, cond=0x7fa530001020) at pthread_cond_wait.c:520

2 __pthread_cond_timedwait (cond=0x7fa530001020, mutex=0x7fa530000ff8, abstime=0x7fa539ff96a0) at pthread_cond_wait.c:656

3 0x00007fa584d00bf2 in gthread_cond_timedwait (cond=0x7fa530001020, mutex=0x7fa530000ff8, abs_timeout=0x7fa539ff96a0) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:872

4 0x00007fa584d0333d in std::condition_variable::wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fa530001020, lock=..., __atime=...) at /usr/include/c++/9/condition_variable:188

5 0x00007fa584d027e1 in std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fa530001020, lock=..., atime=...) at /usr/include/c++/9/condition_variable:121

6 0x00007fa584d01eac in std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x7fa530001020, lock=..., rtime=...) at /usr/include/c++/9/condition_variable:152

7 0x00007fa584cfe916 in lse::chunkserver::PageAlloter::<lambda()>::operator()(void) const (__closure=0x7fa530c5a0b8) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/page_allocator.cpp:45

8 0x00007fa584d007e8 in std::invoke_impl<void, lse::chunkserver::PageAlloter::Start()::<lambda()> >(std::__invoke_other, lse::chunkserver::PageAlloter::<lambda()> &&) (f=...) at /usr/include/c++/9/bits/invoke.h:60

9 0x00007fa584d0073b in std::invoke<lse::chunkserver::PageAlloter::Start()::<lambda()> >(lse::chunkserver::PageAlloter::<lambda()> &&) (fn=...) at /usr/include/c++/9/bits/invoke.h:95

10 0x00007fa584d00699 in std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x7fa530c5a0b8) at /usr/include/c++/9/thread:244

11 0x00007fa584d0061e in std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > >::operator()(void) (this=0x7fa530c5a0b8) at /usr/include/c++/9/thread:251

12 0x00007fa584d005e1 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > > >::_M_run(void) (this=0x7fa530c5a0b0) at /usr/include/c++/9/thread:195

13 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

14 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477

15 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 21 (Thread 0x7fa53a7fc700 (LWP 6885)):

0 0x00007fa5834253bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa53a7fa740, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa58342b047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa53a7fa740, remaining=remaining@entry=0x0) at nanosleep.c:27

2 0x00007fa58345d9bf in usleep (useconds=) at ../sysdeps/posix/usleep.c:32

3 0x00007fa58507c516 in lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher (this=0x7fa530c42250) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/l2pmap_mgr.h:734

4 0x00007fa5850ad5a3 in std::__invoke_impl<int, int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> (f=@0x7fa530c7af70: (int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(class lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey> const)) 0x7fa58507c3dc <lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher()>, t=@0x7fa530c7af68: 0x7fa530c42250) at /usr/include/c++/9/bits/invoke.h:73

5 0x00007fa5850a8211 in std::invoke<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> (fn=@0x7fa530c7af70: (int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(class lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey> const)) 0x7fa58507c3dc <lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher()>) at /usr/include/c++/9/bits/invoke.h:95

6 0x00007fa5850a344c in std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> >::_M_invoke<0ul, 1ul> (this=0x7fa530c7af68) at /usr/include/c++/9/thread:244

7 0x00007fa58509a098 in std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> >::operator() (this=0x7fa530c7af68) at /usr/include/c++/9/thread:251

8 0x00007fa585098d8f in std::thread::_State_impl<std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> > >::_M_run (this=0x7fa530c7af60) at /usr/include/c++/9/thread:195

9 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

10 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477

11 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 20 (Thread 0x7fa53b7fe700 (LWP 6883)):

0 0x0000559a49f33368 in nvme_qpair_resubmit_requests (qpair=qpair@entry=0x2000014a7918, num_requests=num_requests@entry=0) at nvme_qpair.c:1002

1 0x0000559a49f3343e in spdk_nvme_qpair_process_completions (qpair=qpair@entry=0x2000014a7918, max_completions=max_completions@entry=0) at nvme_qpair.c:735

2 0x0000559a49f2e1cc in nvme_pcie_poll_group_process_completions (tgroup=0x559a4b01b5b0, completions_per_qpair=0, disconnected_qpair_cb=0x7fa584d048a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair, void)>) at nvme_pcie.c:2536

3 0x0000559a49f36cc9 in nvme_transport_poll_group_process_completions (tgroup=tgroup@entry=0x559a4b01b5b0, completions_per_qpair=completions_per_qpair@entry=0, disconnected_qpair_cb=disconnected_qpair_cb@entry=0x7fa584d048a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair, void)>) at nvme_transport.c:499

4 0x0000559a49f40c79 in spdk_nvme_poll_group_process_completions (group=, completions_per_qpair=0, disconnected_qpair_cb=0x7fa584d048a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair, void)>) at nvme_poll_group.c:127

5 0x00007fa584d07dfb in lse::chunkserver::SpdkNamespace::Complete (this=0x559a4b19ca10) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/spdk.cpp:408

6 0x00007fa585079ddc in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::Running (this=0x559a4b020460) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1207

7 0x00007fa585074d24 in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}::operator()() const (this=0x559a4b020460) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1152

8 0x00007fa5850ad6be in std::invoke_impl<void, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}>(std::__invoke_other, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}&&) (f=...) at /usr/include/c++/9/bits/invoke.h:60

9 0x00007fa5850a83c8 in std::invoke<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}>(std::invoke_result&&, (lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}&&)...) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95

10 0x00007fa5850a34f9 in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7fa530c41a28) at /usr/include/c++/9/thread:244

11 0x00007fa58509a16e in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> >::operator()() (this=0x7fa530c41a28) at /usr/include/c++/9/thread:251

12 0x00007fa585098dfb in std::thread::_State_impl<std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> > >::_M_run() (this=0x7fa530c41a20) at /usr/include/c++/9/thread:195

13 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

14 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477

15 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 19 (Thread 0x7fa53bfff700 (LWP 6880)):

0 0x00007fa5834253bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa53bffd6f0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa58342b047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa53bffd6f0, remaining=remaining@entry=0x0) at nanosleep.c:27

2 0x00007fa58345d9bf in usleep (useconds=useconds@entry=1000000) at ../sysdeps/posix/usleep.c:32

3 0x00007fa583a4d178 in bthread_usleep (microseconds=microseconds@entry=1000000) at /home/leo/code/thirdparty/brpc/src/bthread/bthread.cpp:363

4 0x00007fa583b40862 in brpc::Server::RunUntilAskedToQuit (this=0x7fa530c5b280) at /home/leo/code/thirdparty/brpc/src/brpc/server.cpp:1693

5 0x00007fa585068779 in lse::chunkserver::Container::execute (this=0x559a4b01d8f0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/container.cpp:182

6 0x00007fa5850858a0 in std::__invoke_impl<void, void (lse::chunkserver::Container::)(), lse::chunkserver::Container> (f=@0x559a4aff1850: (void (lse::chunkserver::Container::)(class lse::chunkserver::Container const)) 0x7fa585067818 <lse::chunkserver::Container::execute()>, t=@0x559a4aff1848: 0x559a4b01d8f0) at /usr/include/c++/9/bits/invoke.h:73

7 0x00007fa58507f616 in std::invoke<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> (fn=@0x559a4aff1850: (void (lse::chunkserver::Container::)(class lse::chunkserver::Container const)) 0x7fa585067818 <lse::chunkserver::Container::execute()>) at /usr/include/c++/9/bits/invoke.h:95

8 0x00007fa585077e9e in std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> >::_M_invoke<0ul, 1ul> (this=0x559a4aff1848) at /usr/include/c++/9/thread:244

9 0x00007fa58507430e in std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> >::operator() (this=0x559a4aff1848) at /usr/include/c++/9/thread:251

10 0x00007fa58506f2cf in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> > >::_M_run (this=0x559a4aff1840) at /usr/include/c++/9/thread:195

11 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

12 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477

13 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 18 (Thread 0x7fa560ff9700 (LWP 6879)):

0 __lll_lock_wait (futex=futex@entry=0x7fa540048cc0, private=0) at lowlevellock.c:52

1 0x00007fa5837400a3 in __GI___pthread_mutex_lock (mutex=0x7fa540048cc0) at ../nptl/pthread_mutex_lock.c:80

2 0x00007fa583a57fa7 in bthread::pthread_mutex_lock_impl (mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:549

3 pthread_mutex_lock (__mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:812

4 pthread_mutex_lock (mutex=mutex@entry=0x7fa540048cc0) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:811

5 0x00007fa58399dea2 in butil::Mutex::lock (this=0x7fa540048cc0) at /usr/local/include/butil/synchronization/lock.h:69

6 std::unique_lock::lock (this=0x7fa5110d06c0, this=0x7fa5110d06c0) at /usr/include/c++/9/bits/unique_lock.h:141

7 std::unique_lock::unique_lock (__m=..., this=0x7fa5110d06c0) at /usr/include/c++/9/bits/unique_lock.h:71

8 braft::NodeImpl::get_status (this=0x7fa540048a20, status=0x7fa5110d0840) at /home/leo/code/thirdparty/braft/src/braft/node.cpp:2656

9 0x00007fa584e2aa47 in lse::chunkserver::Copyset::GetCopysetStaticInfo (this=0x7fa540048750, info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset.cpp:315

10 0x00007fa584e3bf3e in lse::chunkserver::CopysetManager::GetCopysetStaticInfo (this=0x7fa578ce0670, info_vec=std::vector of length 0, capacity 0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:188

11 0x00007fa584e3bbf8 in lse::chunkserver::CopysetManager::GetContainerStaticInfo (this=0x7fa578ce0670, container_info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:164

12 0x00007fa58506919b in lse::chunkserver::Container::GetStatistic (this=0x559a4b022420, info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/container.cpp:210

13 0x00007fa5850ba7ad in lse::chunkserver::ChunkControlServiceImp::Heartbeat (this=0x7ffcfb885bf0, controller=0x7fa5523d4630, request=0x7fa50000bb20, response=0x7fa553c2db70, done=0x7fa550dc3b80) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/control_service_imp.cpp:352

14 0x00007fa5850fe354 in lse::chunkserver::ChunkControlService::CallMethod (this=0x7ffcfb885bf0, method=0x7fa578c72760, controller=0x7fa5523d4630, request=0x7fa50000bb20, response=0x7fa553c2db70, done=0x7fa550dc3b80) at /root/jenkins_S/workspace/CI-CKS-Daily/proto/chunkserver.pb.cc:9467

15 0x00007fa583aa2423 in brpc::policy::ProcessRpcRequest (msg_base=0x7fa54003ed70) at /usr/include/c++/9/bits/unique_ptr.h:381

16 0x00007fa583a9729b in brpc::ProcessInputMessage (void_arg=void_arg@entry=0x7fa54003ed70) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:135

17 0x00007fa583a981e2 in brpc::RunLastMessage::operator() (this=, last_msg=0x7fa54003ed70) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:141

18 std::unique_ptr<brpc::InputMessageBase, brpc::RunLastMessage>::~unique_ptr (this=, __in_chrg=) at /usr/include/c++/9/bits/unique_ptr.h:292

19 brpc::InputMessenger::OnNewMessages (m=0x7fa50453f3d0) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:186

20 0x00007fa583b5bf62 in brpc::Socket::ProcessEvent (arg=0x7fa50453f3d0) at /home/leo/code/thirdparty/brpc/src/brpc/socket.cpp:1017

21 0x00007fa583a62acf in bthread::TaskGroup::task_runner (skip_remained=) at /home/leo/code/thirdparty/brpc/src/bthread/task_group.cpp:295

22 0x00007fa583beb4b1 in bthread_make_fcontext () from /usr/local/lib/libbraft.so

23 0x0000000000000000 in ?? ()

Thread 17 (Thread 0x7fa5617fa700 (LWP 6878)):

0 __lll_lock_wait (futex=futex@entry=0x7fa540048cc0, private=0) at lowlevellock.c:52

1 0x00007fa5837400a3 in __GI___pthread_mutex_lock (mutex=0x7fa540048cc0) at ../nptl/pthread_mutex_lock.c:80

2 0x00007fa583a57fa7 in bthread::pthread_mutex_lock_impl (mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:549

3 pthread_mutex_lock (__mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:812

4 pthread_mutex_lock (mutex=mutex@entry=0x7fa540048cc0) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:811

5 0x00007fa58399dea2 in butil::Mutex::lock (this=0x7fa540048cc0) at /usr/local/include/butil/synchronization/lock.h:69

6 std::unique_lock::lock (this=0x7fa5116d66c0, this=0x7fa5116d66c0) at /usr/include/c++/9/bits/unique_lock.h:141

7 std::unique_lock::unique_lock (__m=..., this=0x7fa5116d66c0) at /usr/include/c++/9/bits/unique_lock.h:71

8 braft::NodeImpl::get_status (this=0x7fa540048a20, status=0x7fa5116d6840) at /home/leo/code/thirdparty/braft/src/braft/node.cpp:2656

9 0x00007fa584e2aa47 in lse::chunkserver::Copyset::GetCopysetStaticInfo (this=0x7fa540048750, info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset.cpp:315

10 0x00007fa584e3bf3e in lse::chunkserver::CopysetManager::GetCopysetStaticInfo (this=0x7fa578ce0670, info_vec=std::vector of length 0, capacity 0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:188

11 0x00007fa584e3bbf8 in lse::chunkserver::CopysetManager::GetContainerStaticInfo (this=0x7fa578ce0670, container_info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:164

12 0x00007fa58506919b in lse::chunkserver::Container::GetStatistic (this=0x559a4b022420, info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/container.cpp:210

13 0x00007fa5850ba7ad in lse::chunkserver::ChunkControlServiceImp::Heartbeat (this=0x7ffcfb885bf0, controller=0x7fa500118190, request=0x7fa550255150, response=0x7fa553adab50, done=0x7fa500be5690) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/control_service_imp.cpp:352

14 0x00007fa5850fe354 in lse::chunkserver::ChunkControlService::CallMethod (this=0x7ffcfb885bf0, method=0x7fa578c72760, controller=0x7fa500118190, request=0x7fa550255150, response=0x7fa553adab50, done=0x7fa500be5690) at /root/jenkins_S/workspace/CI-CKS-Daily/proto/chunkserver.pb.cc:9467

15 0x00007fa583aa2423 in brpc::policy::ProcessRpcRequest (msg_base=0x7fa54003ed70) at /usr/include/c++/9/bits/unique_ptr.h:381

16 0x00007fa583a9729b in brpc::ProcessInputMessage (void_arg=void_arg@entry=0x7fa54003ed70) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:135

17 0x00007fa583a981e2 in brpc::RunLastMessage::operator() (this=, last_msg=0x7fa54003ed70) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:141

18 std::unique_ptr<brpc::InputMessageBase, brpc::RunLastMessage>::~unique_ptr (this=, __in_chrg=) at /usr/include/c++/9/bits/unique_ptr.h:292

19 brpc::InputMessenger::OnNewMessages (m=0x7fa50453e7d0) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:186

20 0x00007fa583b5bf62 in brpc::Socket::ProcessEvent (arg=0x7fa50453e7d0) at /home/leo/code/thirdparty/brpc/src/brpc/socket.cpp:1017

21 0x00007fa583a62acf in bthread::TaskGroup::task_runner (skip_remained=) at /home/leo/code/thirdparty/brpc/src/bthread/task_group.cpp:295

22 0x00007fa583beb4b1 in bthread_make_fcontext () from /usr/local/lib/libbraft.so

23 0x0000000000000000 in ?? ()

Thread 16 (Thread 0x7fa561ffb700 (LWP 6877)):

0 __lll_lock_wait (futex=futex@entry=0x7fa55406eca0, private=0) at lowlevellock.c:52

1 0x00007fa5837400a3 in __GI___pthread_mutex_lock (mutex=0x7fa55406eca0) at ../nptl/pthread_mutex_lock.c:80

2 0x00007fa583a57fa7 in bthread::pthread_mutex_lock_impl (mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:549

3 pthread_mutex_lock (__mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:812

4 pthread_mutex_lock (__mutex=0x7fa55406eca0) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:811

5 0x00007fa5839b46e6 in butil::Mutex::lock (this=) at /usr/local/include/butil/synchronization/lock.h:69

6 std::unique_lock::lock (this=0x7fa4d4a4a480, this=0x7fa4d4a4a480) at /usr/include/c++/9/bits/unique_lock.h:141

7 std::unique_lock::unique_lock (__m=..., this=0x7fa4d4a4a480) at /usr/include/c++/9/bits/unique_lock.h:71

8 braft::FollowerStableClosure::run (this=this@entry=0x7fa55219f760) at /home/leo/code/thirdparty/braft/src/braft/node.cpp:2139

9 0x00007fa5839b4ba4 in braft::FollowerStableClosure::Run (this=0x7fa55219f760) at /home/leo/code/thirdparty/braft/src/braft/node.cpp:2123

10 0x00007fa583995f25 in braft::AppendBatcher::flush (this=this@entry=0x7fa4d4a4a5f0) at /home/leo/code/thirdparty/braft/src/braft/log_manager.cpp:515

11 0x00007fa58398eb63 in braft::LogManager::disk_thread (meta=0x7fa55407fe50, iter=...) at /home/leo/code/thirdparty/braft/src/braft/log_manager.cpp:619

12 0x00007fa583a4e070 in bthread::ExecutionQueueBase::_execute (this=0x7fa55406fe50, head=0x7fa5480370c0, high_priority=, niterated=0x0) at /home/leo/code/thirdparty/brpc/src/bthread/execution_queue.cpp:272

13 0x00007fa583a4fb90 in bthread::ExecutionQueueBase::_execute_tasks (arg=) at /home/leo/code/thirdparty/brpc/src/bthread/execution_queue.cpp:151

14 0x00007fa583a62acf in bthread::TaskGroup::task_runner (skip_remained=) at /home/leo/code/thirdparty/brpc/src/bthread/task_group.cpp:295

15 0x00007fa583beb4b1 in bthread_make_fcontext () from /usr/local/lib/libbraft.so

16 0x0000000000000000 in ?? ()

Thread 15 (Thread 0x7fa5627fc700 (LWP 6876)):

0 __lll_lock_wait (futex=futex@entry=0x7fa540059e88, private=0) at lowlevellock.c:52

1 0x00007fa5837400a3 in __GI___pthread_mutex_lock (mutex=0x7fa540059e88) at ../nptl/pthread_mutex_lock.c:80

2 0x00007fa583a57fa7 in bthread::pthread_mutex_lock_impl (mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:549

3 pthread_mutex_lock (__mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:812

4 pthread_mutex_lock (mutex=mutex@entry=0x7fa540059e88) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:811

5 0x00007fa58398c7b5 in butil::Mutex::lock (this=0x7fa540059e88) at /usr/local/include/butil/synchronization/lock.h:69

6 std::unique_lock::lock (this=0x7fa51c6db560, this=0x7fa51c6db560) at /usr/include/c++/9/bits/unique_lock.h:141

7 std::unique_lock::unique_lock (__m=..., this=0x7fa51c6db560) at /usr/include/c++/9/bits/unique_lock.h:71

8 braft::LogManager::set_disk_id (this=0x7fa540059e70, disk_id=...) at /home/leo/code/thirdparty/braft/src/braft/log_manager.cpp:809

9 0x00007fa58398eb6e in braft::LogManager::disk_thread (meta=0x7fa540059e70, iter=...) at /home/leo/code/thirdparty/braft/src/braft/log_manager.cpp:620

10 0x00007fa583a4e070 in bthread::ExecutionQueueBase::_execute (this=0x7fa540049e70, head=0x7fa54c0d2230, high_priority=, niterated=0x0) at /home/leo/code/thirdparty/brpc/src/bthread/execution_queue.cpp:272

11 0x00007fa583a4fb90 in bthread::ExecutionQueueBase::_execute_tasks (arg=) at /home/leo/code/thirdparty/brpc/src/bthread/execution_queue.cpp:151

12 0x00007fa583a62acf in bthread::TaskGroup::task_runner (skip_remained=) at /home/leo/code/thirdparty/brpc/src/bthread/task_group.cpp:295

13 0x00007fa583beb4b1 in bthread_make_fcontext () from /usr/local/lib/libbraft.so

14 0x0000000000000000 in ?? ()

Thread 14 (Thread 0x7fa562ffd700 (LWP 6875)):

0 __lll_lock_wait (futex=futex@entry=0x7fa540048cc0, private=0) at lowlevellock.c:52

1 0x00007fa5837400a3 in __GI___pthread_mutex_lock (mutex=0x7fa540048cc0) at ../nptl/pthread_mutex_lock.c:80

2 0x00007fa583a57fa7 in bthread::pthread_mutex_lock_impl (mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:549

3 pthread_mutex_lock (__mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:812

4 pthread_mutex_lock (mutex=mutex@entry=0x7fa540048cc0) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:811

5 0x00007fa58399dea2 in butil::Mutex::lock (this=0x7fa540048cc0) at /usr/local/include/butil/synchronization/lock.h:69

6 std::unique_lock::lock (this=0x7fa5113d36c0, this=0x7fa5113d36c0) at /usr/include/c++/9/bits/unique_lock.h:141

7 std::unique_lock::unique_lock (__m=..., this=0x7fa5113d36c0) at /usr/include/c++/9/bits/unique_lock.h:71

8 braft::NodeImpl::get_status (this=0x7fa540048a20, status=0x7fa5113d3840) at /home/leo/code/thirdparty/braft/src/braft/node.cpp:2656

9 0x00007fa584e2aa47 in lse::chunkserver::Copyset::GetCopysetStaticInfo (this=0x7fa540048750, info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset.cpp:315

10 0x00007fa584e3bf3e in lse::chunkserver::CopysetManager::GetCopysetStaticInfo (this=0x7fa578ce0670, info_vec=std::vector of length 0, capacity 0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:188

11 0x00007fa584e3bbf8 in lse::chunkserver::CopysetManager::GetContainerStaticInfo (this=0x7fa578ce0670, container_info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:164

12 0x00007fa58506919b in lse::chunkserver::Container::GetStatistic (this=0x559a4b022420, info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/container.cpp:210

13 0x00007fa5850ba7ad in lse::chunkserver::ChunkControlServiceImp::Heartbeat (this=0x7ffcfb885bf0, controller=0x7fa551e61300, request=0x7fa500c000e0, response=0x7fa550813ce0, done=0x7fa553110ad0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/control_service_imp.cpp:352

14 0x00007fa5850fe354 in lse::chunkserver::ChunkControlService::CallMethod (this=0x7ffcfb885bf0, method=0x7fa578c72760, controller=0x7fa551e61300, request=0x7fa500c000e0, response=0x7fa550813ce0, done=0x7fa553110ad0) at /root/jenkins_S/workspace/CI-CKS-Daily/proto/chunkserver.pb.cc:9467

15 0x00007fa583aa2423 in brpc::policy::ProcessRpcRequest (msg_base=0x7fa54003ed70) at /usr/include/c++/9/bits/unique_ptr.h:381

16 0x00007fa583a9729b in brpc::ProcessInputMessage (void_arg=void_arg@entry=0x7fa54003ed70) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:135

17 0x00007fa583a981e2 in brpc::RunLastMessage::operator() (this=, last_msg=0x7fa54003ed70) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:141

18 std::unique_ptr<brpc::InputMessageBase, brpc::RunLastMessage>::~unique_ptr (this=, __in_chrg=) at /usr/include/c++/9/bits/unique_ptr.h:292

19 brpc::InputMessenger::OnNewMessages (m=0x7fa50453edd0) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:186

20 0x00007fa583b5bf62 in brpc::Socket::ProcessEvent (arg=0x7fa50453edd0) at /home/leo/code/thirdparty/brpc/src/brpc/socket.cpp:1017

21 0x00007fa583a62acf in bthread::TaskGroup::task_runner (skip_remained=) at /home/leo/code/thirdparty/brpc/src/bthread/task_group.cpp:295

22 0x00007fa583beb4b1 in bthread_make_fcontext () from /usr/local/lib/libbraft.so

23 0x0000000000000000 in ?? ()

Thread 13 (Thread 0x7fa5637fe700 (LWP 6874)):

0 __lll_lock_wait (futex=futex@entry=0x7fa540048cc0, private=0) at lowlevellock.c:52

1 0x00007fa5837400a3 in __GI___pthread_mutex_lock (mutex=0x7fa540048cc0) at ../nptl/pthread_mutex_lock.c:80

2 0x00007fa583a57fa7 in bthread::pthread_mutex_lock_impl (mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:549

3 pthread_mutex_lock (__mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:812

4 pthread_mutex_lock (mutex=mutex@entry=0x7fa540048cc0) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:811

5 0x00007fa58399dea2 in butil::Mutex::lock (this=0x7fa540048cc0) at /usr/local/include/butil/synchronization/lock.h:69

6 std::unique_lock::lock (this=0x7fa510bcb6c0, this=0x7fa510bcb6c0) at /usr/include/c++/9/bits/unique_lock.h:141

7 std::unique_lock::unique_lock (__m=..., this=0x7fa510bcb6c0) at /usr/include/c++/9/bits/unique_lock.h:71

8 braft::NodeImpl::get_status (this=0x7fa540048a20, status=0x7fa510bcb840) at /home/leo/code/thirdparty/braft/src/braft/node.cpp:2656

9 0x00007fa584e2aa47 in lse::chunkserver::Copyset::GetCopysetStaticInfo (this=0x7fa540048750, info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset.cpp:315

10 0x00007fa584e3bf3e in lse::chunkserver::CopysetManager::GetCopysetStaticInfo (this=0x7fa578ce0670, info_vec=std::vector of length 0, capacity 0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:188

11 0x00007fa584e3bbf8 in lse::chunkserver::CopysetManager::GetContainerStaticInfo (this=0x7fa578ce0670, container_info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:164

12 0x00007fa58506919b in lse::chunkserver::Container::GetStatistic (this=0x559a4b022420, info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/container.cpp:210

13 0x00007fa5850ba7ad in lse::chunkserver::ChunkControlServiceImp::Heartbeat (this=0x7ffcfb885bf0, controller=0x7fa50009f1f0, request=0x7fa50000a720, response=0x7fa5513070a0, done=0x7fa550b14a20) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/control_service_imp.cpp:352

14 0x00007fa5850fe354 in lse::chunkserver::ChunkControlService::CallMethod (this=0x7ffcfb885bf0, method=0x7fa578c72760, controller=0x7fa50009f1f0, request=0x7fa50000a720, response=0x7fa5513070a0, done=0x7fa550b14a20) at /root/jenkins_S/workspace/CI-CKS-Daily/proto/chunkserver.pb.cc:9467

15 0x00007fa583aa2423 in brpc::policy::ProcessRpcRequest (msg_base=0x7fa54003ed70) at /usr/include/c++/9/bits/unique_ptr.h:381

16 0x00007fa583a9729b in brpc::ProcessInputMessage (void_arg=void_arg@entry=0x7fa54003ed70) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:135

17 0x00007fa583a981e2 in brpc::RunLastMessage::operator() (this=, last_msg=0x7fa54003ed70) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:141

18 std::unique_ptr<brpc::InputMessageBase, brpc::RunLastMessage>::~unique_ptr (this=, __in_chrg=) at /usr/include/c++/9/bits/unique_ptr.h:292

19 brpc::InputMessenger::OnNewMessages (m=0x7fa50453fdd0) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:186

20 0x00007fa583b5bf62 in brpc::Socket::ProcessEvent (arg=0x7fa50453fdd0) at /home/leo/code/thirdparty/brpc/src/brpc/socket.cpp:1017

21 0x00007fa583a62acf in bthread::TaskGroup::task_runner (skip_remained=) at /home/leo/code/thirdparty/brpc/src/bthread/task_group.cpp:295

22 0x00007fa583beb4b1 in bthread_make_fcontext () from /usr/local/lib/libbraft.so

23 0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7fa563fff700 (LWP 6873)):

0 __lll_lock_wait (futex=futex@entry=0x7fa540048cc0, private=0) at lowlevellock.c:52

1 0x00007fa5837400a3 in __GI___pthread_mutex_lock (mutex=0x7fa540048cc0) at ../nptl/pthread_mutex_lock.c:80

2 0x00007fa583a57fa7 in bthread::pthread_mutex_lock_impl (mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:549

3 pthread_mutex_lock (__mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:812

4 pthread_mutex_lock (mutex=mutex@entry=0x7fa540048cc0) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:811

5 0x00007fa58399dea2 in butil::Mutex::lock (this=0x7fa540048cc0) at /usr/local/include/butil/synchronization/lock.h:69

6 std::unique_lock::lock (this=0x7fa4e4ece6c0, this=0x7fa4e4ece6c0) at /usr/include/c++/9/bits/unique_lock.h:141

7 std::unique_lock::unique_lock (__m=..., this=0x7fa4e4ece6c0) at /usr/include/c++/9/bits/unique_lock.h:71

8 braft::NodeImpl::get_status (this=0x7fa540048a20, status=0x7fa4e4ece840) at /home/leo/code/thirdparty/braft/src/braft/node.cpp:2656

9 0x00007fa584e2aa47 in lse::chunkserver::Copyset::GetCopysetStaticInfo (this=0x7fa540048750, info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset.cpp:315

10 0x00007fa584e3bf3e in lse::chunkserver::CopysetManager::GetCopysetStaticInfo (this=0x7fa578ce0670, info_vec=std::vector of length 0, capacity 0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:188

11 0x00007fa584e3bbf8 in lse::chunkserver::CopysetManager::GetContainerStaticInfo (this=0x7fa578ce0670, container_info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:164

12 0x00007fa58506919b in lse::chunkserver::Container::GetStatistic (this=0x559a4b022420, info=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/container.cpp:210

13 0x00007fa5850ba7ad in lse::chunkserver::ChunkControlServiceImp::Heartbeat (this=0x7ffcfb885bf0, controller=0x7fa558c53fa0, request=0x7fa55826f700, response=0x7fa558c54220, done=0x7fa558c54250) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/control_service_imp.cpp:352

14 0x00007fa5850fe354 in lse::chunkserver::ChunkControlService::CallMethod (this=0x7ffcfb885bf0, method=0x7fa578c72760, controller=0x7fa558c53fa0, request=0x7fa55826f700, response=0x7fa558c54220, done=0x7fa558c54250) at /root/jenkins_S/workspace/CI-CKS-Daily/proto/chunkserver.pb.cc:9467

15 0x00007fa583aa2423 in brpc::policy::ProcessRpcRequest (msg_base=0x7fa54b7076b0) at /usr/include/c++/9/bits/unique_ptr.h:381

16 0x00007fa583a9729b in brpc::ProcessInputMessage (void_arg=void_arg@entry=0x7fa54b7076b0) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:135

17 0x00007fa583a981e2 in brpc::RunLastMessage::operator() (this=, last_msg=0x7fa54b7076b0) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:141

18 std::unique_ptr<brpc::InputMessageBase, brpc::RunLastMessage>::~unique_ptr (this=, __in_chrg=) at /usr/include/c++/9/bits/unique_ptr.h:292

19 brpc::InputMessenger::OnNewMessages (m=0x7fa53c097030) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:186

20 0x00007fa583b5bf62 in brpc::Socket::ProcessEvent (arg=0x7fa53c097030) at /home/leo/code/thirdparty/brpc/src/brpc/socket.cpp:1017

21 0x00007fa583a62acf in bthread::TaskGroup::task_runner (skip_remained=) at /home/leo/code/thirdparty/brpc/src/bthread/task_group.cpp:295

22 0x00007fa583beb4b1 in bthread_make_fcontext () from /usr/local/lib/libbraft.so

23 0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7fa570d59700 (LWP 6872)):

0 __lll_lock_wait (futex=futex@entry=0x7fa55406eca0, private=0) at lowlevellock.c:52

1 0x00007fa5837400a3 in __GI___pthread_mutex_lock (mutex=0x7fa55406eca0) at ../nptl/pthread_mutex_lock.c:80

2 0x00007fa583a57fa7 in bthread::pthread_mutex_lock_impl (mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:549

3 pthread_mutex_lock (__mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:812

4 pthread_mutex_lock (mutex=mutex@entry=0x7fa55406eca0) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:811

5 0x00007fa5839a567a in butil::Mutex::lock (this=0x7fa55406eca0) at /usr/local/include/butil/synchronization/lock.h:69

6 std::unique_lock::lock (this=0x7fa51dbf0a10, this=0x7fa51dbf0a10) at /usr/include/c++/9/bits/unique_lock.h:141

7 std::unique_lock::unique_lock (__m=..., this=0x7fa51dbf0a10) at /usr/include/c++/9/bits/unique_lock.h:71

8 braft::NodeImpl::handle_append_entries_request (this=0x7fa55406ea00, cntl=cntl@entry=0x7fa53d3ce460, request=request@entry=0x7fa5492fff70, response=response@entry=0x7fa53d546e50, done=done@entry=0x7fa53db80dd0, from_append_entries_cache=from_append_entries_cache@entry=false) at /home/leo/code/thirdparty/braft/src/braft/node.cpp:2208

9 0x00007fa5839c546a in braft::RaftServiceImpl::append_entries (this=, cntl_base=0x7fa53d3ce460, request=0x7fa5492fff70, response=0x7fa53d546e50, done=0x7fa53db80dd0) at /usr/local/include/brpc/closure_guard.h:55

10 0x00007fa583a08611 in braft::RaftService::CallMethod (this=, method=, controller=, request=, response=, done=) at /home/leo/code/thirdparty/braft/bld/braft/raft.pb.cc:4813

11 0x00007fa583aa2423 in brpc::policy::ProcessRpcRequest (msg_base=0x7fa5400423f0) at /usr/include/c++/9/bits/unique_ptr.h:381

12 0x00007fa583a9729b in brpc::ProcessInputMessage (void_arg=void_arg@entry=0x7fa5400423f0) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:135

13 0x00007fa583a98214 in brpc::RunLastMessage::operator() (this=, last_msg=0x7fa5400423f0) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:141

14 std::unique_ptr<brpc::InputMessageBase, brpc::RunLastMessage>::~unique_ptr (this=, __in_chrg=) at /usr/include/c++/9/bits/unique_ptr.h:292

15 brpc::InputMessenger::OnNewMessages (m=0x7fa54ab261a0) at /usr/include/c++/9/bits/unique_ptr.h:286

16 0x00007fa583b5bf62 in brpc::Socket::ProcessEvent (arg=0x7fa54ab261a0) at /home/leo/code/thirdparty/brpc/src/brpc/socket.cpp:1017

17 0x00007fa583a62acf in bthread::TaskGroup::task_runner (skip_remained=) at /home/leo/code/thirdparty/brpc/src/bthread/task_group.cpp:295

18 0x00007fa583beb4b1 in bthread_make_fcontext () from /usr/local/lib/libbraft.so

19 0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7fa57155a700 (LWP 6871)):

0 __lll_lock_wait (futex=futex@entry=0x7fa540059e88, private=0) at lowlevellock.c:52

1 0x00007fa5837400a3 in __GI___pthread_mutex_lock (mutex=0x7fa540059e88) at ../nptl/pthread_mutex_lock.c:80

2 0x00007fa583a57fa7 in bthread::pthread_mutex_lock_impl (mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:549

3 pthread_mutex_lock (__mutex=) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:812

4 pthread_mutex_lock (mutex=mutex@entry=0x7fa540059e88) at /home/leo/code/thirdparty/brpc/src/bthread/mutex.cpp:811

5 0x00007fa58399072c in butil::Mutex::lock (this=0x7fa540059e88) at /usr/local/include/butil/synchronization/lock.h:69

6 std::unique_lock::lock (this=0x7fa4d4b4b570, this=0x7fa4d4b4b570) at /usr/include/c++/9/bits/unique_lock.h:141

7 std::unique_lock::unique_lock (__m=..., this=0x7fa4d4b4b570) at /usr/include/c++/9/bits/unique_lock.h:71

8 braft::LogManager::get_term (this=0x7fa540059e70, index=1154599) at /home/leo/code/thirdparty/braft/src/braft/log_manager.cpp:737

9 0x00007fa5839cbd46 in braft::Replicator::_fill_common_fields (this=0x7fa54e57d640, request=0x7fa54029a4e0, prev_log_index=, is_heartbeat=) at /home/leo/code/thirdparty/braft/src/braft/replicator.cpp:517

10 0x00007fa5839d14bf in braft::Replicator::_send_entries (this=0x7fa54e57d640) at /home/leo/code/thirdparty/braft/src/braft/replicator.cpp:640

11 0x00007fa5839d3718 in braft::Replicator::_on_rpc_returned (id=, cntl=0x7fa5090c1210, request=0x7fa54f3ad4a0, response=, rpc_send_time=) at /home/leo/code/thirdparty/braft/src/braft/replicator.cpp:510

12 0x00007fa5839d4e3c in brpc::internal::FunctionClosure5<unsigned long, brpc::Controller, braft::AppendEntriesRequest, braft::AppendEntriesResponse*, long>::Run (this=0x7fa54dc796f0) at /usr/local/include/brpc/callback.h:371

13 0x00007fa583a7523b in brpc::Controller::EndRPC (this=0x7fa5090c1210, info=...) at /home/leo/code/thirdparty/brpc/src/brpc/controller.cpp:912

14 0x00007fa583a76a73 in brpc::Controller::OnVersionedRPCReturned (this=this@entry=0x7fa5090c1210, info=..., new_bthread=new_bthread@entry=false, saved_error=saved_error@entry=0) at /home/leo/code/thirdparty/brpc/src/brpc/controller.cpp:695

15 0x00007fa583a9fe83 in brpc::ControllerPrivateAccessor::OnResponse (this=, saved_error=0, id=...) at /home/leo/code/thirdparty/brpc/src/brpc/details/controller_private_accessor.h:48

16 brpc::policy::ProcessRpcResponse (msg_base=0x7fa54c0bacf0) at /home/leo/code/thirdparty/brpc/src/brpc/policy/baidu_rpc_protocol.cpp:619

17 0x00007fa583a9729b in brpc::ProcessInputMessage (void_arg=void_arg@entry=0x7fa54c0bacf0) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:135

18 0x00007fa583a98214 in brpc::RunLastMessage::operator() (this=, last_msg=0x7fa54c0bacf0) at /home/leo/code/thirdparty/brpc/src/brpc/input_messenger.cpp:141

19 std::unique_ptr<brpc::InputMessageBase, brpc::RunLastMessage>::~unique_ptr (this=, __in_chrg=) at /usr/include/c++/9/bits/unique_ptr.h:292

20 brpc::InputMessenger::OnNewMessages (m=0x7fa54ab28fa0) at /usr/include/c++/9/bits/unique_ptr.h:286

21 0x00007fa583b5bf62 in brpc::Socket::ProcessEvent (arg=0x7fa54ab28fa0) at /home/leo/code/thirdparty/brpc/src/brpc/socket.cpp:1017

22 0x00007fa583a62acf in bthread::TaskGroup::task_runner (skip_remained=) at /home/leo/code/thirdparty/brpc/src/bthread/task_group.cpp:295

23 0x00007fa583beb4b1 in bthread_make_fcontext () from /usr/local/lib/libbraft.so

24 0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7fa571d5b700 (LWP 6870)):

0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38

1 0x00007fa583a68948 in bthread::futex_wait_private (timeout=, expected=25900, addr1=0x7fa578c5f338) at /home/leo/code/thirdparty/brpc/src/bthread/sys_futex.h:40

2 bthread::TimerThread::run (this=) at /home/leo/code/thirdparty/brpc/src/bthread/timer_thread.cpp:431

3 0x00007fa583a6940d in bthread::TimerThread::run_this (arg=) at /home/leo/code/thirdparty/brpc/src/bthread/timer_thread.cpp:120

4 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477

5 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7fa57255c700 (LWP 6869)):

0 futex_abstimed_wait_cancelable (private=, abstime=0x7fa57255a6a0, clockid=, expected=0, futex_word=0x7fa578001048) at ../sysdeps/nptl/futex-internal.h:320

1 __pthread_cond_wait_common (abstime=0x7fa57255a6a0, clockid=, mutex=0x7fa578000ff8, cond=0x7fa578001020) at pthread_cond_wait.c:520

2 __pthread_cond_timedwait (cond=0x7fa578001020, mutex=0x7fa578000ff8, abstime=0x7fa57255a6a0) at pthread_cond_wait.c:656

3 0x00007fa584d00bf2 in gthread_cond_timedwait (cond=0x7fa578001020, mutex=0x7fa578000ff8, abs_timeout=0x7fa57255a6a0) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:872

4 0x00007fa584d0333d in std::condition_variable::wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fa578001020, lock=..., __atime=...) at /usr/include/c++/9/condition_variable:188

5 0x00007fa584d027e1 in std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fa578001020, lock=..., atime=...) at /usr/include/c++/9/condition_variable:121

6 0x00007fa584d01eac in std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x7fa578001020, lock=..., rtime=...) at /usr/include/c++/9/condition_variable:152

7 0x00007fa584cfe916 in lse::chunkserver::PageAlloter::<lambda()>::operator()(void) const (__closure=0x7fa578c5a0a8) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/page_allocator.cpp:45

8 0x00007fa584d007e8 in std::invoke_impl<void, lse::chunkserver::PageAlloter::Start()::<lambda()> >(std::__invoke_other, lse::chunkserver::PageAlloter::<lambda()> &&) (f=...) at /usr/include/c++/9/bits/invoke.h:60

9 0x00007fa584d0073b in std::invoke<lse::chunkserver::PageAlloter::Start()::<lambda()> >(lse::chunkserver::PageAlloter::<lambda()> &&) (fn=...) at /usr/include/c++/9/bits/invoke.h:95

10 0x00007fa584d00699 in std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x7fa578c5a0a8) at /usr/include/c++/9/thread:244

11 0x00007fa584d0061e in std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > >::operator()(void) (this=0x7fa578c5a0a8) at /usr/include/c++/9/thread:251

12 0x00007fa584d005e1 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > > >::_M_run(void) (this=0x7fa578c5a0a0) at /usr/include/c++/9/thread:195

13 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

14 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477

15 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7fa573fff700 (LWP 6868)):

0 0x00007fa5834253bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa573ffd740, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa58342b047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa573ffd740, remaining=remaining@entry=0x0) at nanosleep.c:27

2 0x00007fa58345d9bf in usleep (useconds=) at ../sysdeps/posix/usleep.c:32

3 0x00007fa58507c516 in lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher (this=0x7fa5781a2240) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/l2pmap_mgr.h:734

4 0x00007fa5850ad5a3 in std::__invoke_impl<int, int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> (f=@0x7fa578c7af80: (int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(class lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey> const)) 0x7fa58507c3dc <lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher()>, t=@0x7fa578c7af78: 0x7fa5781a2240) at /usr/include/c++/9/bits/invoke.h:73

5 0x00007fa5850a8211 in std::invoke<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> (fn=@0x7fa578c7af80: (int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(class lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey> const)) 0x7fa58507c3dc <lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher()>) at /usr/include/c++/9/bits/invoke.h:95

6 0x00007fa5850a344c in std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> >::_M_invoke<0ul, 1ul> (this=0x7fa578c7af78) at /usr/include/c++/9/thread:244

7 0x00007fa58509a098 in std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> >::operator() (this=0x7fa578c7af78) at /usr/include/c++/9/thread:251

8 0x00007fa585098d8f in std::thread::_State_impl<std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> > >::_M_run (this=0x7fa578c7af70) at /usr/include/c++/9/thread:195

9 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

10 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477

11 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7fa57355e700 (LWP 6866)):

0 0x0000559a49f36cce in nvme_transport_poll_group_process_completions (tgroup=tgroup@entry=0x559a4b0214f0, completions_per_qpair=completions_per_qpair@entry=0, disconnected_qpair_cb=disconnected_qpair_cb@entry=0x7fa584d048a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair, void)>) at nvme_transport.c:499

1 0x0000559a49f40c79 in spdk_nvme_poll_group_process_completions (group=, completions_per_qpair=0, disconnected_qpair_cb=0x7fa584d048a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair, void)>) at nvme_poll_group.c:127

2 0x00007fa584d07dfb in lse::chunkserver::SpdkNamespace::Complete (this=0x559a4b01f9b0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/spdk.cpp:408

3 0x00007fa585079ddc in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::Running (this=0x559a4adb9f10) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1207

4 0x00007fa585074d24 in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}::operator()() const (this=0x559a4adb9f10) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1152

5 0x00007fa5850ad6be in std::invoke_impl<void, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}>(std::__invoke_other, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}&&) (f=...) at /usr/include/c++/9/bits/invoke.h:60

6 0x00007fa5850a83c8 in std::invoke<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}>(std::invoke_result&&, (lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}&&)...) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95

7 0x00007fa5850a34f9 in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7fa578c45c08) at /usr/include/c++/9/thread:244

8 0x00007fa58509a16e in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> >::operator()() (this=0x7fa578c45c08) at /usr/include/c++/9/thread:251

9 0x00007fa585098dfb in std::thread::_State_impl<std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> > >::_M_run() (this=0x7fa578c45c00) at /usr/include/c++/9/thread:195

10 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

11 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477

12 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7fa580dee700 (LWP 6863)):

0 0x00007fa5834253bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa580dec6f0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa58342b047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa580dec6f0, remaining=remaining@entry=0x0) at nanosleep.c:27

2 0x00007fa58345d9bf in usleep (useconds=useconds@entry=1000000) at ../sysdeps/posix/usleep.c:32

3 0x00007fa583a4d178 in bthread_usleep (microseconds=microseconds@entry=1000000) at /home/leo/code/thirdparty/brpc/src/bthread/bthread.cpp:363

4 0x00007fa583b40862 in brpc::Server::RunUntilAskedToQuit (this=0x7fa578c5b2c0) at /home/leo/code/thirdparty/brpc/src/brpc/server.cpp:1693

5 0x00007fa585068779 in lse::chunkserver::Container::execute (this=0x559a4b022420) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/container.cpp:182

6 0x00007fa5850858a0 in std::__invoke_impl<void, void (lse::chunkserver::Container::)(), lse::chunkserver::Container> (f=@0x559a4b015460: (void (lse::chunkserver::Container::)(class lse::chunkserver::Container const)) 0x7fa585067818 <lse::chunkserver::Container::execute()>, t=@0x559a4b015458: 0x559a4b022420) at /usr/include/c++/9/bits/invoke.h:73

7 0x00007fa58507f616 in std::invoke<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> (fn=@0x559a4b015460: (void (lse::chunkserver::Container::)(class lse::chunkserver::Container const)) 0x7fa585067818 <lse::chunkserver::Container::execute()>) at /usr/include/c++/9/bits/invoke.h:95

8 0x00007fa585077e9e in std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> >::_M_invoke<0ul, 1ul> (this=0x559a4b015458) at /usr/include/c++/9/thread:244

9 0x00007fa58507430e in std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> >::operator() (this=0x559a4b015458) at /usr/include/c++/9/thread:251

10 0x00007fa58506f2cf in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> > >::_M_run (this=0x559a4b015450) at /usr/include/c++/9/thread:195

11 0x00007fa583628de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

12 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477

13 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7fa581bf2700 (LWP 6862)):

0 0x00007fa58374849f in __libc_accept (fd=12, addr=..., len=0x0) at ../sysdeps/unix/sysv/linux/accept.c:26

1 0x00007fa582c4d8eb in socket_listener () from /lib/librte_telemetry.so.20.0

2 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477

3 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7fa5823f3700 (LWP 6861)):

0 0x00007fa5834675ce in epoll_wait (epfd=10, events=0x7fa5823f1800, maxevents=1, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30

1 0x00007fa58323ff4a in eal_intr_thread_main () from /lib/librte_eal.so.20.0

2 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477

3 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7fa582bf4700 (LWP 6860)):

0 0x00007fa5834253bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa582bf27d0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa58342b047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa582bf27d0, remaining=remaining@entry=0x0) at nanosleep.c:27

2 0x00007fa58345d9bf in usleep (useconds=) at ../sysdeps/posix/usleep.c:32

3 0x00007fa583a3a76a in bvar::detail::SamplerCollector::run (this=0x559a4adbb440) at /home/leo/code/thirdparty/brpc/src/bvar/detail/sampler.cpp:180

4 0x00007fa583a3c40d in bvar::detail::SamplerCollector::sampling_thread (arg=) at /home/leo/code/thirdparty/brpc/src/bvar/detail/sampler.cpp:110

5 0x00007fa58373d609 in start_thread (arg=) at pthread_create.c:477

6 0x00007fa583467293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7fa582c09d80 (LWP 6857)):

0 0x00007fa5834253bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7ffcfb885a80, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa58342b047 in __GI___nanosleep (requested_time=requested_time@entry=0x7ffcfb885a80, remaining=remaining@entry=0x0) at nanosleep.c:27

2 0x00007fa58345d9bf in usleep (useconds=useconds@entry=1000000) at ../sysdeps/posix/usleep.c:32

3 0x00007fa583a4d178 in bthread_usleep (microseconds=microseconds@entry=1000000) at /home/leo/code/thirdparty/brpc/src/bthread/bthread.cpp:363

4 0x00007fa583b40862 in brpc::Server::RunUntilAskedToQuit (this=0x7ffcfb885e00) at /home/leo/code/thirdparty/brpc/src/brpc/server.cpp:1693

5 0x0000559a49f1c8a9 in main (argc=2, argv=0x7ffcfb886478) at /root/jenkins_S/workspace/CI-CKS-Daily/src/main_control.cpp:125

Detaching from program: /home/leo/deploy/CKS/chunkserver, process 6857 [Inferior 1 (process 6857) detached]

经初步分析每个LWP持有mutex的竞争关系如下： 6871->6876 6872->6877
6873->6876
6874->6876
6875->6876
6876->6876
6877->6877 6878->6876
6879->6876

通过上面thread bt和锁的竞争关系 得出一种可能的执行序列导致死锁情况 bthread 1 调度执行NodeImpl::handle_append_entries_request 获得NodeImpl::_mutex LogManager::append_entries 获得LogManager::_mutex
然后通过bthread::execution_queue_execute(_disk_queue, done) 向_disk_queue提交任务 bthread 2 执行LogManager::disk_thread 调用AppendBatcher::flush 执行FollowerStableClosure::run 去获取NodeImpl::_mutex失败 bthread 3 执行LogManager::set_disk_id 获取LogManager::_mutex失败 至此 其他bthread 获取NodeImpl::_mutex和LogManager::_mutex都会失败 导致braft全面死锁 看来问题是bthread 1同时获得了NodeImpl::_mutex LogManager::_mutex锁 而又不能继续得到调度执行 导致braft死锁 braft可能存在bthread不能被调度的场景吗 期待大神分析解答

cl079074282 commented 3 years ago

补充一点：我们在log_manager中做了callback业务代码的动作

ehds commented 3 years ago

在LogManager::append_entries 里面，bthread::execution_queue_execute(_disk_queue, done) 提交磁盘任务，这里是异步的，提交任务后会立即返回，继续向下执行wakeup_all_waiter，这里会释放掉LogManager::_mutex，并返回到NodeImpl::handle_append_entries_request 释放掉NodeImpl::_mutex，应该是不会发生死锁。

cl079074282 commented 3 years ago

execution_queue_execute

在LogManager::append_entries 里面，bthread::execution_queue_execute(_disk_queue, done) 提交磁盘任务，这里是异步的，提交任务后会立即返回，继续向下执行wakeup_all_waiter，这里会释放掉LogManager::_mutex，并返回到NodeImpl::handle_append_entries_request 释放掉NodeImpl::_mutex，应该是不会发生死锁。

execution_queue_execute是异步执行的没错，但新的bthread去申请 LogManager::_mutex就发生了死锁，所以肯定还是某个bthread 一直持有mutex没有释放，上面的thread bt也是这样

PFZheng commented 3 years ago

bthread::pthread_mutex_lock_impl (mutex=) mutex 变量里有一个owner字段，是持有者的线程id，看一下这个线程在做什么

cl079074282 commented 3 years ago

bthread::pthread_mutex_lock_impl (mutex=) mutex 变量里有一个owner字段，是持有者的线程id，看一下这个线程在做什么

如上 我查看了所有死锁的thread mutex被占用情况 占用mutex的thread 也是lock_wait 所以才一直释放不出来