search

baidu / braft

An industrial-grade C++ implementation of RAFT consensus algorithm based on brpc, widely used inside Baidu to build highly-available distributed systems.
Apache License 2.0
3.95k stars 881 forks source link

braft 进程重启snapshot死锁问题 #317

Open cl079074282 opened 3 years ago

cl079074282 commented 3 years ago

Id Target Id Frame 1 Thread 0x7fa30b6b1dc0 (LWP 70908) "chunkserver" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 2 Thread 0x7fa30b69c700 (LWP 70909) "chunkserver" 0x00007fa30becf3bf in GI_clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa30b69a790, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 3 Thread 0x7fa30ae9b700 (LWP 70910) "eal-intr-thread" 0x00007fa30bf115ce in epoll_wait (epfd=11, events=0x7fa30ae997c0, maxevents=1, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30 4 Thread 0x7fa30a69a700 (LWP 70912) "chunkserver" 0x00007fa30c1f249f in libc_accept (fd=12, addr=..., len=0x0) at ../sysdeps/unix/sysv/linux/accept.c:26 5 Thread 0x7fa309896700 (LWP 70913) "container_thd" 0x00007fa30becf3bf in _GIclock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa3098946b0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 6 Thread 0x7fa2fbfff700 (LWP 70914) "EngineWorkThrea" lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::Running (this=0x555ae9575f10) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1201 7 Thread 0x7fa2fb7fe700 (LWP 70916) "StatisticUpdate" 0x00007fa30becf3bf in GI_clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa2fb7fc6f0, rem=rem@entry=0x7fa2fb7fc6f0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 8 Thread 0x7fa2faffd700 (LWP 70985) "DirtyBrickFlush" 0x00007fa30becf3bf in GI_clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa2faffb700, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 9 Thread 0x7fa2fa7fc700 (LWP 70986) "PageAlloterBack" futex_abstimed_wait_cancelable (private=, abstime=0x7fa2fa7fa660, clockid=, expected=0, futex_word=0x7fa300001cf8) at ../sysdeps/nptl/futex-internal.h:320 10 Thread 0x7fa2f9ffb700 (LWP 70989) "container_thd" syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38 11 Thread 0x7fa2f97fa700 (LWP 70990) "container_thd" 0x00007fa30becf3bf in GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa2a1250670, rem=rem@entry=0x7fa2a1250670) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 12 Thread 0x7fa2f8ff9700 (LWP 70991) "container_thd" 0x00007fa30becf3bf in GI_clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa2f85f6670, rem=rem@entry=0x7fa2f85f6670) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 13 Thread 0x7fa2ebfff700 (LWP 70992) "container_thd" 0x00007fa30becf3bf in GI_clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa2a104e670, rem=rem@entry=0x7fa2a104e670) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 14 Thread 0x7fa2eb7fe700 (LWP 70993) "container_thd" 0x00007fa30becf3bf in GI_clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa2be057670, rem=rem@entry=0x7fa2be057670) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78 15 Thread 0x7fa2eaffd700 (LWP 70994) "container_thd" lll_lock_wait (futex=futex@entry=0x7fa2b480dd40, private=0) at lowlevellock.c:52

Thread 29 (Thread 0x7fa28e7ae700 (LWP 71145)):

0 futex_abstimed_wait_cancelable (private=, abstime=0x7fa28e7ac660, clockid=, expected=0, futex_word=0x7fa29c001cf8) at ../sysdeps/nptl/futex-internal.h:320

1 __pthread_cond_wait_common (abstime=0x7fa28e7ac660, clockid=, mutex=0x7fa29c001ca8, cond=0x7fa29c001cd0) at pthread_cond_wait.c:520

2 __pthread_cond_timedwait (cond=0x7fa29c001cd0, mutex=0x7fa29c001ca8, abstime=0x7fa28e7ac660) at pthread_cond_wait.c:656

3 0x00007fa30d789bf2 in gthread_cond_timedwait (cond=0x7fa29c001cd0, mutex=0x7fa29c001ca8, abs_timeout=0x7fa28e7ac660) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:872

4 0x00007fa30d78c33d in std::condition_variable::wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fa29c001cd0, lock=..., __atime=...) at /usr/include/c++/9/condition_variable:188

5 0x00007fa30d78b7e1 in std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fa29c001cd0, lock=..., atime=...) at /usr/include/c++/9/condition_variable:121

6 0x00007fa30d78aeac in std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x7fa29c001cd0, lock=..., rtime=...) at /usr/include/c++/9/condition_variable:152

7 0x00007fa30d787916 in lse::chunkserver::PageAlloter::<lambda()>::operator()(void) const (__closure=0x7fa29c655ab8) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/page_allocator.cpp:45

8 0x00007fa30d7897e8 in std::invoke_impl<void, lse::chunkserver::PageAlloter::Start()::<lambda()> >(std::__invoke_other, lse::chunkserver::PageAlloter::<lambda()> &&) (f=...) at /usr/include/c++/9/bits/invoke.h:60

9 0x00007fa30d78973b in std::invoke<lse::chunkserver::PageAlloter::Start()::<lambda()> >(lse::chunkserver::PageAlloter::<lambda()> &&) (fn=...) at /usr/include/c++/9/bits/invoke.h:95

10 0x00007fa30d789699 in std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x7fa29c655ab8) at /usr/include/c++/9/thread:244

11 0x00007fa30d78961e in std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > >::operator()(void) (this=0x7fa29c655ab8) at /usr/include/c++/9/thread:251

12 0x00007fa30d7895e1 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > > >::_M_run(void) (this=0x7fa29c655ab0) at /usr/include/c++/9/thread:195

13 0x00007fa30c0d2de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

14 0x00007fa30c1e7609 in start_thread (arg=) at pthread_create.c:477

15 0x00007fa30bf11293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 28 (Thread 0x7fa28efaf700 (LWP 71144)):

0 0x00007fa30becf3bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa28efad700, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa30bed5047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa28efad700, remaining=remaining@entry=0x0) at nanosleep.c:27

2 0x00007fa30bf079bf in usleep (useconds=) at ../sysdeps/posix/usleep.c:32

3 0x00007fa30db04628 in lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher (this=0x7fa29c002090) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/l2pmap_mgr.h:734

4 0x00007fa30db35df7 in std::__invoke_impl<int, int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> (f=@0x7fa29c5657d0: (int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(class lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey> const)) 0x7fa30db044ee <lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher()>, t=@0x7fa29c5657c8: 0x7fa29c002090) at /usr/include/c++/9/bits/invoke.h:73

5 0x00007fa30db30595 in std::invoke<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> (fn=@0x7fa29c5657d0: (int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(class lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey> const)) 0x7fa30db044ee <lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher()>) at /usr/include/c++/9/bits/invoke.h:95

6 0x00007fa30db2b724 in std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> >::_M_invoke<0ul, 1ul> (this=0x7fa29c5657c8) at /usr/include/c++/9/thread:244

7 0x00007fa30db221a0 in std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> >::operator() (this=0x7fa29c5657c8) at /usr/include/c++/9/thread:251

8 0x00007fa30db20e97 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> > >::_M_run (this=0x7fa29c5657c0) at /usr/include/c++/9/thread:195

9 0x00007fa30c0d2de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

10 0x00007fa30c1e7609 in start_thread (arg=) at pthread_create.c:477

11 0x00007fa30bf11293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 27 (Thread 0x7fa2a255c700 (LWP 71011)):

0 0x00007fa30becf3bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa2a255a6f0, rem=rem@entry=0x7fa2a255a6f0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa30bed5047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa2a255a6f0, remaining=remaining@entry=0x7fa2a255a6f0) at nanosleep.c:27

2 0x00007fa30bed4f7e in __sleep (seconds=0) at ../sysdeps/posix/sleep.c:55

3 0x00007fa30db021d7 in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::UpdataStatisticInfo (this=0x555ae994c4e0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1982

4 0x00007fa30dafcebf in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}::operator()() const (this=0x555ae994c4e0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1158

5 0x00007fa30db35e84 in std::invoke_impl<void, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}>(std::__invoke_other, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}&&) (f=...) at /usr/include/c++/9/bits/invoke.h:60

6 0x00007fa30db30673 in std::invoke<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}>(std::invoke_result&&, (lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}&&)...) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95

7 0x00007fa30db2b77d in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7fa29c1a28f8) at /usr/include/c++/9/thread:244

8 0x00007fa30db2220a in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}> >::operator()() (this=0x7fa29c1a28f8) at /usr/include/c++/9/thread:251

9 0x00007fa30db20ecd in std::thread::_State_impl<std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}> > >::_M_run() (this=0x7fa29c1a28f0) at /usr/include/c++/9/thread:195

10 0x00007fa30c0d2de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

11 0x00007fa30c1e7609 in start_thread (arg=) at pthread_create.c:477

12 0x00007fa30bf11293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 26 (Thread 0x7fa2a2d5d700 (LWP 71009)):

0 nvme_pcie_poll_group_process_completions (tgroup=0x555ae994c490, completions_per_qpair=0, disconnected_qpair_cb=0x7fa30d78d8a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair, void)>) at nvme_pcie.c:2526

1 0x0000555ae87aecc9 in nvme_transport_poll_group_process_completions (tgroup=tgroup@entry=0x555ae994c490, completions_per_qpair=completions_per_qpair@entry=0, disconnected_qpair_cb=disconnected_qpair_cb@entry=0x7fa30d78d8a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair, void)>) at nvme_transport.c:499

2 0x0000555ae87b8c79 in spdk_nvme_poll_group_process_completions (group=, completions_per_qpair=0, disconnected_qpair_cb=0x7fa30d78d8a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair, void)>) at nvme_poll_group.c:127

3 0x00007fa30d790dfb in lse::chunkserver::SpdkNamespace::Complete (this=0x555ae994bff0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/spdk.cpp:408

4 0x00007fa30db01eb2 in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::Running (this=0x555ae994c4e0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1207

5 0x00007fa30dafce04 in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}::operator()() const (this=0x555ae994c4e0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1152

6 0x00007fa30db35f12 in std::invoke_impl<void, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}>(std::__invoke_other, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}&&) (f=...) at /usr/include/c++/9/bits/invoke.h:60

7 0x00007fa30db3074c in std::invoke<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}>(std::invoke_result&&, (lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}&&)...) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95

8 0x00007fa30db2b7d1 in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7fa29c1a26f8) at /usr/include/c++/9/thread:244

9 0x00007fa30db22276 in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> >::operator()() (this=0x7fa29c1a26f8) at /usr/include/c++/9/thread:251

10 0x00007fa30db20f03 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> > >::_M_run() (this=0x7fa29c1a26f0) at /usr/include/c++/9/thread:195

11 0x00007fa30c0d2de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

12 0x00007fa30c1e7609 in start_thread (arg=) at pthread_create.c:477

13 0x00007fa30bf11293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 25 (Thread 0x7fa2a3fff700 (LWP 71008)):

0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38

1 0x00007fa30c50117e in bthread::futex_wait_private (timeout=0x0, expected=0, addr1=0x7fa2a3ffcf40) at ./src/bthread/sys_futex.h:40

2 bthread::wait_pthread (pw=..., ptimeout=ptimeout@entry=0x0) at src/bthread/butex.cpp:141

3 0x00007fa30c501ca5 in bthread::butex_wait_from_pthread (abstime=0x0, expected_value=1, b=0x7fa29c1a3df0, g=) at src/bthread/butex.cpp:588

4 bthread::butex_wait (arg=0x7fa29c1a3df0, expected_value=1, abstime=abstime@entry=0x0) at src/bthread/butex.cpp:621

5 0x00007fa30c500bfa in bthread::CountdownEvent::wait (this=this@entry=0x7fa2a3ffd010) at src/bthread/countdown_event.cpp:64

6 0x00007fa30c48d8eb in braft::FirstSnapshotLoadDone::wait_for_run (this=0x7fa2a3ffcff0) at /home/kuhan/braft_720/thirdparty-master/braft/src/braft/snapshot_executor.cpp:390

7 braft::SnapshotExecutor::init (this=0x7fa29c8434e0, options=...) at /home/kuhan/braft_720/thirdparty-master/braft/src/braft/snapshot_executor.cpp:390

8 0x00007fa30c44165f in braft::NodeImpl::init_snapshot_storage (this=0x7fa29c82a360) at /home/kuhan/braft_720/thirdparty-master/braft/src/braft/node.cpp:251

9 0x00007fa30c457ada in braft::NodeImpl::init (this=0x7fa29c82a360, options=...) at /home/kuhan/braft_720/thirdparty-master/braft/src/braft/node.cpp:529

10 0x00007fa30d8b0bc3 in lse::chunkserver::Copyset::Start (this=0x7fa29c67e990, member_info=..., node_options=0x7fa2a3ffd460) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset.cpp:149

11 0x00007fa30d8c2658 in lse::chunkserver::CopysetManager::CreateCopyset (this=0x7fa29c6768d0, group_id=16, member_info=..., node_options=0x0, restart=true) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:116

12 0x00007fa30d8c1d98 in lse::chunkserver::CopysetManager::Start (this=0x7fa29c6768d0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset_manager.cpp:43

13 0x00007fa30daf07d9 in lse::chunkserver::Container::execute (this=0x555ae97d9980) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/container.cpp:177

14 0x00007fa30db0d9a8 in std::__invoke_impl<void, void (lse::chunkserver::Container::)(), lse::chunkserver::Container> (f=@0x555ae97aeff0: (void (lse::chunkserver::Container::)(class lse::chunkserver::Container const)) 0x7fa30daef8f8 <lse::chunkserver::Container::execute()>, t=@0x555ae97aefe8: 0x555ae97d9980) at /usr/include/c++/9/bits/invoke.h:73

15 0x00007fa30db0771e in std::invoke<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> (fn=@0x555ae97aeff0: (void (lse::chunkserver::Container::)(class lse::chunkserver::Container const)) 0x7fa30daef8f8 <lse::chunkserver::Container::execute()>) at /usr/include/c++/9/bits/invoke.h:95

16 0x00007fa30dafff7e in std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> >::_M_invoke<0ul, 1ul> (this=0x555ae97aefe8) at /usr/include/c++/9/thread:244

17 0x00007fa30dafc3ee in std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> >::operator() (this=0x555ae97aefe8) at /usr/include/c++/9/thread:251

18 0x00007fa30daf73af in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> > >::_M_run (this=0x555ae97aefe0) at /usr/include/c++/9/thread:195

19 0x00007fa30c0d2de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

20 0x00007fa30c1e7609 in start_thread (arg=) at pthread_create.c:477

21 0x00007fa30bf11293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 24 (Thread 0x7fa2bc84d700 (LWP 71006)):

0 futex_abstimed_wait_cancelable (private=, abstime=0x7fa2bc84b660, clockid=, expected=0, futex_word=0x7fa2b4001cf8) at ../sysdeps/nptl/futex-internal.h:320

1 __pthread_cond_wait_common (abstime=0x7fa2bc84b660, clockid=, mutex=0x7fa2b4001ca8, cond=0x7fa2b4001cd0) at pthread_cond_wait.c:520

2 __pthread_cond_timedwait (cond=0x7fa2b4001cd0, mutex=0x7fa2b4001ca8, abstime=0x7fa2bc84b660) at pthread_cond_wait.c:656

3 0x00007fa30d789bf2 in gthread_cond_timedwait (cond=0x7fa2b4001cd0, mutex=0x7fa2b4001ca8, abs_timeout=0x7fa2bc84b660) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:872

4 0x00007fa30d78c33d in std::condition_variable::wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fa2b4001cd0, lock=..., __atime=...) at /usr/include/c++/9/condition_variable:188

5 0x00007fa30d78b7e1 in std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fa2b4001cd0, lock=..., atime=...) at /usr/include/c++/9/condition_variable:121

6 0x00007fa30d78aeac in std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x7fa2b4001cd0, lock=..., rtime=...) at /usr/include/c++/9/condition_variable:152

7 0x00007fa30d787916 in lse::chunkserver::PageAlloter::<lambda()>::operator()(void) const (__closure=0x7fa2b4218158) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/page_allocator.cpp:45

8 0x00007fa30d7897e8 in std::invoke_impl<void, lse::chunkserver::PageAlloter::Start()::<lambda()> >(std::__invoke_other, lse::chunkserver::PageAlloter::<lambda()> &&) (f=...) at /usr/include/c++/9/bits/invoke.h:60

9 0x00007fa30d78973b in std::invoke<lse::chunkserver::PageAlloter::Start()::<lambda()> >(lse::chunkserver::PageAlloter::<lambda()> &&) (fn=...) at /usr/include/c++/9/bits/invoke.h:95

10 0x00007fa30d789699 in std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x7fa2b4218158) at /usr/include/c++/9/thread:244

11 0x00007fa30d78961e in std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > >::operator()(void) (this=0x7fa2b4218158) at /usr/include/c++/9/thread:251

12 0x00007fa30d7895e1 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > > >::_M_run(void) (this=0x7fa2b4218150) at /usr/include/c++/9/thread:195

13 0x00007fa30c0d2de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

14 0x00007fa30c1e7609 in start_thread (arg=) at pthread_create.c:477

15 0x00007fa30bf11293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 23 (Thread 0x7fa2bd04e700 (LWP 71005)):

0 0x00007fa30becf3bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa2bd04c700, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa30bed5047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa2bd04c700, remaining=remaining@entry=0x0) at nanosleep.c:27

2 0x00007fa30bf079bf in usleep (useconds=) at ../sysdeps/posix/usleep.c:32

3 0x00007fa30db04628 in lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher (this=0x7fa2b4002090) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/l2pmap_mgr.h:734

4 0x00007fa30db35df7 in std::__invoke_impl<int, int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> (f=@0x7fa2b433ddf0: (int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(class lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey> const)) 0x7fa30db044ee <lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher()>, t=@0x7fa2b433dde8: 0x7fa2b4002090) at /usr/include/c++/9/bits/invoke.h:73

5 0x00007fa30db30595 in std::invoke<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> (fn=@0x7fa2b433ddf0: (int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(class lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey> const)) 0x7fa30db044ee <lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher()>) at /usr/include/c++/9/bits/invoke.h:95

6 0x00007fa30db2b724 in std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> >::_M_invoke<0ul, 1ul> (this=0x7fa2b433dde8) at /usr/include/c++/9/thread:244

7 0x00007fa30db221a0 in std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> >::operator() (this=0x7fa2b433dde8) at /usr/include/c++/9/thread:251

8 0x00007fa30db20e97 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> > >::_M_run (this=0x7fa2b433dde0) at /usr/include/c++/9/thread:195

9 0x00007fa30c0d2de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

10 0x00007fa30c1e7609 in start_thread (arg=) at pthread_create.c:477

11 0x00007fa30bf11293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 22 (Thread 0x7fa2bd84f700 (LWP 71002)):

0 0x00007fa30becf3bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa2bd84d6f0, rem=rem@entry=0x7fa2bd84d6f0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa30bed5047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa2bd84d6f0, remaining=remaining@entry=0x7fa2bd84d6f0) at nanosleep.c:27

2 0x00007fa30bed4f7e in __sleep (seconds=0) at ../sysdeps/posix/sleep.c:55

3 0x00007fa30db021d7 in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::UpdataStatisticInfo (this=0x555ae97a6590) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1982

4 0x00007fa30dafcebf in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}::operator()() const (this=0x555ae97a6590) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1158

5 0x00007fa30db35e84 in std::invoke_impl<void, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}>(std::__invoke_other, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}&&) (f=...) at /usr/include/c++/9/bits/invoke.h:60

6 0x00007fa30db30673 in std::invoke<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}>(std::invoke_result&&, (lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}&&)...) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95

7 0x00007fa30db2b77d in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7fa2b41a28f8) at /usr/include/c++/9/thread:244

8 0x00007fa30db2220a in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}> >::operator()() (this=0x7fa2b41a28f8) at /usr/include/c++/9/thread:251

9 0x00007fa30db20ecd in std::thread::_State_impl<std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}> > >::_M_run() (this=0x7fa2b41a28f0) at /usr/include/c++/9/thread:195

10 0x00007fa30c0d2de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

11 0x00007fa30c1e7609 in start_thread (arg=) at pthread_create.c:477

12 0x00007fa30bf11293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 21 (Thread 0x7fa2bed5d700 (LWP 71001)):

0 0x0000555ae87a8046 in nvme_pcie_qpair_process_completions (qpair=0x2000014a7d18, max_completions=0) at nvme_pcie.c:2378

1 0x0000555ae87ab414 in spdk_nvme_qpair_process_completions (qpair=qpair@entry=0x2000014a7d18, max_completions=max_completions@entry=0) at nvme_qpair.c:714

2 0x0000555ae87a61cc in nvme_pcie_poll_group_process_completions (tgroup=0x555ae9799e20, completions_per_qpair=0, disconnected_qpair_cb=0x7fa30d78d8a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair, void)>) at nvme_pcie.c:2536

3 0x0000555ae87aecc9 in nvme_transport_poll_group_process_completions (tgroup=tgroup@entry=0x555ae9799e20, completions_per_qpair=completions_per_qpair@entry=0, disconnected_qpair_cb=disconnected_qpair_cb@entry=0x7fa30d78d8a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair, void)>) at nvme_transport.c:499

4 0x0000555ae87b8c79 in spdk_nvme_poll_group_process_completions (group=, completions_per_qpair=0, disconnected_qpair_cb=0x7fa30d78d8a8 <lse::chunkserver::NamespaceDisconnectCallback(spdk_nvme_qpair, void)>) at nvme_poll_group.c:127

5 0x00007fa30d790dfb in lse::chunkserver::SpdkNamespace::Complete (this=0x555ae97ddba0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/spdk.cpp:408

6 0x00007fa30db01eb2 in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::Running (this=0x555ae97a6590) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1207

7 0x00007fa30dafce04 in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}::operator()() const (this=0x555ae97a6590) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1152

8 0x00007fa30db35f12 in std::invoke_impl<void, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}>(std::__invoke_other, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}&&) (f=...) at /usr/include/c++/9/bits/invoke.h:60

9 0x00007fa30db3074c in std::invoke<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}>(std::invoke_result&&, (lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}&&)...) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95

10 0x00007fa30db2b7d1 in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7fa2b41a26f8) at /usr/include/c++/9/thread:244

11 0x00007fa30db22276 in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> >::operator()() (this=0x7fa2b41a26f8) at /usr/include/c++/9/thread:251

12 0x00007fa30db20f03 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> > >::_M_run() (this=0x7fa2b41a26f0) at /usr/include/c++/9/thread:195

13 0x00007fa30c0d2de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

14 0x00007fa30c1e7609 in start_thread (arg=) at pthread_create.c:477

15 0x00007fa30bf11293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 20 (Thread 0x7fa2bffff700 (LWP 71000)):

0 0x00007fa30becf3bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa2bfffd6b0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa30bed5047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa2bfffd6b0, remaining=remaining@entry=0x0) at nanosleep.c:27

2 0x00007fa30bf079bf in usleep (useconds=useconds@entry=1000000) at ../sysdeps/posix/usleep.c:32

3 0x00007fa30c5212c8 in bthread_usleep (microseconds=microseconds@entry=1000000) at src/bthread/bthread.cpp:363

4 0x00007fa30c527b92 in brpc::Server::RunUntilAskedToQuit (this=0x7fa2b42b19c0) at src/brpc/server.cpp:1693

5 0x00007fa30daf0859 in lse::chunkserver::Container::execute (this=0x555ae97d07e0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/container.cpp:182

6 0x00007fa30db0d9a8 in std::__invoke_impl<void, void (lse::chunkserver::Container::)(), lse::chunkserver::Container> (f=@0x555ae97caa20: (void (lse::chunkserver::Container::)(class lse::chunkserver::Container const)) 0x7fa30daef8f8 <lse::chunkserver::Container::execute()>, t=@0x555ae97caa18: 0x555ae97d07e0) at /usr/include/c++/9/bits/invoke.h:73

7 0x00007fa30db0771e in std::invoke<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> (fn=@0x555ae97caa20: (void (lse::chunkserver::Container::)(class lse::chunkserver::Container const)) 0x7fa30daef8f8 <lse::chunkserver::Container::execute()>) at /usr/include/c++/9/bits/invoke.h:95

8 0x00007fa30dafff7e in std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> >::_M_invoke<0ul, 1ul> (this=0x555ae97caa18) at /usr/include/c++/9/thread:244

9 0x00007fa30dafc3ee in std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> >::operator() (this=0x555ae97caa18) at /usr/include/c++/9/thread:251

10 0x00007fa30daf73af in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> > >::_M_run (this=0x555ae97caa10) at /usr/include/c++/9/thread:195

11 0x00007fa30c0d2de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

12 0x00007fa30c1e7609 in start_thread (arg=) at pthread_create.c:477

13 0x00007fa30bf11293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 19 (Thread 0x7fa2e8ff9700 (LWP 70998)):

0 __lll_lock_wait (futex=futex@entry=0x7fa2c408fc90, private=0) at lowlevellock.c:52

1 0x00007fa30c1ea0a3 in __GI___pthread_mutex_lock (mutex=0x7fa2c408fc90) at ../nptl/pthread_mutex_lock.c:80

2 0x00007fa30c513267 in bthread::pthread_mutex_lock_impl (mutex=) at src/bthread/mutex.cpp:549

3 pthread_mutex_lock (__mutex=) at src/bthread/mutex.cpp:812

4 pthread_mutex_lock (__mutex=0x7fa2c408fc90) at src/bthread/mutex.cpp:811

5 0x00007fa30c486564 in butil::Mutex::lock (this=) at /usr/local/include/butil/synchronization/lock.h:69

6 std::unique_lock::lock (this=0x7fa290cccda0, this=0x7fa290cccda0) at /usr/include/c++/9/bits/unique_lock.h:141

7 braft::LocalSnapshotCopier::copy_file (this=0x7fa2c408fc80, filename="data") at /home/kuhan/braft_720/thirdparty-master/braft/src/braft/snapshot.cpp:986

8 0x00007fa30c488be2 in braft::LocalSnapshotCopier::copy (this=0x7fa2c408fc80) at /usr/include/c++/9/bits/stl_vector.h:1040

9 0x00007fa30c488e3d in braft::LocalSnapshotCopier::start_copy (arg=) at /home/kuhan/braft_720/thirdparty-master/braft/src/braft/snapshot.cpp:754

10 0x00007fa30c519d1f in bthread::TaskGroup::task_runner (skip_remained=) at src/bthread/task_group.cpp:295

11 0x00007fa30c51d0f1 in bthread_make_fcontext () at /usr/include/c++/9/bits/basic_ios.h:282

12 0x0000000000000000 in ?? ()

Thread 18 (Thread 0x7fa2e97fa700 (LWP 70997)):

0 0x00007fa30becf3bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa2931f1670, rem=rem@entry=0x7fa2931f1670) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa30bed5047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa2931f1670, remaining=remaining@entry=0x7fa2931f1670) at nanosleep.c:27

2 0x00007fa30bed4f7e in __sleep (seconds=0) at ../sysdeps/posix/sleep.c:55

3 0x00007fa30d8b26b2 in lse::chunkserver::Copyset::on_snapshot_load (this=0x7fa2b483e5f0, reader=0x7fa2c8438900) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset.cpp:304

4 0x00007fa30c41a4d5 in braft::FSMCaller::do_snapshot_load (this=0x7fa2b483e1b0, done=0x7fa2c8146860) at /home/kuhan/braft_720/thirdparty-master/braft/src/braft/fsm_caller.cpp:402

5 0x00007fa30c41aa72 in braft::FSMCaller::run (meta=0x7fa2b483e1b0, iter=...) at /usr/local/include/bthread/execution_queue.h:61

6 0x00007fa30c5092c0 in bthread::ExecutionQueueBase::_execute (this=0x7fa2b47e2e80, head=0x7fa2d01b74c0, high_priority=, niterated=0x0) at src/bthread/execution_queue.cpp:272

7 0x00007fa30c50ade0 in bthread::ExecutionQueueBase::_execute_tasks (arg=) at src/bthread/execution_queue.cpp:151

8 0x00007fa30c519d1f in bthread::TaskGroup::task_runner (skip_remained=) at src/bthread/task_group.cpp:295

9 0x00007fa30c51d0f1 in bthread_make_fcontext () at /usr/include/c++/9/bits/basic_ios.h:282

10 0x0000000000000000 in ?? ()

Thread 17 (Thread 0x7fa2e9ffb700 (LWP 70996)):

0 0x00007fa30becf3bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa3081dc670, rem=rem@entry=0x7fa3081dc670) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa30bed5047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa3081dc670, remaining=remaining@entry=0x7fa3081dc670) at nanosleep.c:27

2 0x00007fa30bed4f7e in __sleep (seconds=0) at ../sysdeps/posix/sleep.c:55

3 0x00007fa30d8b26b2 in lse::chunkserver::Copyset::on_snapshot_load (this=0x7fa300b2dbd0, reader=0x7fa2c0046de0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset.cpp:304

4 0x00007fa30c41a4d5 in braft::FSMCaller::do_snapshot_load (this=0x7fa300b2d790, done=0x7fa2c0046580) at /home/kuhan/braft_720/thirdparty-master/braft/src/braft/fsm_caller.cpp:402

5 0x00007fa30c41aa72 in braft::FSMCaller::run (meta=0x7fa300b2d790, iter=...) at /usr/local/include/bthread/execution_queue.h:61

6 0x00007fa30c5092c0 in bthread::ExecutionQueueBase::_execute (this=0x7fa3009c3520, head=0x7fa2c001af10, high_priority=, niterated=0x0) at src/bthread/execution_queue.cpp:272

7 0x00007fa30c50ade0 in bthread::ExecutionQueueBase::_execute_tasks (arg=) at src/bthread/execution_queue.cpp:151

8 0x00007fa30c519d1f in bthread::TaskGroup::task_runner (skip_remained=) at src/bthread/task_group.cpp:295

9 0x00007fa30c51d0f1 in bthread_make_fcontext () at /usr/include/c++/9/bits/basic_ios.h:282

10 0x0000000000000000 in ?? ()

Thread 16 (Thread 0x7fa2ea7fc700 (LWP 70995)):

0 __lll_lock_wait (futex=futex@entry=0x7fa2c41e3ef0, private=0) at lowlevellock.c:52

1 0x00007fa30c1ea0a3 in __GI___pthread_mutex_lock (mutex=0x7fa2c41e3ef0) at ../nptl/pthread_mutex_lock.c:80

2 0x00007fa30c513267 in bthread::pthread_mutex_lock_impl (mutex=) at src/bthread/mutex.cpp:549

3 pthread_mutex_lock (__mutex=) at src/bthread/mutex.cpp:812

4 pthread_mutex_lock (__mutex=0x7fa2c41e3ef0) at src/bthread/mutex.cpp:811

5 0x00007fa30c486564 in butil::Mutex::lock (this=) at /usr/local/include/butil/synchronization/lock.h:69

6 std::unique_lock::lock (this=0x7fa2906c6da0, this=0x7fa2906c6da0) at /usr/include/c++/9/bits/unique_lock.h:141

7 braft::LocalSnapshotCopier::copy_file (this=0x7fa2c41e3ee0, filename="data") at /home/kuhan/braft_720/thirdparty-master/braft/src/braft/snapshot.cpp:986

8 0x00007fa30c488be2 in braft::LocalSnapshotCopier::copy (this=0x7fa2c41e3ee0) at /usr/include/c++/9/bits/stl_vector.h:1040

9 0x00007fa30c488e3d in braft::LocalSnapshotCopier::start_copy (arg=) at /home/kuhan/braft_720/thirdparty-master/braft/src/braft/snapshot.cpp:754

10 0x00007fa30c519d1f in bthread::TaskGroup::task_runner (skip_remained=) at src/bthread/task_group.cpp:295

11 0x00007fa30c51d0f1 in bthread_make_fcontext () at /usr/include/c++/9/bits/basic_ios.h:282

12 0x0000000000000000 in ?? ()

Thread 15 (Thread 0x7fa2eaffd700 (LWP 70994)):

0 __lll_lock_wait (futex=futex@entry=0x7fa2b480dd40, private=0) at lowlevellock.c:52

1 0x00007fa30c1ea0a3 in __GI___pthread_mutex_lock (mutex=0x7fa2b480dd40) at ../nptl/pthread_mutex_lock.c:80

2 0x00007fa30c513267 in bthread::pthread_mutex_lock_impl (mutex=) at src/bthread/mutex.cpp:549

3 pthread_mutex_lock (__mutex=) at src/bthread/mutex.cpp:812

4 pthread_mutex_lock (mutex=mutex@entry=0x7fa2b480dd40) at src/bthread/mutex.cpp:811

5 0x00007fa30c44e29a in butil::Mutex::lock (this=0x7fa2b480dd40) at /usr/local/include/butil/synchronization/lock.h:69

6 std::unique_lock::lock (this=0x7fa2a0240b50, this=0x7fa2a0240b50) at /usr/include/c++/9/bits/unique_lock.h:141

7 std::unique_lock::unique_lock (__m=..., this=0x7fa2a0240b50) at /usr/include/c++/9/bits/unique_lock.h:71

8 braft::NodeImpl::handle_append_entries_request (this=0x7fa2b480daa0, cntl=cntl@entry=0x7fa2cc043710, request=request@entry=0x7fa2d03a2430, response=response@entry=0x7fa2b48097a0, done=done@entry=0x7fa2c405d200, from_append_entries_cache=from_append_entries_cache@entry=false) at /home/kuhan/braft_720/thirdparty-master/braft/src/braft/node.cpp:2208

9 0x00007fa30c46e08a in braft::RaftServiceImpl::append_entries (this=, cntl_base=0x7fa2cc043710, request=0x7fa2d03a2430, response=0x7fa2b48097a0, done=0x7fa2c405d200) at /usr/local/include/brpc/closure_guard.h:55

10 0x00007fa30c4b1231 in braft::RaftService::CallMethod (this=, method=, controller=, request=, response=, done=) at /home/kuhan/braft_720/thirdparty-master/braft/build/braft/raft.pb.cc:4813

11 0x00007fa30c5ad483 in brpc::policy::ProcessRpcRequest (msg_base=0x7fa2c00864f0) at /usr/include/c++/9/bits/unique_ptr.h:381

12 0x00007fa30c55bdfb in brpc::ProcessInputMessage (void_arg=) at src/brpc/input_messenger.cpp:135

13 0x00007fa30c519d1f in bthread::TaskGroup::task_runner (skip_remained=) at src/bthread/task_group.cpp:295

14 0x00007fa30c51d0f1 in bthread_make_fcontext () at /usr/include/c++/9/bits/basic_ios.h:282

15 0x0000000000000000 in ?? ()

Thread 14 (Thread 0x7fa2eb7fe700 (LWP 70993)):

0 0x00007fa30becf3bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa2be057670, rem=rem@entry=0x7fa2be057670) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa30bed5047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa2be057670, remaining=remaining@entry=0x7fa2be057670) at nanosleep.c:27

2 0x00007fa30bed4f7e in __sleep (seconds=0) at ../sysdeps/posix/sleep.c:55

3 0x00007fa30d8b26b2 in lse::chunkserver::Copyset::on_snapshot_load (this=0x7fa2b46376a0, reader=0x7fa2dc3c6d30) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset.cpp:304

4 0x00007fa30c41a4d5 in braft::FSMCaller::do_snapshot_load (this=0x7fa2b461e130, done=0x7fa2d806c7a0) at /home/kuhan/braft_720/thirdparty-master/braft/src/braft/fsm_caller.cpp:402

5 0x00007fa30c41aa72 in braft::FSMCaller::run (meta=0x7fa2b461e130, iter=...) at /usr/local/include/bthread/execution_queue.h:61

6 0x00007fa30c5092c0 in bthread::ExecutionQueueBase::_execute (this=0x7fa2b47dc580, head=0x7fa2d87fc330, high_priority=, niterated=0x0) at src/bthread/execution_queue.cpp:272

7 0x00007fa30c50ade0 in bthread::ExecutionQueueBase::_execute_tasks (arg=) at src/bthread/execution_queue.cpp:151

8 0x00007fa30c519d1f in bthread::TaskGroup::task_runner (skip_remained=) at src/bthread/task_group.cpp:295

9 0x00007fa30c51d0f1 in bthread_make_fcontext () at /usr/include/c++/9/bits/basic_ios.h:282

10 0x0000000000000000 in ?? ()

Thread 13 (Thread 0x7fa2ebfff700 (LWP 70992)):

0 0x00007fa30becf3bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa2a104e670, rem=rem@entry=0x7fa2a104e670) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa30bed5047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa2a104e670, remaining=remaining@entry=0x7fa2a104e670) at nanosleep.c:27

2 0x00007fa30bed4f7e in __sleep (seconds=0) at ../sysdeps/posix/sleep.c:55

3 0x00007fa30d8b26b2 in lse::chunkserver::Copyset::on_snapshot_load (this=0x7fa2b480fcf0, reader=0x7fa2c8438a80) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset.cpp:304

4 0x00007fa30c41a4d5 in braft::FSMCaller::do_snapshot_load (this=0x7fa2b480f1a0, done=0x7fa2c4323500) at /home/kuhan/braft_720/thirdparty-master/braft/src/braft/fsm_caller.cpp:402

5 0x00007fa30c41aa72 in braft::FSMCaller::run (meta=0x7fa2b480f1a0, iter=...) at /usr/local/include/bthread/execution_queue.h:61

6 0x00007fa30c5092c0 in bthread::ExecutionQueueBase::_execute (this=0x7fa2b47de980, head=0x7fa2c80329a0, high_priority=, niterated=0x0) at src/bthread/execution_queue.cpp:272

7 0x00007fa30c50ade0 in bthread::ExecutionQueueBase::_execute_tasks (arg=) at src/bthread/execution_queue.cpp:151

8 0x00007fa30c519d1f in bthread::TaskGroup::task_runner (skip_remained=) at src/bthread/task_group.cpp:295

9 0x00007fa30c51d0f1 in bthread_make_fcontext () at /usr/include/c++/9/bits/basic_ios.h:282

10 0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7fa2f8ff9700 (LWP 70991)):

0 0x00007fa30becf3bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa2f85f6670, rem=rem@entry=0x7fa2f85f6670) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa30bed5047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa2f85f6670, remaining=remaining@entry=0x7fa2f85f6670) at nanosleep.c:27

2 0x00007fa30bed4f7e in __sleep (seconds=0) at ../sysdeps/posix/sleep.c:55

3 0x00007fa30d8b26b2 in lse::chunkserver::Copyset::on_snapshot_load (this=0x7fa2b481b570, reader=0x7fa2c00cf390) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset.cpp:304

4 0x00007fa30c41a4d5 in braft::FSMCaller::do_snapshot_load (this=0x7fa2b481adb0, done=0x7fa2c009a610) at /home/kuhan/braft_720/thirdparty-master/braft/src/braft/fsm_caller.cpp:402

5 0x00007fa30c41aa72 in braft::FSMCaller::run (meta=0x7fa2b481adb0, iter=...) at /usr/local/include/bthread/execution_queue.h:61

6 0x00007fa30c5092c0 in bthread::ExecutionQueueBase::_execute (this=0x7fa2b47dfb80, head=0x7fa2c81b1d20, high_priority=, niterated=0x0) at src/bthread/execution_queue.cpp:272

7 0x00007fa30c50ade0 in bthread::ExecutionQueueBase::_execute_tasks (arg=) at src/bthread/execution_queue.cpp:151

8 0x00007fa30c519d1f in bthread::TaskGroup::task_runner (skip_remained=) at src/bthread/task_group.cpp:295

9 0x00007fa30c51d0f1 in bthread_make_fcontext () at /usr/include/c++/9/bits/basic_ios.h:282

10 0x0000000000000000 in ?? ()

Thread 11 (Thread 0x7fa2f97fa700 (LWP 70990)):

0 0x00007fa30becf3bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa2a1250670, rem=rem@entry=0x7fa2a1250670) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa30bed5047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa2a1250670, remaining=remaining@entry=0x7fa2a1250670) at nanosleep.c:27

2 0x00007fa30bed4f7e in __sleep (seconds=0) at ../sysdeps/posix/sleep.c:55

3 0x00007fa30d8b26b2 in lse::chunkserver::Copyset::on_snapshot_load (this=0x7fa2b47f4bf0, reader=0x7fa2d003e600) at /root/jenkins_S/workspace/CI-CKS-Daily/src/consistency/copyset.cpp:304

4 0x00007fa30c41a4d5 in braft::FSMCaller::do_snapshot_load (this=0x7fa2b47faa30, done=0x7fa2dc3c7950) at /home/kuhan/braft_720/thirdparty-master/braft/src/braft/fsm_caller.cpp:402

5 0x00007fa30c41aa72 in braft::FSMCaller::run (meta=0x7fa2b47faa30, iter=...) at /usr/local/include/bthread/execution_queue.h:61

6 0x00007fa30c5092c0 in bthread::ExecutionQueueBase::_execute (this=0x7fa2b47dc880, head=0x7fa2d02e4390, high_priority=, niterated=0x0) at src/bthread/execution_queue.cpp:272

7 0x00007fa30c50ade0 in bthread::ExecutionQueueBase::_execute_tasks (arg=) at src/bthread/execution_queue.cpp:151

8 0x00007fa30c519d1f in bthread::TaskGroup::task_runner (skip_remained=) at src/bthread/task_group.cpp:295

9 0x00007fa30c51d0f1 in bthread_make_fcontext () at /usr/include/c++/9/bits/basic_ios.h:282

10 0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7fa2f9ffb700 (LWP 70989)):

0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38

1 0x00007fa30c51f318 in bthread::futex_wait_private (timeout=, expected=2280, addr1=0x7fa300224bc8) at ./src/bthread/sys_futex.h:40

2 bthread::TimerThread::run (this=) at src/bthread/timer_thread.cpp:431

3 0x00007fa30c51fddd in bthread::TimerThread::run_this (arg=) at src/bthread/timer_thread.cpp:120

4 0x00007fa30c1e7609 in start_thread (arg=) at pthread_create.c:477

5 0x00007fa30bf11293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 9 (Thread 0x7fa2fa7fc700 (LWP 70986)):

0 futex_abstimed_wait_cancelable (private=, abstime=0x7fa2fa7fa660, clockid=, expected=0, futex_word=0x7fa300001cf8) at ../sysdeps/nptl/futex-internal.h:320

1 __pthread_cond_wait_common (abstime=0x7fa2fa7fa660, clockid=, mutex=0x7fa300001ca8, cond=0x7fa300001cd0) at pthread_cond_wait.c:520

2 __pthread_cond_timedwait (cond=0x7fa300001cd0, mutex=0x7fa300001ca8, abstime=0x7fa2fa7fa660) at pthread_cond_wait.c:656

3 0x00007fa30d789bf2 in gthread_cond_timedwait (cond=0x7fa300001cd0, mutex=0x7fa300001ca8, abs_timeout=0x7fa2fa7fa660) at /usr/include/x86_64-linux-gnu/c++/9/bits/gthr-default.h:872

4 0x00007fa30d78c33d in std::condition_variable::wait_until_impl<std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fa300001cd0, lock=..., __atime=...) at /usr/include/c++/9/condition_variable:188

5 0x00007fa30d78b7e1 in std::condition_variable::wait_until<std::chrono::_V2::steady_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > > (this=0x7fa300001cd0, lock=..., atime=...) at /usr/include/c++/9/condition_variable:121

6 0x00007fa30d78aeac in std::condition_variable::wait_for<long, std::ratio<1l, 1000l> > (this=0x7fa300001cd0, lock=..., rtime=...) at /usr/include/c++/9/condition_variable:152

7 0x00007fa30d787916 in lse::chunkserver::PageAlloter::<lambda()>::operator()(void) const (__closure=0x7fa3002209f8) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/page_allocator.cpp:45

8 0x00007fa30d7897e8 in std::invoke_impl<void, lse::chunkserver::PageAlloter::Start()::<lambda()> >(std::__invoke_other, lse::chunkserver::PageAlloter::<lambda()> &&) (f=...) at /usr/include/c++/9/bits/invoke.h:60

9 0x00007fa30d78973b in std::invoke<lse::chunkserver::PageAlloter::Start()::<lambda()> >(lse::chunkserver::PageAlloter::<lambda()> &&) (fn=...) at /usr/include/c++/9/bits/invoke.h:95

10 0x00007fa30d789699 in std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x7fa3002209f8) at /usr/include/c++/9/thread:244

11 0x00007fa30d78961e in std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > >::operator()(void) (this=0x7fa3002209f8) at /usr/include/c++/9/thread:251

12 0x00007fa30d7895e1 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<lse::chunkserver::PageAlloter::Start()::<lambda()> > > >::_M_run(void) (this=0x7fa3002209f0) at /usr/include/c++/9/thread:195

13 0x00007fa30c0d2de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

14 0x00007fa30c1e7609 in start_thread (arg=) at pthread_create.c:477

15 0x00007fa30bf11293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 8 (Thread 0x7fa2faffd700 (LWP 70985)):

0 0x00007fa30becf3bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa2faffb700, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa30bed5047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa2faffb700, remaining=remaining@entry=0x0) at nanosleep.c:27

2 0x00007fa30bf079bf in usleep (useconds=) at ../sysdeps/posix/usleep.c:32

3 0x00007fa30db04628 in lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher (this=0x7fa300002090) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/l2pmap_mgr.h:734

4 0x00007fa30db35df7 in std::__invoke_impl<int, int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> (f=@0x7fa30054db10: (int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(class lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey> const)) 0x7fa30db044ee <lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher()>, t=@0x7fa30054db08: 0x7fa300002090) at /usr/include/c++/9/bits/invoke.h:73

5 0x00007fa30db30595 in std::invoke<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> (fn=@0x7fa30054db10: (int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(class lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey> const)) 0x7fa30db044ee <lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::RunningFlusher()>) at /usr/include/c++/9/bits/invoke.h:95

6 0x00007fa30db2b724 in std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> >::_M_invoke<0ul, 1ul> (this=0x7fa30054db08) at /usr/include/c++/9/thread:244

7 0x00007fa30db221a0 in std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> >::operator() (this=0x7fa30054db08) at /usr/include/c++/9/thread:251

8 0x00007fa30db20e97 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<int (lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::)(), lse::chunkserver::L2PMapMgr<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>> > >::_M_run (this=0x7fa30054db00) at /usr/include/c++/9/thread:195

9 0x00007fa30c0d2de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

10 0x00007fa30c1e7609 in start_thread (arg=) at pthread_create.c:477

11 0x00007fa30bf11293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 7 (Thread 0x7fa2fb7fe700 (LWP 70916)):

0 0x00007fa30becf3bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa2fb7fc6f0, rem=rem@entry=0x7fa2fb7fc6f0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa30bed5047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa2fb7fc6f0, remaining=remaining@entry=0x7fa2fb7fc6f0) at nanosleep.c:27

2 0x00007fa30bed4f7e in __sleep (seconds=0) at ../sysdeps/posix/sleep.c:55

3 0x00007fa30db021d7 in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::UpdataStatisticInfo (this=0x555ae9575f10) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1982

4 0x00007fa30dafcebf in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}::operator()() const (this=0x555ae9575f10) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1158

5 0x00007fa30db35e84 in std::invoke_impl<void, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}>(std::__invoke_other, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}&&) (f=...) at /usr/include/c++/9/bits/invoke.h:60

6 0x00007fa30db30673 in std::invoke<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}>(std::invoke_result&&, (lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}&&)...) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95

7 0x00007fa30db2b77d in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7fa3001a28f8) at /usr/include/c++/9/thread:244

8 0x00007fa30db2220a in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}> >::operator()() (this=0x7fa3001a28f8) at /usr/include/c++/9/thread:251

9 0x00007fa30db20ecd in std::thread::_State_impl<std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#2}> > >::_M_run() (this=0x7fa3001a28f0) at /usr/include/c++/9/thread:195

10 0x00007fa30c0d2de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

11 0x00007fa30c1e7609 in start_thread (arg=) at pthread_create.c:477

12 0x00007fa30bf11293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 6 (Thread 0x7fa2fbfff700 (LWP 70914)):

0 lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::Running (this=0x555ae9575f10) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1201

1 0x00007fa30dafce04 in lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}::operator()() const (this=0x555ae9575f10) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/agera_storage.h:1152

2 0x00007fa30db35f12 in std::invoke_impl<void, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}>(std::__invoke_other, lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}&&) (f=...) at /usr/include/c++/9/bits/invoke.h:60

3 0x00007fa30db3074c in std::invoke<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}>(std::invoke_result&&, (lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}&&)...) (__fn=...) at /usr/include/c++/9/bits/invoke.h:95

4 0x00007fa30db2b7d1 in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7fa3001a26f8) at /usr/include/c++/9/thread:244

5 0x00007fa30db22276 in std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> >::operator()() (this=0x7fa3001a26f8) at /usr/include/c++/9/thread:251

6 0x00007fa30db20f03 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<lse::chunkserver::AgeraImpl<lse::chunkserver::StorageValue, lse::chunkserver::StorageKey>::StartWorker()::{lambda()#1}> > >::_M_run() (this=0x7fa3001a26f0) at /usr/include/c++/9/thread:195

7 0x00007fa30c0d2de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

8 0x00007fa30c1e7609 in start_thread (arg=) at pthread_create.c:477

9 0x00007fa30bf11293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 5 (Thread 0x7fa309896700 (LWP 70913)):

0 0x00007fa30becf3bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa3098946b0, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa30bed5047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa3098946b0, remaining=remaining@entry=0x0) at nanosleep.c:27

2 0x00007fa30bf079bf in usleep (useconds=useconds@entry=1000000) at ../sysdeps/posix/usleep.c:32

3 0x00007fa30c5212c8 in bthread_usleep (microseconds=microseconds@entry=1000000) at src/bthread/bthread.cpp:363

4 0x00007fa30c527b92 in brpc::Server::RunUntilAskedToQuit (this=0x7fa300220c40) at src/brpc/server.cpp:1693

5 0x00007fa30daf0859 in lse::chunkserver::Container::execute (this=0x555ae97de510) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/container.cpp:182

6 0x00007fa30db0d9a8 in std::__invoke_impl<void, void (lse::chunkserver::Container::)(), lse::chunkserver::Container> (f=@0x555ae97dc230: (void (lse::chunkserver::Container::)(lse::chunkserver::Container const)) 0x7fa30daef8f8 <lse::chunkserver::Container::execute()>, t=@0x555ae97dc228: 0x555ae97de510) at /usr/include/c++/9/bits/invoke.h:73

7 0x00007fa30db0771e in std::invoke<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> (fn=@0x555ae97dc230: (void (lse::chunkserver::Container::)(lse::chunkserver::Container const)) 0x7fa30daef8f8 <lse::chunkserver::Container::execute()>) at /usr/include/c++/9/bits/invoke.h:95

8 0x00007fa30dafff7e in std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> >::_M_invoke<0ul, 1ul> (this=0x555ae97dc228) at /usr/include/c++/9/thread:244

9 0x00007fa30dafc3ee in std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> >::operator() (this=0x555ae97dc228) at /usr/include/c++/9/thread:251

10 0x00007fa30daf73af in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (lse::chunkserver::Container::)(), lse::chunkserver::Container> > >::_M_run (this=0x555ae97dc220) at /usr/include/c++/9/thread:195

11 0x00007fa30c0d2de4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6

12 0x00007fa30c1e7609 in start_thread (arg=) at pthread_create.c:477

13 0x00007fa30bf11293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 4 (Thread 0x7fa30a69a700 (LWP 70912)):

0 0x00007fa30c1f249f in __libc_accept (fd=12, addr=..., len=0x0) at ../sysdeps/unix/sysv/linux/accept.c:26

1 0x00007fa30b6f58eb in socket_listener () from /lib/librte_telemetry.so.20.0

2 0x00007fa30c1e7609 in start_thread (arg=) at pthread_create.c:477

3 0x00007fa30bf11293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 3 (Thread 0x7fa30ae9b700 (LWP 70910)):

0 0x00007fa30bf115ce in epoll_wait (epfd=11, events=0x7fa30ae997c0, maxevents=1, timeout=-1) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30

1 0x00007fa30bce9f4a in eal_intr_thread_main () from /lib/librte_eal.so.20.0

2 0x00007fa30c1e7609 in start_thread (arg=) at pthread_create.c:477

3 0x00007fa30bf11293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 2 (Thread 0x7fa30b69c700 (LWP 70909)):

0 0x00007fa30becf3bf in __GI___clock_nanosleep (clock_id=clock_id@entry=0, flags=flags@entry=0, req=req@entry=0x7fa30b69a790, rem=rem@entry=0x0) at ../sysdeps/unix/sysv/linux/clock_nanosleep.c:78

1 0x00007fa30bed5047 in __GI___nanosleep (requested_time=requested_time@entry=0x7fa30b69a790, remaining=remaining@entry=0x0) at nanosleep.c:27

2 0x00007fa30bf079bf in usleep (useconds=) at ../sysdeps/posix/usleep.c:32

3 0x00007fa30c4fb52a in bvar::detail::SamplerCollector::run (this=0x555ae9577440) at src/bvar/detail/sampler.cpp:180

4 0x00007fa30c4fcc5d in bvar::detail::SamplerCollector::sampling_thread (arg=) at src/bvar/detail/sampler.cpp:110

5 0x00007fa30c1e7609 in start_thread (arg=) at pthread_create.c:477

6 0x00007fa30bf11293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Thread 1 (Thread 0x7fa30b6b1dc0 (LWP 70908)):

0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38

1 0x00007fa30c50117e in bthread::futex_wait_private (timeout=0x0, expected=0, addr1=0x7ffc20b32790) at ./src/bthread/sys_futex.h:40

2 bthread::wait_pthread (pw=..., ptimeout=ptimeout@entry=0x0) at src/bthread/butex.cpp:141

3 0x00007fa30c501ca5 in bthread::butex_wait_from_pthread (abstime=0x0, expected_value=1, b=0x555ae98c9200, g=) at src/bthread/butex.cpp:588

4 bthread::butex_wait (arg=0x555ae98c9200, expected_value=1, abstime=abstime@entry=0x0) at src/bthread/butex.cpp:621

5 0x00007fa30c500bfa in bthread::CountdownEvent::wait (this=0x555ae97d12f0) at src/bthread/countdown_event.cpp:64

6 0x00007fa30dad6a07 in lse::chunkserver::Container::wait (this=0x555ae97d9980) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/container.h:31

7 0x00007fa30dad39e4 in lse::chunkserver::ChunkServer::GetDeviceInfo (this=0x7ffc20b331c0, io_interface=std::vector of length 1, capacity 1 = {...}, status=0, device_name="trtype=PCIE traddr=0000:00:0b.0") at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/chunk_server.cpp:44

8 0x00007fa30dadd444 in std::__invoke_impl<void, void (lse::chunkserver::ChunkServer::&)(std::vector<lse::chunkserver::IoInterface, std::allocator<lse::chunkserver::IoInterface> >, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >), lse::chunkserver::ChunkServer&, std::vector<lse::chunkserver::IoInterface, std::allocator<lse::chunkserver::IoInterface> >, int, std::cxx11::basic_string<char, std::char_traits, std::allocator >&> (f=@0x555ae97dea00: (void (lse::chunkserver::ChunkServer::)(class lse::chunkserver::ChunkServer const, class std::vector<lse::chunkserver::IoInterface, std::allocator<lse::chunkserver::IoInterface> >, int, class std::cxx11::basic_string<char, std::char_traits, std::allocator >)) 0x7fa30dad317a <lse::chunkserver::ChunkServer::GetDeviceInfo(std::vector<lse::chunkserver::IoInterface, std::allocator<lse::chunkserver::IoInterface> >, int, std::cxx11::basic_string<char, std::char_traits, std::allocator >)>, __t=@0x555ae97dea30: 0x7ffc20b331c0) at /usr/include/c++/9/bits/invoke.h:73

9 0x00007fa30dadcbc3 in std::invoke<void (lse::chunkserver::ChunkServer::&)(std::vector<lse::chunkserver::IoInterface, std::allocator<lse::chunkserver::IoInterface*> >, int, std::cxx11::basic_string<char, std::char_traits, std::allocator >), lse::chunkserver::ChunkServer&, std::vector<lse::chunkserver::IoInterface, std::allocator<lse::chunkserver::IoInterface> >, int, std::cxx11::basic_string<char, std::char_traits, std::allocator >&> (fn=@0x555ae97dea00: (void (lse::chunkserver::ChunkServer::)(class lse::chunkserver::ChunkServer const, class std::vector<lse::chunkserver::IoInterface, std::allocator<lse::chunkserver::IoInterface> >, int, class std::__cxx11::basic_string<char, std::char_traits, std::allocator >)) 0x7fa30dad317a <lse::chunkserver::ChunkServer::GetDeviceInfo(std::vector<lse::chunkserver::IoInterface, std::allocator<lse::chunkserver::IoInterface*> >, int, std::__cxx11::basic_string<char, std::char_traits, std::allocator >)>) at /usr/include/c++/9/bits/invoke.h:95

10 0x00007fa30dadc0ba in std::_Bind<void (lse::chunkserver::ChunkServer::(lse::chunkserver::ChunkServer, std::_Placeholder<1>, std::_Placeholder<2>, std::cxx11::basic_string<char, std::char_traits, std::allocator >))(std::vector<lse::chunkserver::IoInterface, std::allocator<lse::chunkserver::IoInterface> >, int, std::cxx11::basic_string<char, std::char_traits, std::allocator >)>::call<void, std::vector<lse::chunkserver::IoInterface, std::allocator<lse::chunkserver::IoInterface> >&&, int&&, 0ul, 1ul, 2ul, 3ul>(std::tuple<std::vector<lse::chunkserver::IoInterface, std::allocator<lse::chunkserver::IoInterface> >&&, int&&>&&, std::_Index_tuple<0ul, 1ul, 2ul, 3ul>) (this=0x555ae97dea00, args=...) at /usr/include/c++/9/functional:400

11 0x00007fa30dadad81 in std::_Bind<void (lse::chunkserver::ChunkServer::(lse::chunkserver::ChunkServer, std::_Placeholder<1>, std::_Placeholder<2>, std::cxx11::basic_string<char, std::char_traits, std::allocator >))(std::vector<lse::chunkserver::IoInterface, std::allocator<lse::chunkserver::IoInterface> >, int, std::cxx11::basic_string<char, std::char_traits, std::allocator >)>::operator()<std::vector<lse::chunkserver::IoInterface, std::allocator<lse::chunkserver::IoInterface> >, int, void>(std::vector<lse::chunkserver::IoInterface, std::allocator<lse::chunkserver::IoInterface> >&&, int&&) (this=0x555ae97dea00) at /usr/include/c++/9/functional:484

12 0x00007fa30dad97a6 in std::_Function_handler<void (std::vector<lse::chunkserver::IoInterface, std::allocator<lse::chunkserver::IoInterface> >, int), std::_Bind<void (lse::chunkserver::ChunkServer::(lse::chunkserver::ChunkServer, std::_Placeholder<1>, std::_Placeholder<2>, std::cxx11::basic_string<char, std::char_traits, std::allocator >))(std::vector<lse::chunkserver::IoInterface, std::allocator<lse::chunkserver::IoInterface> >, int, std::cxx11::basic_string<char, std::char_traits, std::allocator >)> >::_M_invoke(std::_Any_data const&, std::vector<lse::chunkserver::IoInterface, std::allocator<lse::chunkserver::IoInterface> >&&, int&&) (functor=..., args#0=..., __args#1=@0x7ffc20b32bcc: 0) at /usr/include/c++/9/bits/std_function.h:300

13 0x00007fa30d7956d1 in std::function<void (std::vector<lse::chunkserver::IoInterface, std::allocator<lse::chunkserver::IoInterface> >, int)>::operator()(std::vector<lse::chunkserver::IoInterface, std::allocator<lse::chunkserver::IoInterface> >, int) const (this=0x7ffc20b33090, args#0=std::vector of length 0, capacity 0, args#1=0) at /usr/include/c++/9/bits/std_function.h:688

14 0x00007fa30d78f56e in lse::chunkserver::SpdkControllerCollection::RegisterController (this=0x555ae9577290, attachment=0x7ffc20b33090, trid=0x2000068aa128, ctrlr=0x2000068aa100) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/spdk.cpp:251

15 0x00007fa30d78dec4 in lse::chunkserver::SpdkControllerCollection::AttachCallback (attachment=0x7ffc20b33090, trid=0x2000068aa128, ctrlr=0x2000068aa100, opts=0x2000068ab728) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/spdk.cpp:86

16 0x0000555ae87adb80 in nvme_ctrlr_poll_internal (probe_ctx=0x555ae994b730, ctrlr=0x2000068aa100) at nvme.c:745

17 spdk_nvme_probe_poll_async (probe_ctx=probe_ctx@entry=0x555ae994b730) at nvme.c:1561

18 0x0000555ae87add58 in nvme_init_controllers (probe_ctx=) at nvme.c:755

19 spdk_nvme_probe (trid=, cb_ctx=0x7ffc20b33090, probe_cb=0x7fa30d78d8c4 <lse::chunkserver::SpdkControllerCollection::ProbeCallback(void, spdk_nvme_transport_id const, spdk_nvme_ctrlr_opts*)>, attach_cb=, remove_cb=) at nvme.c:903

20 0x00007fa30d78e848 in lse::chunkserver::SpdkControllerCollection::RegisterController(char const, std::function<void (std::vector<lse::chunkserver::IoInterface, std::allocator<lse::chunkserver::IoInterface*> >, int)>) (this=0x555ae9577290, transport_id=0x555ae97d7640 "trtype=PCIE traddr=0000:00:0b.0", callback=...) at /root/jenkins_S/workspace/CI-CKS-Daily/src/storage/spdk.cpp:163

21 0x00007fa30dad5320 in lse::chunkserver::ChunkServer::InitSpdkDevice (this=0x7ffc20b331c0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/chunk_server.cpp:140

22 0x00007fa30dad4ced in lse::chunkserver::ChunkServer::Initialize (this=0x7ffc20b331c0) at /root/jenkins_S/workspace/CI-CKS-Daily/src/container/chunk_server.cpp:119

23 0x0000555ae8794426 in main (argc=2, argv=0x7ffc20b33b28) at /root/jenkins_S/workspace/CI-CKS-Daily/src/main_control.cpp:97

Undefined command: "qut". Try "help". Detaching from program: /home/kuhan/deployCS/cks_805/chunkserver, process 70908 [Inferior 1 (process 70908) detached]

PFZheng commented 3 years ago

有两个疑问：

  1. Thread 25 看起来在等待一个 pthread 线程，需要看下被等待的线程在做什么为什么没结束，这个是关键；
  2. 很多线程里有 sleep，这个是在做什么。 主要是问题1。