search

apache / brpc

brpc is an Industrial-grade RPC framework using C++ Language, which is often used in high performance system such as Search, Storage, Machine learning, Advertisement, Recommendation etc. "brpc" means "better RPC".
https://brpc.apache.org
Apache License 2.0
16.56k stars 3.98k forks source link

brpc死锁 #2259

Closed xiedeacc closed 1 year ago

xiedeacc commented 1 year ago

Describe the bug (描述bug) 进程运行一段时间后，突然死锁。service的方法中会使用DynamicPartitionChannel访问另外一个服务，开启和不开启usercode_in_pthread都会出现死锁

Versions (各种版本) OS:18.04 Compiler: gcc11.3.0 brpc:1.3.0 compile from source protobuf:3.19.2

wwbmmm commented 1 year ago

能贴一下死锁时各个线程的堆栈吗？

xiedeacc commented 1 year ago 
process: /proc/1924/mem
thread: 0x7fa54ecb8080, lwp: 1924, type: 1
#0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
#1  0x000000000a900148 in _ZN4brpc8Acceptor4JoinEv!()+71 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a8d312b in _ZN4brpc6Server4JoinEv!()+42 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000082e8d3 in _ZN4wifi2ad7ranking7service18RankingServiceImpl3RunEv!()+1666 in /data/app/nads-rankingdump/bin/ranking
#4  0x00000000005b07a4 in main!()+259 in /data/app/nads-rankingdump/bin/ranking
#5  0x00007fa54d51ac87 in __libc_start_main()+230 in /lib/x86_64-linux-gnu/libc.so.6 at libc-start.c:310
#6  0x000000000080713a in _start!()+41 in /data/app/nads-rankingdump/bin/ranking

thread: 0x7fa50d1fa700, lwp: 2008, type: 1
#0  0x00007fa54e6cf9e2 in __libc_recvfrom()+98 in /lib/x86_64-linux-gnu/libpthread.so.0 at recvfrom.c:27
#1  0x000000000a83182a in _ZN5nacos24UdpNamingServiceListener18listenerThreadFuncEPv!()+601 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a826bb0 in _ZN5nacos6Thread10threadFuncEPv!()+31 in /data/app/nads-rankingdump/bin/ranking
#3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa50d9fb700, lwp: 2007, type: 1
#0  0x00007fa54d5dd680 in __nanosleep()+64 in /lib/x86_64-linux-gnu/libc.so.6 at nanosleep.c:28
#1  0x00007fa54d5dd55a in __sleep()+57 in /lib/x86_64-linux-gnu/libc.so.6 at sleep.c:55
#2  0x000000000a832cd0 in _ZN5nacos18SubscriptionPoller17pollingThreadFuncEPv!()+799 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000a826bb0 in _ZN5nacos6Thread10threadFuncEPv!()+31 in /data/app/nads-rankingdump/bin/ranking
#4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa50e1fc700, lwp: 2006, type: 1
#0  0x00007fa54e6cc065 in __pthread_cond_timedwait()+821 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:205
#1  0x000000000a826f41 in _ZN5nacos13DelayedWorker3runEv!()+640 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a86eb75 in _ZN5nacos10ThreadPool11runInThreadEPv!()+180 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000a826bb0 in _ZN5nacos6Thread10threadFuncEPv!()+31 in /data/app/nads-rankingdump/bin/ranking
#4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa50e9fd700, lwp: 2005, type: 1
#0  0x00007fa54e6cc065 in __pthread_cond_timedwait()+821 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:205
#1  0x000000000a826f41 in _ZN5nacos13DelayedWorker3runEv!()+640 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a86eb75 in _ZN5nacos10ThreadPool11runInThreadEPv!()+180 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000a826bb0 in _ZN5nacos6Thread10threadFuncEPv!()+31 in /data/app/nads-rankingdump/bin/ranking
#4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa50f1fe700, lwp: 2004, type: 1
#0  0x00007fa54e6cc065 in __pthread_cond_timedwait()+821 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:205
#1  0x000000000a826f41 in _ZN5nacos13DelayedWorker3runEv!()+640 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a86eb75 in _ZN5nacos10ThreadPool11runInThreadEPv!()+180 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000a826bb0 in _ZN5nacos6Thread10threadFuncEPv!()+31 in /data/app/nads-rankingdump/bin/ranking
#4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa50f9ff700, lwp: 2003, type: 1
#0  0x00007fa54e6cc065 in __pthread_cond_timedwait()+821 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:205
#1  0x000000000a826f41 in _ZN5nacos13DelayedWorker3runEv!()+640 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a86eb75 in _ZN5nacos10ThreadPool11runInThreadEPv!()+180 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000a826bb0 in _ZN5nacos6Thread10threadFuncEPv!()+31 in /data/app/nads-rankingdump/bin/ranking
#4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5155fe700, lwp: 2002, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5163fe700, lwp: 2001, type: 1
#0  0x00007fa54d61a947 in epoll_wait()+87 in /lib/x86_64-linux-gnu/libc.so.6 at epoll_wait.c:30
#1  0x000000000a8b7841 in _ZN4brpc15EventDispatcher3RunEv!()+64 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a8b7969 in _ZN4brpc15EventDispatcher7RunThisEPv!()+8 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#5  0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#6  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#7  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa516bff700, lwp: 2000, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa27b9a in _ZN7bthread9TaskGroup9wait_taskEPm!()+121 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa2ac4f in _ZN7bthread9TaskGroup13run_main_taskEv!()+126 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5179fe700, lwp: 1999, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5181ff700, lwp: 1998, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa518ffb700, lwp: 1997, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5197fc700, lwp: 1996, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa51a5fc700, lwp: 1995, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa51adfd700, lwp: 1994, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa51b5fe700, lwp: 1993, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa51c3fe700, lwp: 1992, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa51cbff700, lwp: 1991, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa51d9ff700, lwp: 1990, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa51e5fc700, lwp: 1989, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa51effd700, lwp: 1988, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa51f7fe700, lwp: 1987, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa27b9a in _ZN7bthread9TaskGroup9wait_taskEPm!()+121 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa2ac4f in _ZN7bthread9TaskGroup13run_main_taskEv!()+126 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5205fe700, lwp: 1986, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa520dff700, lwp: 1985, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa521bfe700, lwp: 1984, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5223ff700, lwp: 1983, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa522ffd700, lwp: 1982, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5237fe700, lwp: 1981, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa523fff700, lwp: 1980, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa524dfe700, lwp: 1979, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5255ff700, lwp: 1978, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5261fd700, lwp: 1977, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa526bfe700, lwp: 1975, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa27b9a in _ZN7bthread9TaskGroup9wait_taskEPm!()+121 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa2ac4f in _ZN7bthread9TaskGroup13run_main_taskEv!()+126 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5273ff700, lwp: 1974, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa52857f700, lwp: 1973, type: 1
#0  0x00007fa54e6cfd50 in __nanosleep()+64 in /lib/x86_64-linux-gnu/libpthread.so.0 at nanosleep.c:28
#1  0x000000000a87ba56 in AbslInternalSleepFor_lts_20220623!()+133 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a877a9d in _ZN4wifi2ad7ranking4util11RankingUtil10SleepUntilERKN4absl12lts_202206234TimeEPVb!()+76 in /data/app/nads-rankingdump/bin/ranking
#3  0x00000000008662b9 in _ZN4wifi2ad7ranking7feature12RankingCache19BackGroundUpdateRunEv!()+312 in /data/app/nads-rankingdump/bin/ranking
#4  0x00007fa54dbdcf64 in std::execute_native_thread_routine()+19 in /data/app/nads-rankingdump/lib/libstdc++.so.6 at thread.cc:82
#5  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#6  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa52d5ec700, lwp: 1972, type: 1
#0  0x00007fa54e6cfd50 in __nanosleep()+64 in /lib/x86_64-linux-gnu/libpthread.so.0 at nanosleep.c:28
#1  0x00000000008582a6 in _ZN12redis_client4util7SlotMap6ReloadEv!()+181 in /data/app/nads-rankingdump/bin/ranking
#2  0x00007fa54dbdcf64 in std::execute_native_thread_routine()+19 in /data/app/nads-rankingdump/lib/libstdc++.so.6 at thread.cc:82
#3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa52dded700, lwp: 1971, type: 1
#0  0x00007fa54e6cfd50 in __nanosleep()+64 in /lib/x86_64-linux-gnu/libpthread.so.0 at nanosleep.c:28
#1  0x000000000a877926 in _ZN4wifi2ad7ranking4util11RankingUtil5SleepEl!()+101 in /data/app/nads-rankingdump/bin/ranking
#2  0x0000000000834110 in _ZN4wifi2ad7ranking7service18RankingServiceImpl9PrintTaskEv!()+175 in /data/app/nads-rankingdump/bin/ranking
#3  0x00007fa54dbdcf64 in std::execute_native_thread_routine()+19 in /data/app/nads-rankingdump/lib/libstdc++.so.6 at thread.cc:82
#4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa52e5ee700, lwp: 1970, type: 1
#0  0x00007fa54e6cfd50 in __nanosleep()+64 in /lib/x86_64-linux-gnu/libpthread.so.0 at nanosleep.c:28
#1  0x000000000a877926 in _ZN4wifi2ad7ranking4util11RankingUtil5SleepEl!()+101 in /data/app/nads-rankingdump/bin/ranking
#2  0x0000000000839b6c in _ZN4wifi2ad7ranking7service18RankingServiceImpl12ShutdownTaskEv!()+587 in /data/app/nads-rankingdump/bin/ranking
#3  0x00007fa54dbdcf64 in std::execute_native_thread_routine()+19 in /data/app/nads-rankingdump/lib/libstdc++.so.6 at thread.cc:82
#4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa52edef700, lwp: 1969, type: 1
#0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
#1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
#3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa52f5f0700, lwp: 1968, type: 1
#0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
#1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
#3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa52fdf1700, lwp: 1967, type: 1
#0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
#1  inlined            in __gthread_cond_wait at gthr-default.h:865
#1  inlined            in std::__condvar::wait at std_mutex.h:155
#1  0x00007fa54dbd7390 in std::condition_variable::wait()+15 in /data/app/nads-rankingdump/lib/libstdc++.so.6 at condition_variable.cc:41
#2  0x000000000084eb2c in _ZN4wifi2ad7ranking7service18RankingServiceImpl8ScanListEv!()+395 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000083a95b in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+762 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
#5  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#6  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5305f2700, lwp: 1965, type: 1
#0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
#1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
#3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa530df3700, lwp: 1964, type: 1
#0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
#1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
#3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5315f4700, lwp: 1962, type: 1
#0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
#1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
#3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa531df5700, lwp: 1961, type: 1
#0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
#1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
#3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5325f6700, lwp: 1960, type: 1
#0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
#1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
#3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa532df7700, lwp: 1959, type: 1
#0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
#1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
#3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5335f8700, lwp: 1958, type: 1
#0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
#1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
#3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa533df9700, lwp: 1957, type: 1
#0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
#1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
#3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5345fa700, lwp: 1956, type: 1
#0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
#1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
#3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa534dfb700, lwp: 1955, type: 1
#0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
#1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
#3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5355fc700, lwp: 1954, type: 1
#0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
#1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
#3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa535dfd700, lwp: 1953, type: 1
#0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
#1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
#3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5365fe700, lwp: 1952, type: 1
#0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
#1  0x000000000083a7b3 in _ZN5boost9executors17basic_thread_pool13worker_threadEv!()+338 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000a87afa0 in thread_proxy!()+111 in /data/app/nads-rankingdump/bin/ranking
#3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa536dff700, lwp: 1951, type: 1
#0  0x00007fa54d5dd680 in __nanosleep()+64 in /lib/x86_64-linux-gnu/libc.so.6 at nanosleep.c:28
#1  0x00007fa54d5dd55a in __sleep()+57 in /lib/x86_64-linux-gnu/libc.so.6 at sleep.c:55
#2  0x000000000085578a in _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN4wifi2ad7ranking7feature13FeatureClient17InitFeatureClientEvEUlvE_EEEEE6_M_runEv!()+57 in /data/app/nads-rankingdump/bin/ranking
#3  0x00007fa54dbdcf64 in std::execute_native_thread_routine()+19 in /data/app/nads-rankingdump/lib/libstdc++.so.6 at thread.cc:82
#4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa539ffc700, lwp: 1950, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa27b9a in _ZN7bthread9TaskGroup9wait_taskEPm!()+121 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa2ac4f in _ZN7bthread9TaskGroup13run_main_taskEv!()+126 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa53a7fd700, lwp: 1949, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa53b3fd700, lwp: 1948, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa53bdfe700, lwp: 1947, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa53c5ff700, lwp: 1946, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa27b9a in _ZN7bthread9TaskGroup9wait_taskEPm!()+121 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa2ac4f in _ZN7bthread9TaskGroup13run_main_taskEv!()+126 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa53d1fd700, lwp: 1945, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa53e3fe700, lwp: 1944, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa53ebff700, lwp: 1943, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa53f7f9700, lwp: 1942, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8fe8d8 in _ZN4brpc6policy17ProcessRpcRequestEPNS_16InputMessageBaseE!()+4519 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8e7307 in _ZN4brpc14InputMessenger13OnNewMessagesEPNS_6SocketE!()+1302 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8c167e in _ZN4brpc6Socket12ProcessEventEPv!()+13 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000aa2a68e in _ZN7bthread9TaskGroup11task_runnerEl!()+61 in /data/app/nads-rankingdump/bin/ranking
#11 0x000000000aa2acd8 in _ZN7bthread9TaskGroup13run_main_taskEv!()+263 in /data/app/nads-rankingdump/bin/ranking
#12 0x000000000aa3046c in _ZN7bthread11TaskControl13worker_threadEPv!()+331 in /data/app/nads-rankingdump/bin/ranking
#13 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#14 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa53fffa700, lwp: 1941, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa25895 in _ZN7bthread11TimerThread3runEv!()+3076 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa26299 in _ZN7bthread11TimerThread8run_thisEPv!()+8 in /data/app/nads-rankingdump/bin/ranking
#3  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#4  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5407fb700, lwp: 1940, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8faa6f in _ZN4brpc6policyL24CallMethodInBackupThreadEPv!()+46 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8afa95 in _ZN4brpc18UserCodeBackupPool19UserCodeRunningLoopEv!()+276 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8b01c9 in _ZN4brpcL14UserCodeRunnerEPv!()+8 in /data/app/nads-rankingdump/bin/ranking
#10 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#11 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa540ffc700, lwp: 1939, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8faa6f in _ZN4brpc6policyL24CallMethodInBackupThreadEPv!()+46 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8afa95 in _ZN4brpc18UserCodeBackupPool19UserCodeRunningLoopEv!()+276 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8b01c9 in _ZN4brpcL14UserCodeRunnerEPv!()+8 in /data/app/nads-rankingdump/bin/ranking
#10 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#11 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5417fd700, lwp: 1938, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8faa6f in _ZN4brpc6policyL24CallMethodInBackupThreadEPv!()+46 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8afa95 in _ZN4brpc18UserCodeBackupPool19UserCodeRunningLoopEv!()+276 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8b01c9 in _ZN4brpcL14UserCodeRunnerEPv!()+8 in /data/app/nads-rankingdump/bin/ranking
#10 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#11 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa541ffe700, lwp: 1937, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8faa6f in _ZN4brpc6policyL24CallMethodInBackupThreadEPv!()+46 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8afa95 in _ZN4brpc18UserCodeBackupPool19UserCodeRunningLoopEv!()+276 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8b01c9 in _ZN4brpcL14UserCodeRunnerEPv!()+8 in /data/app/nads-rankingdump/bin/ranking
#10 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#11 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5427ff700, lwp: 1936, type: 1
#0  0x00007fa54d614539 in syscall!()+25 in /lib/x86_64-linux-gnu/libc.so.6 at syscall.S:38
#1  0x000000000aa40306 in _ZN7bthread12wait_pthreadERNS_18ButexPthreadWaiterEP8timespec!()+69 in /data/app/nads-rankingdump/bin/ranking
#2  0x000000000aa413b3 in _ZN7bthread10butex_waitEPviPK8timespec!()+770 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa37829 in bthread_id_join!()+184 in /data/app/nads-rankingdump/bin/ranking
#4  0x000000000a8e63f8 in _ZN4brpc16SelectiveChannel10CallMethodEPKN6google8protobuf16MethodDescriptorEPNS2_13RpcControllerEPKNS2_7MessageEPS8_PNS2_7ClosureE!()+359 in /data/app/nads-rankingdump/bin/ranking
#5  0x00000000008545f9 in _ZN4wifi2ad7ranking7feature13FeatureClient22GetUserFeaturesWithTTLERKSsRN10tensorflow8FeaturesEi!()+408 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000084f7ba in _ZN4wifi2ad7ranking7service18RankingServiceImpl4RankEPN6google8protobuf13RpcControllerEPKNS0_8protocol6engine7ranking11RankRequestEPNSA_12RankResponseEPNS5_7ClosureE!()+409 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a8faa6f in _ZN4brpc6policyL24CallMethodInBackupThreadEPv!()+46 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a8afa95 in _ZN4brpc18UserCodeBackupPool19UserCodeRunningLoopEv!()+276 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a8b01c9 in _ZN4brpcL14UserCodeRunnerEPv!()+8 in /data/app/nads-rankingdump/bin/ranking
#10 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#11 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5439fe700, lwp: 1935, type: 1
#0  0x00007fa54e6cbad3 in __pthread_cond_wait()+579 in /lib/x86_64-linux-gnu/libpthread.so.0 at futex-internal.h:88
#1  0x000000000088a10b in do_completion!()+170 in /data/app/nads-rankingdump/bin/ranking
#2  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#3  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5441ff700, lwp: 1934, type: 1
#0  0x00007fa54d60dbb9 in __poll()+73 in /lib/x86_64-linux-gnu/libc.so.6 at poll.c:29
#1  0x000000000088a354 in do_io!()+403 in /data/app/nads-rankingdump/bin/ranking
#2  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#3  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa545c7f700, lwp: 1931, type: 1
#0  0x00007fa54d60dbb9 in __poll()+73 in /lib/x86_64-linux-gnu/libc.so.6 at poll.c:29
#1  0x00007fa54e916ef9 in <unknown>() in /usr/lib/x86_64-linux-gnu/libcurl.so.4
#2  0x00007fa54e911212 in curl_multi_wait!()+817 in /usr/lib/x86_64-linux-gnu/libcurl.so.4
#3  0x00007fa54e909970 in curl_easy_perform!()+207 in /usr/lib/x86_64-linux-gnu/libcurl.so.4
#4  0x000000000a8156c2 in _ZN5nacos7HTTPCli16httpPostInternalERKSsRSt4listISsSaISsEES2_S2_l!()+625 in /data/app/nads-rankingdump/bin/ranking
#5  0x000000000a815c90 in _ZN5nacos7HTTPCli8httpPostERKSsRSt4listISsSaISsEES6_S2_l!()+111 in /data/app/nads-rankingdump/bin/ranking
#6  0x000000000a81a869 in _ZN5nacos16NoOpHttpDelegate8httpPostERKSsRSt4listISsSaISsEES6_S2_l!()+216 in /data/app/nads-rankingdump/bin/ranking
#7  0x000000000a85deeb in _ZN5nacos11ConfigProxy6reqAPIEiRKSsRSt4listISsSaISsEES6_S2_l!()+2010 in /data/app/nads-rankingdump/bin/ranking
#8  0x000000000a82288e in _ZN5nacos12ClientWorker17checkListenedKeysEv!()+1789 in /data/app/nads-rankingdump/bin/ranking
#9  0x000000000a825163 in _ZN5nacos12ClientWorker12performWatchEv!()+82 in /data/app/nads-rankingdump/bin/ranking
#10 0x000000000a826b4c in _ZN5nacos12ClientWorker14listenerThreadEPv!()+59 in /data/app/nads-rankingdump/bin/ranking
#11 0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#12 0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0x7fa5475ff700, lwp: 1929, type: 1
#0  0x00007fa54d5dd680 in __nanosleep()+64 in /lib/x86_64-linux-gnu/libc.so.6 at nanosleep.c:28
#1  0x00007fa54d610557 in usleep()+86 in /lib/x86_64-linux-gnu/libc.so.6 at usleep.c:32
#2  0x000000000aa5be3a in _ZN4bvar6detail16SamplerCollector3runEv!()+601 in /data/app/nads-rankingdump/bin/ranking
#3  0x000000000aa5c319 in _ZN4bvar6detail16SamplerCollector15sampling_threadEPv!()+8 in /data/app/nads-rankingdump/bin/ranking
#4  0x00007fa54e6c56db in start_thread()+218 in /lib/x86_64-linux-gnu/libpthread.so.0 at pthread_create.c:463
#5  0x00007fa54d61a61f in __GI___clone!()+62 in /lib/x86_64-linux-gnu/libc.so.6 at clone.S:95

thread: 0, lwp: 0, type: 0
#0  0x0000000000000000 no information for frame
xiedeacc commented 1 year ago

https://github.com/apache/brpc/issues/1648 和这个问题应该是同一个问题

wwbmmm commented 1 year ago

看起来和 #1648 #644 是类似的问题，由于并发bthread数量过大导致mmap数量超限，无法创建更多bthread，降级到pthread上执行，导致死锁。可以尝试调大内核参数vm.max_map_count来支持更多的bthread

xiedeacc commented 1 year ago

vm.max_map_count = 262144 这个值也不小了吧

xiedeacc commented 1 year ago

而且brpc_max_concurrency设置的为5000

wwbmmm commented 1 year ago

DynamicPartitionChannel 这个一次会并发访问多少个分片呢

xiedeacc commented 1 year ago

8个分片

wwbmmm commented 1 year ago

可以在启动服务之后观察内置服务的bthread_count指标以及进程内存指标的变化，看是否会不断增长

xiedeacc commented 1 year ago

会增长，也会下降，最多到50了，然后又下来了

xiedeacc commented 1 year ago 
bthread_count : 37
bthread_group_status : "1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 "
bthread_key_count : 0
bthread_keytable_count : 0
bthread_keytable_memory : 0
bthread_signal_second : 987
bthread_stack_count : 191
bthread_switch_second : 2095
bthread_timer_scheduled_second : 295
bthread_timer_triggered_second : 14
bthread_timer_usage : 0.000139691
bthread_worker_count : 32
bthread_worker_usage : 32.6478
bvar_dump_interval : 10
bvar_sampler_collector_usage : 0.00116603
gcc_version : "11.3.0"
iobuf_block_count : 215
iobuf_block_count_hit_tls_threshold : 19283
iobuf_block_memory : 1761280
iobuf_newbigview_second : 1
kernel_version : "Linux ad-nads-rankingdump7.wxxdc 5.4.193-0504193-generic #202205121146 SMP Thu May 12 12:05:07 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
"
pgrp : 28346
pid : 28346
ppid : 1966
process_cmdline : "./bin/ranking
"
process_context_switches_involuntary_second : 5
process_context_switches_voluntary_second : 2025
process_cpu_usage : 0.658
process_cpu_usage_system : 0.325
process_cpu_usage_user : 0.333
process_disk_read_bytes_second : 0
process_disk_write_bytes_second : 250249
process_faults_major : 0
process_faults_minor_second : 270
process_fd_count : 718
process_inblocks_second : 0
process_io_read_bytes_second : 712818
process_io_read_second : 854
process_io_write_bytes_second : 299397
process_io_write_second : 967
process_memory_data_and_stack : 1138036736
process_memory_resident : 158076928
process_memory_shared : 33538048
process_memory_text : 215818240
process_memory_virtual : 1470058496
process_nice : 0
process_outblocks_second : 489
process_priority : 20
process_thread_count : 65
process_uptime : 313.753962
process_username : "unknown (No such device or address)"
process_work_dir : "/data/app/nads-rankingdump"
ranking_ad_num_avg : 3
ranking_cache_hit_rate : 9943
ranking_cache_miss_num : 5
ranking_channel_init_times : 0
ranking_default_response_num : 0
ranking_fast_fail_num : 0
ranking_get_ctx_fea_count : 87498
ranking_get_ctx_fea_latency : 0.007
ranking_get_ctx_fea_latency_80 : 0
ranking_get_ctx_fea_latency_90 : 0
ranking_get_ctx_fea_latency_99 : 0
ranking_get_ctx_fea_latency_999 : 0
ranking_get_ctx_fea_latency_9999 : 0
ranking_get_ctx_fea_max_latency : 1
ranking_get_ctx_fea_qps : 284
ranking_get_fea_count : 87487
ranking_get_fea_latency : 59
ranking_get_fea_latency_80 : 0
ranking_get_fea_latency_90 : 0
ranking_get_fea_latency_99 : 0
ranking_get_fea_latency_999 : 0
ranking_get_fea_latency_9999 : 0
ranking_get_fea_max_latency : 170
ranking_get_fea_qps : 286
ranking_get_item_fea_count : 87492
ranking_get_item_fea_latency : 0.038
ranking_get_item_fea_latency_80 : 0
ranking_get_item_fea_latency_90 : 0
ranking_get_item_fea_latency_99 : 0
ranking_get_item_fea_latency_999 : 0
ranking_get_item_fea_latency_9999 : 0
ranking_get_item_fea_max_latency : 1
ranking_get_item_fea_qps : 287
ranking_get_user_fea_count : 87500
ranking_get_user_fea_latency : 42
ranking_get_user_fea_latency_80 : 0
ranking_get_user_fea_latency_90 : 0
ranking_get_user_fea_latency_99 : 0
ranking_get_user_fea_latency_999 : 0
ranking_get_user_fea_latency_9999 : 0
ranking_get_user_fea_max_latency : 117
ranking_get_user_fea_qps : 283
ranking_idea_not_equal : 0
ranking_idea_num : 0
ranking_index_empty_num : 0
ranking_index_less_20_num : 0
ranking_index_less_20_rate : 0.000
ranking_item_fea_null : 0
ranking_item_fea_num_error : 0
ranking_miss_model_num : 0
ranking_model_load_failed_num : 0
ranking_parse_error_num : 0
ranking_predict_count : 0
ranking_predict_error : 0
ranking_predict_fail_num : 0
ranking_predict_latency : 0.000
ranking_predict_latency_80 : 0
ranking_predict_latency_90 : 0
ranking_predict_latency_99 : 0
ranking_predict_latency_999 : 0
ranking_predict_latency_9999 : 0
ranking_predict_max_latency : 0
ranking_predict_qps : 0
ranking_predict_sucess_num : 0
ranking_prepare_count : 0
ranking_prepare_error : 0
ranking_prepare_latency : 0.000
ranking_prepare_latency_80 : 0
ranking_prepare_latency_90 : 0
ranking_prepare_latency_99 : 0
ranking_prepare_latency_999 : 0
ranking_prepare_latency_9999 : 0
ranking_prepare_max_latency : 0
ranking_prepare_qps : 0
ranking_rank_count : 87483
ranking_rank_latency : 99
ranking_rank_latency_80 : 0
ranking_rank_latency_90 : 0
ranking_rank_latency_99 : 0
ranking_rank_latency_999 : 0
ranking_rank_latency_9999 : 0
ranking_rank_max_latency : 247
ranking_rank_qps : 285
ranking_reply_empty_num : 0
ranking_reply_error_num : 0
ranking_req_num : 283
ranking_result_error : 0
ranking_score_1_exception : 0
ranking_score_high_exception : 0
ranking_score_neg_exception : 0
ranking_update_fail_num : 0
ranking_user_fea_empty_req : 283
ranking_user_fea_req : 283
ranking_user_fea_req_empty_rate : 10035
rpc_channel_connection_count : 8
rpc_compressions : "snappy gzip zlib"
rpc_controller_count : 40
rpc_event_thread_second : 387
rpc_health_check_count : 27
rpc_keepwrite_second : 2
rpc_load_balancer : "c_md5 wr c_murmurhash wrr rr c_ketama random la"
rpc_naming_service : "https consul redis file list http discovery dlist nacos remotefile"
rpc_profilers : "contention"
rpc_protocols : "baidu_std streaming_rpc hulu_pbrpc sofa_pbrpc rtmp http public_pbrpc nova_pbrpc redis nshead mongo ubrpc_compack memcache nshead_mcpack ubrpc_mcpack2 esp h2"
rpc_revision : "undefined"
rpc_server_10003_builtin_service_count : 19
rpc_server_10003_connection_count : 89
rpc_server_10003_eps : 0
rpc_server_10003_error : 0
rpc_server_10003_service_count : 1
rpc_server_10003_start_time : "2023/05/22-16:32:19"
rpc_server_10003_uptime : 313.417985
rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_concurrency : 31
rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_count : 87480
rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_eps : 0
rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_error : 0
rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_latency : 57759
rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_latency_80 : 109728
rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_latency_90 : 138437
rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_latency_99 : 198365
rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_latency_999 : 240210
rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_latency_9999 : 250536
rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_max_latency : 283137
rpc_server_10003_wifi_ad_protocol_engine_ranking_rank_service_rank_qps : 282
rpc_server_count : 1
rpc_socket_count : 99
rpc_waitepollout_count : 0
rpc_waitepollout_second : 0
rpcz_sampling_ratio : 1.000
system_core_count : 32
system_loadavg_15m : 0.420
system_loadavg_1m : 0.910
system_loadavg_5m : 0.660
wwbmmm commented 1 year ago

看bthread_stack_count和内存占用都不大。 我看你的堆栈里有CallMethodInBackupThread，是不是开启了usercode_in_pthread=true呢？

xiedeacc commented 1 year ago

image

xiedeacc commented 1 year ago

没有开启

xiedeacc commented 1 year ago

我的done是自己管理的，有快速拒绝机制，可能会瞬间把自己管理的队列一下子全部释放掉（会调done->Run()）

xiedeacc commented 1 year ago

不过就算done不自己管理，也会出现同样的状况，我已经测试过了

xiedeacc commented 1 year ago

把DynamicPartitionChannel换成普通Channle后这个问题修复了，目前观察是这样的，再观察几天看看

xiedeacc commented 1 year ago

brpc问题实在太多了，有负盛名

wwbmmm commented 1 year ago

看堆栈里有brpc::policy::CallMethodInBackupThread：

image

根据brpc 1.3.0的代码，CallMethodInBackupThread会在EndRunningCallMethodInPool中使用：

image

EndRunningCallMethodInPool只会在FLAGS_usercode_in_pthread为true的时候走到：

image

所以FLAGS_usercode_in_pthread的值应该为true，太奇怪了。

xiedeacc commented 1 year ago

FLAGS_usercode_in_pthread为true和不为false我都编译过，运行过，反正都会死锁

xiedeacc commented 1 year ago

看起来是DynamicPartitionChannel的bug

xiedeacc commented 1 year ago

基本可以确定DynamicPartitionChannel，改完之后，目前没有发生死锁了

dandyhuang commented 1 year ago

DynamicPartitionChannel

请教一下 ，有什么业务形态会用到DynamicPartitionChannel这个功能呢

xiedeacc commented 1 year ago

改完后，死锁概率降低了，但是还是会发生，目前只是把brpc_thread_num改成几百来规避这个死锁，但是brpc看起来是无法解决bthread使用pthread锁导致的死锁问题，brpc这个问题真的太无语了

xiedeacc commented 1 year ago

DynamicPartitionChannel

请教一下 ，有什么业务形态会用到DynamicPartitionChannel这个功能呢

多库

dandyhuang commented 1 year ago

DynamicPartitionChannel

启多个channel呢