search

baidu / sofa-pbrpc

A light-weight RPC implement of google protobuf RPC framework.
Other
2.13k stars 656 forks source link

sofa client出core #213

Closed scottzzq closed 7 years ago

scottzzq commented 7 years ago

0 0x0000000000cb30a4 in async_result_init (orig_handler=..., this=) at thirdparty/boost/asio/async_result.hpp:62

1 post<boost::_bi::bind_t<void, void ()(google::protobuf::Closure), boost::_bi::list1<boost::_bi::value<google::protobuf::Closure*> > >&> (this=,

handler=...) at thirdparty/boost/asio/impl/io_service.hpp:106

2 post<boost::_bi::bind_t<void, void ()(google::protobuf::Closure), boost::_bi::list1<boost::_bi::value<google::protobuf::Closure*> > > > (handler=...,

this=<optimized out>) at ./sofa/pbrpc/thread_group_impl.h:222

3 post (handle=0x35375bf40, this=) at ./sofa/pbrpc/thread_group_impl.h:232

4 sofa::pbrpc::SimpleRpcChannelImpl::DoneCallback (this=0x1b5663900, done=0x35375bf40, cntl=...) at sofa/pbrpc/simple_rpc_channel_impl.cc:180

5 0x0000000000c9936b in operator() (a0=..., this=) at thirdparty/boost/function/function_template.hpp:767

6 sofa::pbrpc::RpcControllerImpl::Done (this=0x31d82b520, error_code=, reason=...) at ./sofa/pbrpc/rpc_controller_impl.h:222

7 0x0000000000c8f27e in sofa::pbrpc::RpcClientImpl::CallMethod (this=, request=request@entry=0x31394c480, response=response@entry=0x22442f4a0, cntl=...)

at sofa/pbrpc/rpc_client_impl.cc:227

8 0x0000000000cb4012 in sofa::pbrpc::SimpleRpcChannelImpl::CallMethod (this=0x1b5663900, method=, controller=0x313135880, request=0x31394c480,

response=0x22442f4a0, done=<optimized out>) at sofa/pbrpc/simple_rpc_channel_impl.cc:147

9 0x0000000000cb9986 in sofa::pbrpc::DynamicRpcChannelImpl::CallMethod (this=0x1770a3a00, method=0x1b3de0120, controller=0x313135880, request=0x31394c480,

response=0x22442f4a0, done=0x35375bf40) at sofa/pbrpc/dynamic_rpc_channel_impl.cc:204

10 0x0000000000b84a45 in libzp::Cluster::DoNodeSyncTask (arg=0x313134900) at jtable-sofa-client/libzp/src/zp_cluster.cc:1148

11 0x0000000000b84b1f in libzp::Cluster::AddNodeAsyncTask (node=..., context=) at jtable-sofa-client/libzp/src/zp_cluster.cc:1154

12 0x0000000000b89951 in libzp::Cluster::DoMgetAsyncTask (arg=0x34b4aee00) at jtable-sofa-client/libzp/src/zp_cluster.cc:871

13 0x0000000000b8a895 in libzp::Cluster::DoAsyncTask_get (arg=arg@entry=0x34b4aee00) at jtable-sofa-client/libzp/src/zp_cluster.cc:934

14 0x0000000000b8a9cd in libzp::Cluster::AddAsyncTask_get (this=this@entry=0x2f06240, context=context@entry=0x34b4aee00) at jtable-sofa-client/libzp/src/zp_cluster.cc:982

15 0x0000000000b8abae in libzp::Cluster::Amget (this=0x2f06240, table=..., keys=...,

函数调用栈如图， 求解答，谢谢！

scottzzq commented 7 years ago

@qinzuoyan 不知道是否和编译器有关，我们使用gcc5.2编译的

qinzuoyan commented 7 years ago

看起来应当不是编译器的问题。我觉得很大可能是RpcClient对象销毁后，继续发起RPC请求导致的。原因如下：

  1. 首先这是在调用Closure回调函数的时候出core；
  2. 第7层栈显示，因为进入了_is_running=false的分支，所以调用了cntl->Done()函数以给用户返回错误：rpc_client_impl.cc#L221（注，你的代码应当不是最新的，所以代码行数有偏差）。我推断此时RpcClientImpl应当是调用过Stop()函数，进一步推断是RpcClient对象析构调用了RpcClient::ShutDown()函数导致。
  3. RpcClientImpl::Stop()中会stop掉_callback_thread_group：rpc_client_impl.cc#L114。
  4. 第4层栈显示，在调用cntl->Done()的时候，需要将回调函数分派到callback_thread_group中去：simple_rpc_channel_impl.cc#L178。如果此时callback_thread_group已经Stop掉了，可能会导致core。
  5. 因为没有更多信息，以上只是我的初步猜测。

建议：检查下你的RpcClient对象的生命周期，是否在析构后还可能存在继续发送请求的情况？因为创建RpcChannel时，构造函数传入的是RpcClient的裸指针，所以如果RpcClient对象析构了，但是RpcChannel还在继续使用，那么就会使用RpcClient对象的野指针，造成core。

scottzzq commented 7 years ago

经过排查，的确是这种情况，排查了好几天，之前一直以为是boost库的问题和编译器的问题。非常感谢@qinzuoyan 。

qinzuoyan commented 7 years ago

OK，不客气

On Thu, Nov 2, 2017 at 10:40 AM, scottzzq notifications@github.com wrote:

经过排查，的确是这种情况，排查了好几天，之前一直以为是boost库的问题和编译器的问题。非常感谢@qinzuoyan https://github.com/qinzuoyan 。

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/baidu/sofa-pbrpc/issues/213#issuecomment-341301285, or mute the thread https://github.com/notifications/unsubscribe-auth/AAgQPTDh1a_Q_8PibMEazK41_TEo3zgrks5sySuHgaJpZM4QONUK .