chriskohlhoff / asio

Asio C++ Library
http://think-async.com/Asio
4.77k stars 1.2k forks source link

crash when sending data using async_write_some #773

Open code1w opened 3 years ago

code1w commented 3 years ago

I am using Asio only Asio Only Source Code to handle my network class. I have been troubling for a while and I can't get the reason why. My server has been running for more than 20 hours for a long time. During this period, I stress test the server uninterruptedly. Sometimes it crashes, sometimes it does not crash. My code

void AsioChannel::AsyncSendData(const std::shared_ptr<Buffer>& packet)
{
  sendding_.store(true);
  auto self = shared_from_this();
  socket_.async_write_some(asio::buffer(packet->peek(), packet->readableBytes()),
      [self, packet](const std::error_code& err, std::size_t writen)
      {
          self->AsyncWriteSomeCallback(err,packet,writen);
      });
}

void  AsioChannel::AsyncWriteSomeCallback(const std::error_code& error, const std::shared_ptr<Buffer>& packet, std::size_t writen)
{
  if (error)
  {
    return;
  }

  if (writen < packet->readableBytes())
  {
      packet->retrieve(writen);
      AsyncSendData(packet);
  }
  else
  {
      if(!waitwriteq_.empty())
      {
          auto next = waitwriteq_.front();
          waitwriteq_.pop();
          AsyncSendData(next);
      }
      else
      {
          sendding_.store(false);
      }
  }

}

The crash stack is as follow:

#0  0x00007f172a8255f7 in raise () from /lib64/libc.so.6
#1  0x00007f172a826e28 in abort () from /lib64/libc.so.6
#2  0x00007f172a865317 in __libc_message () from /lib64/libc.so.6
#3  0x00007f172a86cfe1 in _int_free () from /lib64/libc.so.6
#4  0x00007f172b92b49d in allocate<asio::detail::thread_info_base::default_tag> (size=168, this_thread=<optimized out>) at /opt/env/include/c++/10.2.0/ext/atomicity.h:97
#5  allocate (size=168, this_thread=<optimized out>) at asio/detail/thread_info_base.hpp:73
#6  asio_handler_allocate (size=168) at asio/impl/handler_alloc_hook.ipp:34
#7  allocate<net::AsioChannel::AsyncSendData(const std::shared_ptr<Buffer>&)::<lambda(const std::error_code&, std::size_t)> > (h=..., s=168)
    at asio/detail/handler_alloc_helpers.hpp:70
#8  allocate (n=1, this=<synthetic pointer>) at asio/detail/handler_alloc_helpers.hpp:127
#9  allocate (handler=...) at asio/detail/reactive_socket_send_op.hpp:103
#10 async_send<asio::const_buffers_1, net::AsioChannel::AsyncSendData(const std::shared_ptr<Buffer>&)::<lambda(const std::error_code&, std::size_t)>, asio::execution::any_executor<asio::execution::context_as_t<asio::execution_context&>, asio::execution::detail::blocking::never_t<0>, asio::execution::prefer_only<asio::execution::detail::blocking::possibly_t<0> >, asio::execution::prefer_only<asio::execution::detail::outstanding_work::tracked_t<0> >, asio::execution::prefer_only<asio::execution::detail::outstanding_work::untracked_t<0> >, asio::execution::prefer_only<asio::execution::detail::relationship::fork_t<0> >, asio::execution::prefer_only<asio::execution::detail::relationship::continuation_t<0> > > > (io_ex=..., handler=..., flags=0,
    buffers=..., impl=..., this=0x7f17080009f8) at asio/detail/reactive_socket_service_base.hpp:281
#11 operator()<net::AsioChannel::AsyncSendData(const std::shared_ptr<Buffer>&)::<lambda(const std::error_code&, std::size_t)>, asio::const_buffers_1> (flags=0, buffers=...,
    handler=<unknown type in /libgamesh.net.so, CU 0x0, DIE 0x6e7c8>, this=<optimized out>)
    at asio/basic_stream_socket.hpp:1005
#12 initiate<asio::basic_stream_socket<asio::ip::tcp>::initiate_async_send, net::AsioChannel::AsyncSendData(const std::shared_ptr<Buffer>&)::<lambda(const std::error_code&, std::size_t)>, const asio::const_buffers_1&, int> (token=<unknown type in /libgamesh.net.so, CU 0x0, DIE 0x6e78e>,
    initiation=<optimized out>) at asio/async_result.hpp:151
#13 async_initiate<net::AsioChannel::AsyncSendData(const std::shared_ptr<Buffer>&)::<lambda(const std::error_code&, std::size_t)>, void(std::error_code, long unsigned int), asio::basic_stream_socket<asio::ip::tcp>::initiate_async_send, const asio::const_buffers_1&, int> (token=..., initiation=<optimized out>)
    at asio/async_result.hpp:364
#14 async_write_some<asio::const_buffers_1, net::AsioChannel::AsyncSendData(const std::shared_ptr<Buffer>&)::<lambda(const std::error_code&, std::size_t)> > (
    handler=<unknown type in /libgamesh.net.so, CU 0x0, DIE 0x6e6e1>, buffers=..., this=0x7f1708143480)
    at asio/basic_stream_socket.hpp:855
#15 net::AsioChannel::AsyncSendData (this=0x7f1708143460, packet=...) at net/asio_channel.cpp:326
#16 0x00007f172b92b964 in net::AsioChannel::SendPacket (this=<optimized out>, packet=..., size=<optimized out>)
    at net/asio_channel.cpp:284
#17 0x00007f172b93cb1e in net::AsioServerHandler::SendPacket (this=<optimized out>, pPacket=..., size=<optimized out>) at /opt/env/include/c++/10.2.0/bits/shared_ptr_base.h:1324
#18 0x00007f172b946831 in operator() (__closure=<optimized out>) at /opt/env/include/c++/10.2.0/bits/shared_ptr_base.h:1324
#19 __invoke_impl<void, net::Asio_SendPacket(uint32_t, char const*, uint32_t)::<lambda()>&> (__f=...) at /opt/env/include/c++/10.2.0/bits/invoke.h:60
#20 __invoke_r<void, net::Asio_SendPacket(uint32_t, char const*, uint32_t)::<lambda()>&> (__fn=...) at /opt/env/include/c++/10.2.0/bits/invoke.h:110
#21 std::_Function_handler<void(), net::Asio_SendPacket(uint32_t, char const*, uint32_t)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...)
    at /opt/env/include/c++/10.2.0/bits/std_function.h:291
#22 0x00007f172c42ee26 in std::function<void ()>::operator()() const (this=<optimized out>) at /opt/env/include/c++/10.2.0/bits/std_function.h:248
#23 0x00007f172b93b0b4 in asio_handler_invoke<std::function<void()> > (function=...)
    at asio/handler_invoke_hook.hpp:85
#24 invoke<std::function<void()>, std::function<void()> > (context=..., function=...)
    at asio/detail/handler_invoke_helpers.hpp:54
#25 complete<std::function<void()> > (handler=..., function=..., this=<synthetic pointer>)
    at asio/detail/handler_work.hpp:424
#26 asio::detail::completion_handler<std::function<void ()>, asio::io_context::basic_executor_type<std::allocator<void>, 0u> >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (owner=0x5011470, base=0x7f15c7b2f4a0) at asio/detail/completion_handler.hpp:73
#27 0x00007f172b932d88 in complete (bytes_transferred=0, ec=..., owner=0x5011470, this=0x7f15c7b2f4a0)
    at asio/detail/scheduler_operation.hpp:39
#28 asio::detail::scheduler::do_run_one (this=this@entry=0x5011470, lock=..., this_thread=..., ec=...)
    at asio/detail/impl/scheduler.ipp:480
#29 0x00007f172b934336 in asio::detail::scheduler::run (this=0x5011470, ec=...)
    at asio/detail/impl/scheduler.ipp:203
---Type <return> to continue, or q <return> to quit---
#30 0x00007f172b9394c6 in run (this=0x50113c8) at asio/impl/io_context.ipp:62
#31 net::AsioEventLoop::Run (this=0x50113c0) at net/asio_event_loop.cpp:30
#32 0x00007f172b942ae1 in net::EventLoopThread::Run(std::function<int ()> const&, std::function<int ()> const&) (this=0x4fe9ec0, pre=..., post=...)
    at /opt/env/include/c++/10.2.0/bits/unique_ptr.h:421
#33 0x00007f172b942e6f in __invoke_impl<void, void (net::EventLoopThread::*&)(const std::function<int()>&, const std::function<int()>&), net::EventLoopThread*&, std::function<int()>&, std::function<int()>&> (__t=<optimized out>, __f=<optimized out>) at /opt/env/include/c++/10.2.0/bits/invoke.h:89
#34 __invoke<void (net::EventLoopThread::*&)(const std::function<int()>&, const std::function<int()>&), net::EventLoopThread*&, std::function<int()>&, std::function<int()>&> (
    __fn=<optimized out>) at /opt/env/include/c++/10.2.0/bits/invoke.h:95
#35 __call<void, 0, 1, 2> (__args=<optimized out>, this=<optimized out>) at /opt/env/include/c++/10.2.0/functional:416
#36 operator()<> (this=<optimized out>) at /opt/env/include/c++/10.2.0/functional:499
#37 __invoke_impl<void, std::_Bind<void (net::EventLoopThread::*(net::EventLoopThread*, std::function<int()>, std::function<int()>))(const std::function<int()>&, const std::function<int()>&)> > (__f=<optimized out>) at /opt/env/include/c++/10.2.0/bits/invoke.h:60
#38 __invoke<std::_Bind<void (net::EventLoopThread::*(net::EventLoopThread*, std::function<int()>, std::function<int()>))(const std::function<int()>&, const std::function<int()>&)> > (__fn=<optimized out>) at /opt/env/include/c++/10.2.0/bits/invoke.h:95
#39 _M_invoke<0> (this=<optimized out>) at /opt/env/include/c++/10.2.0/thread:264
#40 operator() (this=<optimized out>) at /opt/env/include/c++/10.2.0/thread:271
#41 std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::_Bind<void (net::EventLoopThread::*(net::EventLoopThread*, std::function<int ()>, std::function<int ()>))(std::function<int ()> const&, std::function<int ()> const&)> > > >::_M_run() (this=<optimized out>) at /opt/env/include/c++/10.2.0/thread:215
#42 0x00007f172aca2f70 in execute_native_thread_routine () at ../../../../../libstdc++-v3/src/c++11/thread.cc:80
#43 0x00007f172c1c8dc5 in start_thread () from /lib64/libpthread.so.0
#44 0x00007f172a8e61cd in clone () from /lib64/libc.so.6
(gdb)

By the way : io_service run in single thread

code1w commented 3 years ago

PS: Some more info:

Compiled with g++ 10.2.0 in C++17 mode

Running libstdc++ of GCC10 for C++17 feature support

Running on Red Hat Enterprise Linux Server release 7.2 (Maipo)

Everything compiled with -O2 
code1w commented 3 years ago

I do not know why it crash at asio_handler_allocate function , should i open ASIO_DISABLE_SMALL_BLOCK_RECYCLING macro?

lefterise commented 3 years ago

you pass the shared pointer by reference. this means that it doesn't increase the usecount maybe this is the reason. please include the code that starts the send chain as well. the one that checks the sendding_ variable.

on an unrelated note, the error handling code does not reset the sendding_ variable to false