Closed greenday9 closed 8 years ago
This issue can be reproduced with the following test program.
void post_method_handler( const shared_ptr< Session > session )
{
const auto request = session->get_request( );
size_t content_length = request->get_header( "Content-Length", 0 );
session->fetch( content_length, [ request ]( const shared_ptr< Session > session, const Bytes & body )
{
fprintf( stdout, "%.*s\n", ( int ) body.size( ), body.data( ) );
session->sleep_for( std::chrono::milliseconds(5), []( const shared_ptr< Session > session )
{
session->yield( OK, "Hello, World!", { { "Content-Length", "13" }, { "Connection", "close" } } );
}
);
} );
}
int main( const int, const char** )
{
auto resource = make_shared< Resource >( );
resource->set_path( "/resource" );
resource->set_method_handler( "POST", post_method_handler );
auto settings = make_shared< Settings >( );
settings->set_port( 8010 );
settings->set_worker_limit( 10 );
Service service;
service.publish( resource );
service.start( settings );
return EXIT_SUCCESS;
}
Sending a load of requests into the test program causes it to coredump very quickly. Note that sometimes I see other crashes besides an unhandled exception. It may be a multi-threading issue since I don't see it if I set worker_limit to 1.
Actually, this does not appear to be related to the number of worker threads. Even with worker_limit set to 1, the same issue occurs. It just takes longer.
I'll look at wrapping the connection_timeout_handler in a try-catch block and logging the error when it occurs. This will fix the issue.
Give me a couple of hours to confirm the fix.
How are you interacting with the above example? curl, wget, Restbed::Http?
From a precursory review of the stack-trace it appears the socket is timing out, calling close and ASIO is throwing an exception on a bad socket.
void SocketImpl::connection_timeout_handler( const error_code& error )
{
if ( error or m_timer->expires_at( ) > steady_clock::now( ) )
{
return;
}
close( ); //this method calls m_socket->close.
}
My hunch is the socket is killed by the client, resulting in 'bad descriptor'. The session times out and calls close on an invalid descriptor resulting in the system_error.
FYI. I also ran with valgrind and I see the following error before crashing. There are other similar errors that follow.
==45696== Invalid read of size 8
==45696== at 0x5260514: std::__shared_ptr<asio::basic_waitable_timer<std::chrono::_V2::steady_clock, asio::wait_traits<std::chrono::_V2::steady_clock>, asio::waitable_timer_service<std::chrono::_V2::steady_clock, asio::wait_traits<std::chrono::_V2::steady_clock> > >, (__gnu_cxx::_Lock_policy)2>::operator->() const (shared_ptr_base.h:1055)
==45696== by 0x5291953: restbed::detail::SocketImpl::connection_timeout_handler(std::error_code const&) (socket_impl.cpp:440)
==45696== by 0x52AE069: void std::_Mem_fn_base<void (restbed::detail::SocketImpl::*)(std::error_code const&), true>::operator()<std::error_code const&, void>(restbed::detail::SocketImpl*, std::error_code const&) const (in /home/kgreenwell/adev/my-repo/sdk/lib/librestbed.so)
==45696== by 0x52ADE82: void std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>::__call<void, std::error_code const&, 0ul, 1ul>(std::tuple<std::error_code const&>&&, std::_Index_tuple<0ul, 1ul>) (functional:1074)
==45696== by 0x52ADAEA: void std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>::operator()<std::error_code const&, void>(std::error_code const&) (functional:1133)
==45696== by 0x52AD488: asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>::operator()() (bind_handler.hpp:64)
==45696== by 0x52ACD48: void asio::asio_handler_invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, ...) (handler_invoke_hook.hpp:68)
==45696== by 0x52AC6C5: void asio_handler_invoke_helpers::invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>&) (handler_invoke_helpers.hpp:37)
==45696== by 0x52ABD0A: void asio::detail::handler_work<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::system_executor>::complete<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>&) (handler_work.hpp:81)
==45696== by 0x52AAB48: asio::detail::wait_handler<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (wait_handler.hpp:71)
==45696== by 0x524294F: asio::detail::scheduler_operation::complete(void*, std::error_code const&, unsigned long) (scheduler_operation.hpp:39)
==45696== by 0x5243609: asio::detail::scheduler::do_run_one(asio::detail::scoped_lock<asio::detail::posix_mutex>&, asio::detail::scheduler_thread_info&, std::error_code const&) (scheduler.ipp:375)
==45696== by 0x52431F5: asio::detail::scheduler::run(std::error_code&) (scheduler.ipp:146)
==45696== by 0x5243928: asio::io_context::run() (io_context.ipp:59)
==45696== by 0x5257FBD: restbed::Service::start(std::shared_ptr<restbed::Settings const> const&) (service.cpp:197)
==45696== by 0x40D9F0: main (testprogram.cpp:49)
==45696== Address 0xd80fbb8 is 72 bytes inside a block of size 136 free'd
==45696== at 0x4C2B131: operator delete(void*) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==45696== by 0x528E659: __gnu_cxx::new_allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> >::deallocate(std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2>*, unsigned long) (new_allocator.h:110)
==45696== by 0x528E0A6: std::allocator_traits<std::allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> > >::deallocate(std::allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> >&, std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2>*, unsigned long) (alloc_traits.h:517)
==45696== by 0x528D7FB: std::__allocated_ptr<std::allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> > >::~__allocated_ptr() (allocated_ptr.h:72)
==45696== by 0x528F4E8: std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2>::_M_destroy() (shared_ptr_base.h:539)
==45696== by 0x40ECC7: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:167)
==45696== by 0x40E6E8: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:659)
==45696== by 0x525372F: std::__shared_ptr<restbed::detail::SocketImpl, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:925)
==45696== by 0x5253767: std::shared_ptr<restbed::detail::SocketImpl>::~shared_ptr() (shared_ptr.h:93)
==45696== by 0x5254B85: restbed::detail::RequestImpl::~RequestImpl() (request_impl.hpp:38)
==45696== by 0x5254C5D: std::default_delete<restbed::detail::RequestImpl>::operator()(restbed::detail::RequestImpl*) const (unique_ptr.h:76)
==45696== by 0x5253E12: std::unique_ptr<restbed::detail::RequestImpl, std::default_delete<restbed::detail::RequestImpl> >::~unique_ptr() (unique_ptr.h:236)
==45696== by 0x5252504: restbed::Request::~Request() (request.cpp:65)
==45696== by 0x52CEBE0: void __gnu_cxx::new_allocator<restbed::Request>::destroy<restbed::Request>(restbed::Request*) (new_allocator.h:124)
==45696== by 0x52CEAC6: void std::allocator_traits<std::allocator<restbed::Request> >::destroy<restbed::Request>(std::allocator<restbed::Request>&, restbed::Request*) (alloc_traits.h:542)
==45696== by 0x52CE548: std::_Sp_counted_ptr_inplace<restbed::Request, std::allocator<restbed::Request>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr_base.h:531)
==45696== by 0x40EC91: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:150)
==45696== by 0x40E6E8: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:659)
==45696== by 0x40E307: std::__shared_ptr<restbed::Request const, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:925)
==45696== by 0x40E323: std::shared_ptr<restbed::Request const>::~shared_ptr() (shared_ptr.h:93)
==45696== by 0x52CEF56: restbed::detail::SessionImpl::~SessionImpl() (session_impl.cpp:80)
==45696== by 0x52CEF95: restbed::detail::SessionImpl::~SessionImpl() (session_impl.cpp:83)
==45696== by 0x5275D45: std::default_delete<restbed::detail::SessionImpl>::operator()(restbed::detail::SessionImpl*) const (unique_ptr.h:76)
==45696== by 0x527557A: std::unique_ptr<restbed::detail::SessionImpl, std::default_delete<restbed::detail::SessionImpl> >::~unique_ptr() (unique_ptr.h:236)
==45696== by 0x526F038: restbed::Session::~Session() (session.cpp:50)
==45696== by 0x5285348: void __gnu_cxx::new_allocator<restbed::Session>::destroy<restbed::Session>(restbed::Session*) (new_allocator.h:124)
==45696== by 0x5285314: void std::allocator_traits<std::allocator<restbed::Session> >::destroy<restbed::Session>(std::allocator<restbed::Session>&, restbed::Session*) (alloc_traits.h:542)
==45696== by 0x5285222: std::_Sp_counted_ptr_inplace<restbed::Session, std::allocator<restbed::Session>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr_base.h:531)
==45696== by 0x40EC91: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:150)
==45696== by 0x40E6E8: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:659)
==45696== by 0x40F081: std::__shared_ptr<restbed::Session, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:925)
==45696== by 0x40F09D: std::shared_ptr<restbed::Session>::~shared_ptr() (shared_ptr.h:93)
==45696== by 0x5275133: restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1}::~error_code() (session.cpp:238)
==45696== by 0x527425B: std::_Function_base::_Base_manager<restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1}>::_M_destroy(std::_Any_data&, std::integral_constant<bool, false>) (functional:1726)
==45696== by 0x527356E: std::_Function_base::_Base_manager<restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1}>::_M_manager(std::_Any_data&, restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1} const&, std::_Manager_operation) (functional:1750)
==45696== by 0x40E296: std::_Function_base::~_Function_base() (functional:1830)
==45696== by 0x5244441: std::function<void (std::error_code const&, unsigned long)>::~function() (functional:1974)
==45696== by 0x529101B: restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}::~function() (socket_impl.cpp:334)
==45696== by 0x52929B5: asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::basic_streambuf_ref<std::allocator<char> >, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}>::~read_until_delim_string_op() (read_until.hpp:668)
==45696== by 0x5299D4B: asio::detail::binder2<asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::basic_streambuf_ref<std::allocator<char> >, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}>, std::error_code, unsigned long>::~binder2() (bind_handler.hpp:126)
==45696== by 0x5299E5B: asio::detail::reactive_socket_recv_op<asio::mutable_buffers_1, asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::basic_streambuf_ref<std::allocator<char> >, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}> >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (reactive_socket_recv_op.hpp:107)
==45696== by 0x524294F: asio::detail::scheduler_operation::complete(void*, std::error_code const&, unsigned long) (scheduler_operation.hpp:39)
==45696== by 0x52A1562: asio::detail::epoll_reactor::descriptor_state::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (epoll_reactor.ipp:695)
==45696== by 0x524294F: asio::detail::scheduler_operation::complete(void*, std::error_code const&, unsigned long) (scheduler_operation.hpp:39)
==45696== by 0x5243609: asio::detail::scheduler::do_run_one(asio::detail::scoped_lock<asio::detail::posix_mutex>&, asio::detail::scheduler_thread_info&, std::error_code const&) (scheduler.ipp:375)
==45696== by 0x52431F5: asio::detail::scheduler::run(std::error_code&) (scheduler.ipp:146)
==45696== by 0x5243928: asio::io_context::run() (io_context.ipp:59)
==45696== by 0x5257FBD: restbed::Service::start(std::shared_ptr<restbed::Settings const> const&) (service.cpp:197)
==45696== by 0x40D9F0: main (testprogram.cpp:49)
In terms of generating load into the example. I use the following simple node script.
E.g. node load_test.js localhost 1000 2
var http = require('http');
var postData = "testing 123";
var requestCount = 0;
if(process.argv.length < 4)
{
console.log('ERROR: Missing hostname and TPS!');
process.exit(1);
}
var iterations = 1;
var tps = process.argv[3];
if(process.argv.length > 4)
{
iterations = process.argv[4];
}
console.log(`connecting to ${process.argv[2]} at ${tps} TPS with ${iterations} iterations`);
var options = {
host: process.argv[2],
path: '/resource',
port: '8010',
method: 'POST',
headers: {
'Content-Type' : 'text/plain',
'Content-Length' : Buffer.byteLength(postData)
}
};
callback = function(response) {
if(response.statusCode != 200)
console.log(`STATUS: ${response.statusCode}`);
}
setInterval(function() {
for(var i = 0; i < iterations; i++)
{
var req = http.request(options, callback);
req.write(postData);
req.end();
requestCount++;
}
}, 1000 / tps);
setInterval(function() {
console.log(`TPS: ${requestCount}`);
requestCount = 0;
}, 1000);
Thanks.
for(var i = 0; i < iterations; i++)
{
var req = http.request(options, callback);
req.write(postData);
req.end();
requestCount++;
}
As stated previously. I believe each request (client) has a restricted scope and is being reclaimed after the end of the for-loop/req.end. This causes the client to close the connection and once the timeout is fired it creates the bad socket operation on the server.
Curious as to why ASIO would throw an error on socket->close and not just report success. Either way poor handling on our end.
I pushed fix for this issue, further validation required.
@callum-kirby please provide a regression test.
@greenday9 We've managed to replicate this issue and resolve it via @callum-kirby's fix. Please can you confirm on your end.
It runs longer now. But I've hit a coredump. I'm running the same example program with the same load injection script.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fb5e866d318 in asio::detail::reactive_socket_service_base::close (this=0x28, impl=..., ec=...)
at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/impl/reactive_socket_service_base.ipp:105
105 reactor_.deregister_descriptor(impl.socket_, impl.reactor_data_,
(gdb) where
#0 0x00007fb5e866d318 in asio::detail::reactive_socket_service_base::close (this=0x28, impl=..., ec=...)
at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/impl/reactive_socket_service_base.ipp:105
#1 0x00007fb5e867057c in asio::stream_socket_service<asio::ip::tcp>::close (this=0x0, impl=..., ec=...)
at /home/user/extern/restbed/dependency/asio/asio/include/asio/stream_socket_service.hpp:162
#2 0x00007fb5e866f86b in asio::basic_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >::close (this=0x2645730)
at /home/user/extern/restbed/dependency/asio/asio/include/asio/basic_socket.hpp:352
#3 0x00007fb5e8659f0e in restbed::detail::SocketImpl::close (this=0x220ce80)
at /home/user/extern/restbed/source/corvusoft/restbed/detail/socket_impl.cpp:93
#4 0x00007fb5e865b9a2 in restbed::detail::SocketImpl::connection_timeout_handler (this=0x220ce80, error=...)
at /home/user/extern/restbed/source/corvusoft/restbed/detail/socket_impl.cpp:449
#5 0x00007fb5e867810c in std::_Mem_fn_base<void (restbed::detail::SocketImpl::*)(std::error_code const&), true>::operator()<std::error_code const&, void> (
this=0x7ffdd413b490, __object=0x220ce80) at /usr/local/include/c++/5.4.0/functional:600
#6 0x00007fb5e8677f25 in std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>::__call<void, std::error_code const&, 0ul, 1ul>(std::tuple<std::error_code const&>&&, std::_Index_tuple<0ul, 1ul>) (this=0x7ffdd413b490,
__args=<unknown type in /home/user/sdk/lib/librestbed.so, CU 0x29bac1, DIE 0x2f9106>) at /usr/local/include/c++/5.4.0/functional:1074
#7 0x00007fb5e8677b8d in std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>::operator()<std::error_code const&, void>(std::error_code const&) (this=0x7ffdd413b490) at /usr/local/include/c++/5.4.0/functional:1133
#8 0x00007fb5e867752b in asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>::operator()() (this=0x7ffdd413b490)
at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/bind_handler.hpp:64
#9 0x00007fb5e8676deb in asio::asio_handler_invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, ...) (function=...)
at /home/user/extern/restbed/dependency/asio/asio/include/asio/handler_invoke_hook.hpp:68
#10 0x00007fb5e8676768 in asio_handler_invoke_helpers::invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>&) (function=..., context=...)
at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/handler_invoke_helpers.hpp:37
#11 0x00007fb5e8675dad in asio::detail::handler_work<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::system_executor>::complete<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>&) (this=0x7ffdd413b4bf, function=..., handler=...)
at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/handler_work.hpp:81
#12 0x00007fb5e8674beb in asio::detail::wait_handler<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (owner=0x1292f80, base=0x1671ea0)
at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/wait_handler.hpp:71
#13 0x00007fb5e860c950 in asio::detail::scheduler_operation::complete (this=0x1671ea0, owner=0x1292f80, ec=..., bytes_transferred=0)
at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/scheduler_operation.hpp:39
#14 0x00007fb5e860d60a in asio::detail::scheduler::do_run_one (this=0x1292f80, lock=..., this_thread=..., ec=...)
at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/impl/scheduler.ipp:375
#15 0x00007fb5e860d1f6 in asio::detail::scheduler::run (this=0x1292f80, ec=...)
at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/impl/scheduler.ipp:146
#16 0x00007fb5e860d929 in asio::io_context::run (this=0x1292f10)
at /home/user/extern/restbed/dependency/asio/asio/include/asio/impl/io_context.ipp:59
#17 0x00007fb5e8621fbe in restbed::Service::start (this=0x7ffdd413b7f0, settings=...)
at /home/user/extern/restbed/source/corvusoft/restbed/service.cpp:197
#18 0x000000000040d9f1 in main () at /home/user/testprogram/testprogram.cpp:49
valgrind also reports a memory issue as well during runs. It appears in some cases SocketImpl is deleted (via Session::yield) and then the connect_timeout_handler callback fires on the the deleted SocketImpl.
==41342== Invalid read of size 8
==41342== at 0x5260514: std::__shared_ptr<asio::basic_waitable_timer<std::chrono::_V2::steady_clock, asio::wait_traits<std::chrono::_V2::steady_clock>, asio::waitable_timer_service<std::chrono::_V2::steady_clock, asio::wait_traits<std::chrono::_V2::steady_clock> > >, (__gnu_cxx::_Lock_policy)2>::operator->() const (shared_ptr_base.h:1055)
==41342== by 0x5291953: restbed::detail::SocketImpl::connection_timeout_handler(std::error_code const&) (socket_impl.cpp:442)
==41342== by 0x52AE10B: void std::_Mem_fn_base<void (restbed::detail::SocketImpl::*)(std::error_code const&), true>::operator()<std::error_code const&, void>(restbed::detail::SocketImpl*, std::error_code const&) const (in /home/kgreenwell/adev/my-repo/sdk/lib/librestbed.so)
==41342== by 0x52ADF24: void std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>::__call<void, std::error_code const&, 0ul, 1ul>(std::tuple<std::error_code const&>&&, std::_Index_tuple<0ul, 1ul>) (functional:1074)
==41342== by 0x52ADB8C: void std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>::operator()<std::error_code const&, void>(std::error_code const&) (functional:1133)
==41342== by 0x52AD52A: asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>::operator()() (bind_handler.hpp:64)
==41342== by 0x52ACDEA: void asio::asio_handler_invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, ...) (handler_invoke_hook.hpp:68)
==41342== by 0x52AC767: void asio_handler_invoke_helpers::invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>&) (handler_invoke_helpers.hpp:37)
==41342== by 0x52ABDAC: void asio::detail::handler_work<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::system_executor>::complete<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>&) (handler_work.hpp:81)
==41342== by 0x52AABEA: asio::detail::wait_handler<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (wait_handler.hpp:71)
==41342== by 0x524294F: asio::detail::scheduler_operation::complete(void*, std::error_code const&, unsigned long) (scheduler_operation.hpp:39)
==41342== by 0x5243609: asio::detail::scheduler::do_run_one(asio::detail::scoped_lock<asio::detail::posix_mutex>&, asio::detail::scheduler_thread_info&, std::error_code const&) (scheduler.ipp:375)
==41342== by 0x52431F5: asio::detail::scheduler::run(std::error_code&) (scheduler.ipp:146)
==41342== by 0x5243928: asio::io_context::run() (io_context.ipp:59)
==41342== by 0x5257FBD: restbed::Service::start(std::shared_ptr<restbed::Settings const> const&) (service.cpp:197)
==41342== by 0x40D9F0: main (testprogram.cpp:49)
==41342== Address 0xe7bf808 is 72 bytes inside a block of size 136 free'd
==41342== at 0x4C2B131: operator delete(void*) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==41342== by 0x528E659: __gnu_cxx::new_allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> >::deallocate(std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2>*, unsigned long) (new_allocator.h:110)
==41342== by 0x528E0A6: std::allocator_traits<std::allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> > >::deallocate(std::allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> >&, std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2>*, unsigned long) (alloc_traits.h:517)
==41342== by 0x528D7FB: std::__allocated_ptr<std::allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> > >::~__allocated_ptr() (allocated_ptr.h:72)
==41342== by 0x528F4E8: std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2>::_M_destroy() (shared_ptr_base.h:539)
==41342== by 0x40ECC7: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:167)
==41342== by 0x40E6E8: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:659)
==41342== by 0x525372F: std::__shared_ptr<restbed::detail::SocketImpl, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:925)
==41342== by 0x5253767: std::shared_ptr<restbed::detail::SocketImpl>::~shared_ptr() (shared_ptr.h:93)
==41342== by 0x5254B85: restbed::detail::RequestImpl::~RequestImpl() (request_impl.hpp:38)
==41342== by 0x5254C5D: std::default_delete<restbed::detail::RequestImpl>::operator()(restbed::detail::RequestImpl*) const (unique_ptr.h:76)
==41342== by 0x5253E12: std::unique_ptr<restbed::detail::RequestImpl, std::default_delete<restbed::detail::RequestImpl> >::~unique_ptr() (unique_ptr.h:236)
==41342== by 0x5252504: restbed::Request::~Request() (request.cpp:65)
==41342== by 0x52CEC84: void __gnu_cxx::new_allocator<restbed::Request>::destroy<restbed::Request>(restbed::Request*) (new_allocator.h:124)
==41342== by 0x52CEB6A: void std::allocator_traits<std::allocator<restbed::Request> >::destroy<restbed::Request>(std::allocator<restbed::Request>&, restbed::Request*) (alloc_traits.h:542)
==41342== by 0x52CE5EC: std::_Sp_counted_ptr_inplace<restbed::Request, std::allocator<restbed::Request>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr_base.h:531)
==41342== by 0x40EC91: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:150)
==41342== by 0x40E6E8: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:659)
==41342== by 0x40E307: std::__shared_ptr<restbed::Request const, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:925)
==41342== by 0x40E323: std::shared_ptr<restbed::Request const>::~shared_ptr() (shared_ptr.h:93)
==41342== by 0x52CEFFA: restbed::detail::SessionImpl::~SessionImpl() (session_impl.cpp:80)
==41342== by 0x52CF039: restbed::detail::SessionImpl::~SessionImpl() (session_impl.cpp:83)
==41342== by 0x5275D45: std::default_delete<restbed::detail::SessionImpl>::operator()(restbed::detail::SessionImpl*) const (unique_ptr.h:76)
==41342== by 0x527557A: std::unique_ptr<restbed::detail::SessionImpl, std::default_delete<restbed::detail::SessionImpl> >::~unique_ptr() (unique_ptr.h:236)
==41342== by 0x526F038: restbed::Session::~Session() (session.cpp:50)
==41342== by 0x5285348: void __gnu_cxx::new_allocator<restbed::Session>::destroy<restbed::Session>(restbed::Session*) (new_allocator.h:124)
==41342== by 0x5285314: void std::allocator_traits<std::allocator<restbed::Session> >::destroy<restbed::Session>(std::allocator<restbed::Session>&, restbed::Session*) (alloc_traits.h:542)
==41342== by 0x5285222: std::_Sp_counted_ptr_inplace<restbed::Session, std::allocator<restbed::Session>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr_base.h:531)
==41342== by 0x40EC91: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:150)
==41342== by 0x40E6E8: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:659)
==41342== by 0x40F081: std::__shared_ptr<restbed::Session, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:925)
==41342== by 0x40F09D: std::shared_ptr<restbed::Session>::~shared_ptr() (shared_ptr.h:93)
==41342== by 0x5275133: restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1}::~error_code() (session.cpp:238)
==41342== by 0x527425B: std::_Function_base::_Base_manager<restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1}>::_M_destroy(std::_Any_data&, std::integral_constant<bool, false>) (functional:1726)
==41342== by 0x527356E: std::_Function_base::_Base_manager<restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1}>::_M_manager(std::_Any_data&, restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1} const&, std::_Manager_operation) (functional:1750)
==41342== by 0x40E296: std::_Function_base::~_Function_base() (functional:1830)
==41342== by 0x5244441: std::function<void (std::error_code const&, unsigned long)>::~function() (functional:1974)
==41342== by 0x529101B: restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}::~function() (socket_impl.cpp:336)
==41342== by 0x5292A57: asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::basic_streambuf_ref<std::allocator<char> >, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}>::~read_until_delim_string_op() (read_until.hpp:668)
==41342== by 0x5299DED: asio::detail::binder2<asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::basic_streambuf_ref<std::allocator<char> >, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}>, std::error_code, unsigned long>::~binder2() (bind_handler.hpp:126)
==41342== by 0x5299EFD: asio::detail::reactive_socket_recv_op<asio::mutable_buffers_1, asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::basic_streambuf_ref<std::allocator<char> >, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}> >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (reactive_socket_recv_op.hpp:107)
==41342== by 0x524294F: asio::detail::scheduler_operation::complete(void*, std::error_code const&, unsigned long) (scheduler_operation.hpp:39)
==41342== by 0x52A1604: asio::detail::epoll_reactor::descriptor_state::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (epoll_reactor.ipp:695)
==41342== by 0x524294F: asio::detail::scheduler_operation::complete(void*, std::error_code const&, unsigned long) (scheduler_operation.hpp:39)
==41342== by 0x5243609: asio::detail::scheduler::do_run_one(asio::detail::scoped_lock<asio::detail::posix_mutex>&, asio::detail::scheduler_thread_info&, std::error_code const&) (scheduler.ipp:375)
==41342== by 0x52431F5: asio::detail::scheduler::run(std::error_code&) (scheduler.ipp:146)
==41342== by 0x5243928: asio::io_context::run() (io_context.ipp:59)
==41342== by 0x5257FBD: restbed::Service::start(std::shared_ptr<restbed::Settings const> const&) (service.cpp:197)
==41342== by 0x40D9F0: main (testprogram.cpp:49)
Attempting to replicate.
A few notes:
a) Please make sure you do indeed close the session on your end.
session->sleep_for( std::chrono::milliseconds(5), []( const shared_ptr< Session > session )
{
//session->yield( OK, "Hello, World!", { { "Content-Length", "13" }, { "Connection", "close" } } );
session->close( OK, "Hello, World!", { { "Content-Length", "13" }, { "Connection", "close" } } );
} );
b) Do you in fact have 10 cores available where the machine is running?
Here is the suggested approach to setting the worker limit.
settings->set_worker_limit( thread::hardware_concurrency( ) );
Yep confirmed on Ubuntu 10x threads on 4 core system; see what I can do.
@ben-crowhurst
a) How does close and yield differ? My assumption is that yield() leaves the connection open which is what we want in the case the client sends follow on requests. It is artificial in the test program (particularly since we return the "Connection: close" header).
b) No, we don't have 10 cores. This should be optimized for the number of cores/etc. However, the test program is simply replicating the issue. Thanks for the suggestion on using thread::hardware_concurrency().
@greenday9 You are correct. The yield with 'Connection: close' caught my eye and decided to mention it.
Attempting to resolve timer issue now.
Ok moving the SocketImpl::connection_timeout_handler to a static method resolves this issue.
Still attempting to locate how this situation arises in the first place.
@greenday9 Not seeing a service crash anymore, please rerun your stress suite.
@ben-crowhurst The situation arises as follows, given presence of worker threads:
I've confirmed the above scenario through painful testing with lots of debug messages with thread IDs :)
Just catching and silencing the exception that you pushed is definitely not a proper fix for this. This also explains why making connection_timeout_handler() static "fixed" the issue -- it was no longer operating on an already-destroyed object.
I tried fixing it by deriving SocketImpl from std::enable_shared_from_this
auto socket = shared_from_this(); socket->close();
That took care of the socket not getting destroyed (because the shared_ptr had its refcount upped) asynchronously.
However, that is still not a proper fix because a certain race condition remains: what if there are multiple worker threads and data becomes available for asio::async_read() at the very same precise moment that connection_timeout_handler() is called? The result will be that connection_timeout_handler() will close the socket that the asio::async_read() completion handler has already begun getting data from which is a nasty race condition.
Therefore the proper fix is to never allow async_read() / async_write() completion handlers run concurrently with connection_timeout_handler() by using an asio::io_service::strand.
I'll do a PR momentarily with that fix implemented.
My apologies – I am still quite new to Git. Will do a proper PR shortly.
It is definitely much more stable, but there is still an issue. If I kill the load injection tool while the test program is heavily loaded, I see the unhandled exception or coredump. This is easy to reproduce especially when the program is running with valgrind. See the following valgrind trace:
==48317== Thread 7:
==48317== Invalid read of size 8
==48317== at 0x5270528: std::__shared_ptr<asio::basic_waitable_timer<std::chrono::_V2::steady_clock, asio::wait_traits<std::chrono::_V2::steady_clock>, asio::waitable_timer_service<std::chrono::_V2::steady_clock, asio::wait_traits<std::chrono::_V2::steady_clock> > >, (__gnu_cxx::_Lock_policy)2>::operator->() const (shared_ptr_base.h:1055)
==48317== by 0x52A2005: restbed::detail::SocketImpl::connection_timeout_handler(std::error_code const&) (socket_impl.cpp:441)
==48317== by 0x52CDE8B: void std::_Mem_fn_base<void (restbed::detail::SocketImpl::*)(std::error_code const&), true>::operator()<std::error_code const&, void>(restbed::detail::SocketImpl*, std::error_code const&) const (in /home/user/sdk/lib/librestbed.so)
==48317== by 0x52CDA68: void std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>::__call<void, std::error_code const&, 0ul, 1ul>(std::tuple<std::error_code const&>&&, std::_Index_tuple<0ul, 1ul>) (functional:1074)
==48317== by 0x52CD440: void std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>::operator()<std::error_code const&, void>(std::error_code const&) (functional:1133)
==48317== by 0x52CCB9A: asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>::operator()() (bind_handler.hpp:64)
==48317== by 0x52CC3B2: void asio::asio_handler_invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, ...) (handler_invoke_hook.hpp:68)
==48317== by 0x52CBB4D: void asio_handler_invoke_helpers::invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>&) (handler_invoke_helpers.hpp:37)
==48317== by 0x52CEA46: void asio::detail::asio_handler_invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>*) (bind_handler.hpp:105)
==48317== by 0x52CE810: void asio_handler_invoke_helpers::invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>, asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&) (handler_invoke_helpers.hpp:37)
==48317== by 0x52CE61E: void asio::detail::strand_service::dispatch<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::strand_service::strand_impl*&, asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&) (strand_service.hpp:61)
==48317== by 0x52CE4E2: asio::async_result<asio::handler_type<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>, void (), void>::type>::type asio::io_context::strand::dispatch<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&&) (io_context_strand.hpp:221)
==48317== by 0x52CE46B: void asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>::operator()<std::error_code>(std::error_code const&) (wrapped_handler.hpp:86)
==48317== by 0x52CE42A: asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>::operator()() (bind_handler.hpp:64)
==48317== by 0x52CE3D5: asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >::operator()() (wrapped_handler.hpp:190)
==48317== by 0x52CE2F9: void asio::asio_handler_invoke<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> > >(asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >&, ...) (handler_invoke_hook.hpp:68)
==48317== by 0x52CE00E: void asio_handler_invoke_helpers::invoke<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >(asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >&, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>&) (handler_invoke_helpers.hpp:37)
==48317== by 0x52CDBC6: void asio::detail::asio_handler_invoke<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >, asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >(asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >&, asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >*) (wrapped_handler.hpp:274)
==48317== by 0x52CD738: void asio_handler_invoke_helpers::invoke<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >, asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> > >(asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >&, asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >&) (handler_invoke_helpers.hpp:37)
==48317== by 0x52CDDDC: void asio::detail::handler_work<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >, asio::system_executor>::complete<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> > >(asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >&, asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >&) (handler_work.hpp:81)
==48317== by 0x52CD8FF: asio::detail::completion_handler<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> > >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (completion_handler.hpp:69)
==48317== by 0x52CD34F: void asio::detail::strand_service::dispatch<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> > >(asio::detail::strand_service::strand_impl*&, asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >&) (strand_service.hpp:87)
==48317== by 0x52CCAEA: asio::async_result<asio::handler_type<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >, void (), void>::type>::type asio::io_context::strand::dispatch<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> > >(asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >&&) (io_context_strand.hpp:221)
==48317== by 0x52CC287: void asio::detail::asio_handler_invoke<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>(asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>&, asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>*) (wrapped_handler.hpp:231)
==48317== by 0x52CBA4E: void asio_handler_invoke_helpers::invoke<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running> >(asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>&, asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>&) (handler_invoke_helpers.hpp:37)
==48317== by 0x52CAE08: void asio::detail::handler_work<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, asio::system_executor>::complete<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code> >(asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>&, asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>&) (handler_work.hpp:81)
==48317== by 0x52C9761: asio::detail::wait_handler<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running> >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (wait_handler.hpp:71)
==48317== by 0x525282D: asio::detail::scheduler_operation::complete(void*, std::error_code const&, unsigned long) (scheduler_operation.hpp:39)
==48317== by 0x52534E7: asio::detail::scheduler::do_run_one(asio::detail::scoped_lock<asio::detail::posix_mutex>&, asio::detail::scheduler_thread_info&, std::error_code const&) (scheduler.ipp:375)
==48317== by 0x52530D3: asio::detail::scheduler::run(std::error_code&) (scheduler.ipp:146)
==48317== by 0x5253806: asio::io_context::run() (io_context.ipp:59)
==48317== by 0x526792A: restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}::operator()() const (service.cpp:190)
==48317== by 0x526CA5F: void std::_Bind_simple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2} ()>::_M_invoke<>(std::_Index_tuple<>) (functional:1531)
==48317== by 0x526C9C8: std::_Bind_simple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2} ()>::operator()() (functional:1520)
==48317== by 0x526C967: std::thread::_Impl<std::_Bind_simple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2} ()> >::_M_run() (thread:115)
==48317== by 0x5CC2C8F: execute_native_thread_routine (thread.cc:84)
==48317== by 0x6869DC4: start_thread (in /usr/lib64/libpthread-2.17.so)
==48317== by 0x659728C: clone (in /usr/lib64/libc-2.17.so)
==48317== Address 0x12578058 is 72 bytes inside a block of size 152 free'd
==48317== at 0x4C2B131: operator delete(void*) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==48317== by 0x529E791: __gnu_cxx::new_allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> >::deallocate(std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2>*, unsigned long) (new_allocator.h:110)
==48317== by 0x529E1D8: std::allocator_traits<std::allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> > >::deallocate(std::allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> >&, std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2>*, unsigned long) (alloc_traits.h:517)
==48317== by 0x529D92D: std::__allocated_ptr<std::allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> > >::~__allocated_ptr() (allocated_ptr.h:72)
==48317== by 0x529F620: std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2>::_M_destroy() (shared_ptr_base.h:539)
==48317== by 0x40EDD7: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:167)
==48317== by 0x40E7F8: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:659)
==48317== by 0x5263745: std::__shared_ptr<restbed::detail::SocketImpl, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:925)
==48317== by 0x526377D: std::shared_ptr<restbed::detail::SocketImpl>::~shared_ptr() (shared_ptr.h:93)
==48317== by 0x5264B9B: restbed::detail::RequestImpl::~RequestImpl() (request_impl.hpp:38)
==48317== by 0x5264C73: std::default_delete<restbed::detail::RequestImpl>::operator()(restbed::detail::RequestImpl*) const (unique_ptr.h:76)
==48317== by 0x5263E28: std::unique_ptr<restbed::detail::RequestImpl, std::default_delete<restbed::detail::RequestImpl> >::~unique_ptr() (unique_ptr.h:236)
==48317== by 0x526248C: restbed::Request::~Request() (request.cpp:65)
==48317== by 0x52EF81A: void __gnu_cxx::new_allocator<restbed::Request>::destroy<restbed::Request>(restbed::Request*) (new_allocator.h:124)
==48317== by 0x52EF700: void std::allocator_traits<std::allocator<restbed::Request> >::destroy<restbed::Request>(std::allocator<restbed::Request>&, restbed::Request*) (alloc_traits.h:542)
==48317== by 0x52EF182: std::_Sp_counted_ptr_inplace<restbed::Request, std::allocator<restbed::Request>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr_base.h:531)
==48317== by 0x40EDA1: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:150)
==48317== by 0x40E7F8: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:659)
==48317== by 0x40E417: std::__shared_ptr<restbed::Request const, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:925)
==48317== by 0x40E433: std::shared_ptr<restbed::Request const>::~shared_ptr() (shared_ptr.h:93)
==48317== by 0x52EFB8E: restbed::detail::SessionImpl::~SessionImpl() (session_impl.cpp:80)
==48317== by 0x52EFBCD: restbed::detail::SessionImpl::~SessionImpl() (session_impl.cpp:83)
==48317== by 0x5285DE7: std::default_delete<restbed::detail::SessionImpl>::operator()(restbed::detail::SessionImpl*) const (unique_ptr.h:76)
==48317== by 0x528561C: std::unique_ptr<restbed::detail::SessionImpl, std::default_delete<restbed::detail::SessionImpl> >::~unique_ptr() (unique_ptr.h:236)
==48317== by 0x527F04C: restbed::Session::~Session() (session.cpp:50)
==48317== by 0x52953EC: void __gnu_cxx::new_allocator<restbed::Session>::destroy<restbed::Session>(restbed::Session*) (new_allocator.h:124)
==48317== by 0x52953B8: void std::allocator_traits<std::allocator<restbed::Session> >::destroy<restbed::Session>(std::allocator<restbed::Session>&, restbed::Session*) (alloc_traits.h:542)
==48317== by 0x52952C6: std::_Sp_counted_ptr_inplace<restbed::Session, std::allocator<restbed::Session>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr_base.h:531)
==48317== by 0x40EDA1: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:150)
==48317== by 0x40E7F8: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:659)
==48317== by 0x40F191: std::__shared_ptr<restbed::Session, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:925)
==48317== by 0x40F1AD: std::shared_ptr<restbed::Session>::~shared_ptr() (shared_ptr.h:93)
==48317== by 0x52851D5: restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1}::~error_code() (session.cpp:238)
==48317== by 0x528426F: std::_Function_base::_Base_manager<restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1}>::_M_destroy(std::_Any_data&, std::integral_constant<bool, false>) (functional:1726)
==48317== by 0x5283582: std::_Function_base::_Base_manager<restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1}>::_M_manager(std::_Any_data&, restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1} const&, std::_Manager_operation) (functional:1750)
==48317== by 0x40E3A6: std::_Function_base::~_Function_base() (functional:1830)
==48317== by 0x525431F: std::function<void (std::error_code const&, unsigned long)>::~function() (functional:1974)
==48317== by 0x52A158D: restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}::~function() (socket_impl.cpp:335)
==48317== by 0x52A15E9: asio::detail::wrapped_handler<asio::io_context::strand, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}, asio::detail::is_continuation_if_running>::~wrapped_handler() (wrapped_handler.hpp:48)
==48317== by 0x52A36B3: asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::basic_streambuf_ref<std::allocator<char> >, asio::detail::wrapped_handler<asio::io_context::strand, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}, asio::detail::is_continuation_if_running> >::~read_until_delim_string_op() (read_until.hpp:668)
==48317== by 0x52ADE4B: asio::detail::binder2<asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::basic_streambuf_ref<std::allocator<char> >, asio::detail::wrapped_handler<asio::io_context::strand, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}, asio::detail::is_continuation_if_running> >, std::error_code, unsigned long>::~binder2() (bind_handler.hpp:126)
==48317== by 0x52ADF5B: asio::detail::reactive_socket_recv_op<asio::mutable_buffers_1, asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::basic_streambuf_ref<std::allocator<char> >, asio::detail::wrapped_handler<asio::io_context::strand, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}, asio::detail::is_continuation_if_running> > >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (reactive_socket_recv_op.hpp:107)
==48317== by 0x525282D: asio::detail::scheduler_operation::complete(void*, std::error_code const&, unsigned long) (scheduler_operation.hpp:39)
==48317== by 0x52BEDB8: asio::detail::epoll_reactor::descriptor_state::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (epoll_reactor.ipp:695)
==48317== by 0x525282D: asio::detail::scheduler_operation::complete(void*, std::error_code const&, unsigned long) (scheduler_operation.hpp:39)
==48317== by 0x52534E7: asio::detail::scheduler::do_run_one(asio::detail::scoped_lock<asio::detail::posix_mutex>&, asio::detail::scheduler_thread_info&, std::error_code const&) (scheduler.ipp:375)
==48317== by 0x52530D3: asio::detail::scheduler::run(std::error_code&) (scheduler.ipp:146)
==48317== by 0x5253806: asio::io_context::run() (io_context.ipp:59)
==48317== by 0x5267FD1: restbed::Service::start(std::shared_ptr<restbed::Settings const> const&) (service.cpp:197)
==48317== by 0x40DB00: main (testprogram.cpp:49)
==48317==
OK so now it's the other way around. I've done more debugging and the reason for this crash is this:
I have been able to reproduce case 3B quite a few times (not 100% reliably though due to its async nature) and it resulted in an identical crash every time.
The fix is to bring back your static connection_timeout_handler() and shared_from_this magic. I tried it and it does work fine, no more crashes after substantial abuse.
Note that the strand fix is still needed because the original race scenario is still valid.
I'll submit a PR shortly.
Looks good with the latest change. I haven't been able to reproduce the unhandled exception or coredump.
Yay! :)
Great news. I'm looking into a regression test so as to avoid this being reintroduced during the next refactor.
Hi @ben-crowhurst + others on this thread,
I believe I hit this issue earlier today using restbed 4.5 when returning a c. 9MB file to a web browser. I was running the service single threaded, and it appeared that a second request was coming in from the client while the data resulting from the first request was being transmitted.
(Historically I've run on Linux and not seen this issue; today I was running on Windows. The error code the socket was closed following data transmission was "invalid file descriptor".)
It seems the solution to this issue was to wrap the timeout handler in an asio strand?
My question is why was this only done for the call on lines 290 + 613 and not the calls on lines 415 + 464?
I added the calls into my codebase and it resolved my issue.
Thanks Chris
We’ve run into an issue with restbed coredumping due to an unhandled exception.
This was encountered with an application compiled on AWS Linux with gcc 5.4.0
The following error message is generated to stderr when the unhandled exception is encountered.
This is the stack backtrace from the core file which is not too helpful.
Using gdb, I tracked down where std::system_error is being thrown from asio. See below: