Corvusoft / restbed

Corvusoft's Restbed framework brings asynchronous RESTful functionality to C++14 applications.
http://www.corvusoft.co.uk
Other
1.93k stars 377 forks source link

restbed unhandled exception #125

Closed greenday9 closed 8 years ago

greenday9 commented 8 years ago

We’ve run into an issue with restbed coredumping due to an unhandled exception.

This was encountered with an application compiled on AWS Linux with gcc 5.4.0

$ cat /etc/system-release Amazon Linux AMI release 2016.03

The following error message is generated to stderr when the unhandled exception is encountered.

terminate called after throwing an instance of 'std::system_error' what(): close: Bad file descriptor

This is the stack backtrace from the core file which is not too helpful.

#0  0x00007f52aa5e35f7 in raise () from /lib64/libc.so.6
#1  0x00007f52aa5e4ce8 in abort () from /lib64/libc.so.6
#2  0x00007f52aaf1689d in __gnu_cxx::__verbose_terminate_handler () at ../../.././libstdc++-v3/libsupc++/vterminate.cc:95
#3  0x00007f52aaf14906 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../.././libstdc++-v3/libsupc++/eh_terminate.cc:47
#4  0x00007f52aaf14951 in std::terminate () at ../../.././libstdc++-v3/libsupc++/eh_terminate.cc:57
#5  0x00007f52aaf3e998 in std::execute_native_thread_routine (__p=<optimized out>) at ../../../.././libstdc++-v3/src/c++11/thread.cc:92
#6  0x00007f52aa399dc5 in start_thread () from /lib64/libpthread.so.0
#7  0x00007f52aa6a4c9d in clone () from /lib64/libc.so.6

Using gdb, I tracked down where std::system_error is being thrown from asio. See below:

#0  std::system_error::system_error (this=0x7f527400dce0) at /usr/local/include/c++/5.4.0/system_error:333
#1  0x00007f52ad0282b9 in asio::detail::throw_exception<std::system_error> (e=...)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/throw_exception.hpp:42
#2  0x00007f52ad02546b in asio::detail::do_throw_error (err=..., location=0x7f52ad0bbd3b "close")
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/impl/throw_error.ipp:49
#3  0x00007f52ad0253d1 in asio::detail::throw_error (err=..., location=0x7f52ad0bbd3b "close")
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/throw_error.hpp:41
#4  0x00007f52ad08ad59 in asio::basic_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >::close (this=0x7f5270002d30)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/basic_socket.hpp:353
#5  0x00007f52ad07541a in restbed::detail::SocketImpl::close (this=0x7f5280005080)
    at /home/user/extern/restbed/source/corvusoft/restbed/detail/socket_impl.cpp:91
#6  0x00007f52ad076ece in restbed::detail::SocketImpl::connection_timeout_handler (this=0x7f5280005080, error=...)
    at /home/user/extern/restbed/source/corvusoft/restbed/detail/socket_impl.cpp:444
#7  0x00007f52ad093670 in std::_Mem_fn_base<void (restbed::detail::SocketImpl::*)(std::error_code const&), true>::operator()<std::error_code const&, void> (
    this=0x7f52967fbb10, __object=0x7f5280005080) at /usr/local/include/c++/5.4.0/functional:600
#8  0x00007f52ad093489 in std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>::__call<void, std::error_code const&, 0ul, 1ul>(std::tuple<std::error_code const&>&&, std::_Index_tuple<0ul, 1ul>) (this=0x7f52967fbb10,
    __args=<unknown type in /home/hpe-service/sdk/lib/librestbed.so, CU 0x28db95, DIE 0x2ead8e>) at /usr/local/include/c++/5.4.0/functional:1074
#9  0x00007f52ad0930f1 in std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>::operator()<std::error_code const&, void>(std::error_code const&) (this=0x7f52967fbb10) at /usr/local/include/c++/5.4.0/functional:1133
#10 0x00007f52ad092a8f in asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>::operator()() (this=0x7f52967fbb10)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/bind_handler.hpp:64
#11 0x00007f52ad09234f in asio::asio_handler_invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, ...) (function=...)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/handler_invoke_hook.hpp:68
#12 0x00007f52ad091ccc in asio_handler_invoke_helpers::invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>&) (function=..., context=...)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/handler_invoke_helpers.hpp:37
#13 0x00007f52ad091311 in asio::detail::handler_work<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::system_executor>::complete<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>&) (this=0x7f52967fbb3f, function=..., handler=...)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/handler_work.hpp:81
#14 0x00007f52ad0900bd in asio::detail::wait_handler<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (owner=0xdeb5d0, base=0x7f5288002150)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/wait_handler.hpp:71
#15 0x00007f52ad025d1c in asio::detail::scheduler_operation::complete (this=0x7f5288002150, owner=0xdeb5d0, ec=..., bytes_transferred=0)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/scheduler_operation.hpp:39
#16 0x00007f52ad0269d6 in asio::detail::scheduler::do_run_one (this=0xdeb5d0, lock=..., this_thread=..., ec=...)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/impl/scheduler.ipp:375
#17 0x00007f52ad0265c2 in asio::detail::scheduler::run (this=0xdeb5d0, ec=...)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/impl/scheduler.ipp:146
#18 0x00007f52ad026cf5 in asio::io_context::run (this=0xde9550)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/impl/io_context.ipp:59
#19 0x00007f52ad03e365 in restbed::Service::<lambda()>::operator()(void) const (__closure=0xdcb028)
    at /home/user/extern/restbed/source/corvusoft/restbed/service.cpp:178
#20 0x00007f52ad043260 in std::_Bind_simple<restbed::Service::start(const std::shared_ptr<const restbed::Settings>&)::<lambda()>()>::_M_invoke<>(std::_Index_tuple<>)
    (this=0xdcb028) at /usr/local/include/c++/5.4.0/functional:1531
#21 0x00007f52ad0431c9 in std::_Bind_simple<restbed::Service::start(const std::shared_ptr<const restbed::Settings>&)::<lambda()>()>::operator()(void) (this=0xdcb028)
    at /usr/local/include/c++/5.4.0/functional:1520
#22 0x00007f52ad043168 in std::thread::_Impl<std::_Bind_simple<restbed::Service::start(const std::shared_ptr<const restbed::Settings>&)::<lambda()>()> >::_M_run(void)
    (this=0xdcb010) at /usr/local/include/c++/5.4.0/thread:115
#23 0x00007f52aaf3e8e0 in std::execute_native_thread_routine (__p=<optimized out>) at ../../../.././libstdc++-v3/src/c++11/thread.cc:84
#24 0x00007f52aa399dc5 in start_thread () from /lib64/libpthread.so.0
#25 0x00007f52aa6a4c9d in clone () from /lib64/libc.so.6
greenday9 commented 8 years ago

This issue can be reproduced with the following test program.


void post_method_handler( const shared_ptr< Session > session )
{
    const auto request = session->get_request( );

    size_t content_length = request->get_header( "Content-Length", 0 );

    session->fetch( content_length, [ request ]( const shared_ptr< Session > session, const Bytes & body )
    {
        fprintf( stdout, "%.*s\n", ( int ) body.size( ), body.data( ) );

        session->sleep_for( std::chrono::milliseconds(5), []( const shared_ptr< Session > session )
                           {
                               session->yield( OK, "Hello, World!", { { "Content-Length", "13" }, { "Connection", "close" } } );
                           }
                           );

    } );
}

int main( const int, const char** )
{
    auto resource = make_shared< Resource >( );
    resource->set_path( "/resource" );
    resource->set_method_handler( "POST", post_method_handler );

    auto settings = make_shared< Settings >( );
    settings->set_port( 8010 );
    settings->set_worker_limit( 10 );

    Service service;
    service.publish( resource );
    service.start( settings );

    return EXIT_SUCCESS;
}

Sending a load of requests into the test program causes it to coredump very quickly. Note that sometimes I see other crashes besides an unhandled exception. It may be a multi-threading issue since I don't see it if I set worker_limit to 1.

greenday9 commented 8 years ago

Actually, this does not appear to be related to the number of worker threads. Even with worker_limit set to 1, the same issue occurs. It just takes longer.

ben-crowhurst commented 8 years ago

I'll look at wrapping the connection_timeout_handler in a try-catch block and logging the error when it occurs. This will fix the issue.

Give me a couple of hours to confirm the fix.

ben-crowhurst commented 8 years ago

How are you interacting with the above example? curl, wget, Restbed::Http?

ben-crowhurst commented 8 years ago

From a precursory review of the stack-trace it appears the socket is timing out, calling close and ASIO is throwing an exception on a bad socket.

void SocketImpl::connection_timeout_handler( const error_code& error )
{
    if ( error or m_timer->expires_at( ) > steady_clock::now( ) )
    {
        return;
    }

    close( ); //this method calls m_socket->close.
}

My hunch is the socket is killed by the client, resulting in 'bad descriptor'. The session times out and calls close on an invalid descriptor resulting in the system_error.

greenday9 commented 8 years ago

FYI. I also ran with valgrind and I see the following error before crashing. There are other similar errors that follow.

==45696== Invalid read of size 8
==45696==    at 0x5260514: std::__shared_ptr<asio::basic_waitable_timer<std::chrono::_V2::steady_clock, asio::wait_traits<std::chrono::_V2::steady_clock>, asio::waitable_timer_service<std::chrono::_V2::steady_clock, asio::wait_traits<std::chrono::_V2::steady_clock> > >, (__gnu_cxx::_Lock_policy)2>::operator->() const (shared_ptr_base.h:1055)
==45696==    by 0x5291953: restbed::detail::SocketImpl::connection_timeout_handler(std::error_code const&) (socket_impl.cpp:440)
==45696==    by 0x52AE069: void std::_Mem_fn_base<void (restbed::detail::SocketImpl::*)(std::error_code const&), true>::operator()<std::error_code const&, void>(restbed::detail::SocketImpl*, std::error_code const&) const (in /home/kgreenwell/adev/my-repo/sdk/lib/librestbed.so)
==45696==    by 0x52ADE82: void std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>::__call<void, std::error_code const&, 0ul, 1ul>(std::tuple<std::error_code const&>&&, std::_Index_tuple<0ul, 1ul>) (functional:1074)
==45696==    by 0x52ADAEA: void std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>::operator()<std::error_code const&, void>(std::error_code const&) (functional:1133)
==45696==    by 0x52AD488: asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>::operator()() (bind_handler.hpp:64)
==45696==    by 0x52ACD48: void asio::asio_handler_invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, ...) (handler_invoke_hook.hpp:68)
==45696==    by 0x52AC6C5: void asio_handler_invoke_helpers::invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>&) (handler_invoke_helpers.hpp:37)
==45696==    by 0x52ABD0A: void asio::detail::handler_work<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::system_executor>::complete<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>&) (handler_work.hpp:81)
==45696==    by 0x52AAB48: asio::detail::wait_handler<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (wait_handler.hpp:71)
==45696==    by 0x524294F: asio::detail::scheduler_operation::complete(void*, std::error_code const&, unsigned long) (scheduler_operation.hpp:39)
==45696==    by 0x5243609: asio::detail::scheduler::do_run_one(asio::detail::scoped_lock<asio::detail::posix_mutex>&, asio::detail::scheduler_thread_info&, std::error_code const&) (scheduler.ipp:375)
==45696==    by 0x52431F5: asio::detail::scheduler::run(std::error_code&) (scheduler.ipp:146)
==45696==    by 0x5243928: asio::io_context::run() (io_context.ipp:59)
==45696==    by 0x5257FBD: restbed::Service::start(std::shared_ptr<restbed::Settings const> const&) (service.cpp:197)
==45696==    by 0x40D9F0: main (testprogram.cpp:49)
==45696==  Address 0xd80fbb8 is 72 bytes inside a block of size 136 free'd
==45696==    at 0x4C2B131: operator delete(void*) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==45696==    by 0x528E659: __gnu_cxx::new_allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> >::deallocate(std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2>*, unsigned long) (new_allocator.h:110)
==45696==    by 0x528E0A6: std::allocator_traits<std::allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> > >::deallocate(std::allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> >&, std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2>*, unsigned long) (alloc_traits.h:517)
==45696==    by 0x528D7FB: std::__allocated_ptr<std::allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> > >::~__allocated_ptr() (allocated_ptr.h:72)
==45696==    by 0x528F4E8: std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2>::_M_destroy() (shared_ptr_base.h:539)
==45696==    by 0x40ECC7: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:167)
==45696==    by 0x40E6E8: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:659)
==45696==    by 0x525372F: std::__shared_ptr<restbed::detail::SocketImpl, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:925)
==45696==    by 0x5253767: std::shared_ptr<restbed::detail::SocketImpl>::~shared_ptr() (shared_ptr.h:93)
==45696==    by 0x5254B85: restbed::detail::RequestImpl::~RequestImpl() (request_impl.hpp:38)
==45696==    by 0x5254C5D: std::default_delete<restbed::detail::RequestImpl>::operator()(restbed::detail::RequestImpl*) const (unique_ptr.h:76)
==45696==    by 0x5253E12: std::unique_ptr<restbed::detail::RequestImpl, std::default_delete<restbed::detail::RequestImpl> >::~unique_ptr() (unique_ptr.h:236)
==45696==    by 0x5252504: restbed::Request::~Request() (request.cpp:65)
==45696==    by 0x52CEBE0: void __gnu_cxx::new_allocator<restbed::Request>::destroy<restbed::Request>(restbed::Request*) (new_allocator.h:124)
==45696==    by 0x52CEAC6: void std::allocator_traits<std::allocator<restbed::Request> >::destroy<restbed::Request>(std::allocator<restbed::Request>&, restbed::Request*) (alloc_traits.h:542)
==45696==    by 0x52CE548: std::_Sp_counted_ptr_inplace<restbed::Request, std::allocator<restbed::Request>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr_base.h:531)
==45696==    by 0x40EC91: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:150)
==45696==    by 0x40E6E8: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:659)
==45696==    by 0x40E307: std::__shared_ptr<restbed::Request const, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:925)
==45696==    by 0x40E323: std::shared_ptr<restbed::Request const>::~shared_ptr() (shared_ptr.h:93)
==45696==    by 0x52CEF56: restbed::detail::SessionImpl::~SessionImpl() (session_impl.cpp:80)
==45696==    by 0x52CEF95: restbed::detail::SessionImpl::~SessionImpl() (session_impl.cpp:83)
==45696==    by 0x5275D45: std::default_delete<restbed::detail::SessionImpl>::operator()(restbed::detail::SessionImpl*) const (unique_ptr.h:76)
==45696==    by 0x527557A: std::unique_ptr<restbed::detail::SessionImpl, std::default_delete<restbed::detail::SessionImpl> >::~unique_ptr() (unique_ptr.h:236)
==45696==    by 0x526F038: restbed::Session::~Session() (session.cpp:50)
==45696==    by 0x5285348: void __gnu_cxx::new_allocator<restbed::Session>::destroy<restbed::Session>(restbed::Session*) (new_allocator.h:124)
==45696==    by 0x5285314: void std::allocator_traits<std::allocator<restbed::Session> >::destroy<restbed::Session>(std::allocator<restbed::Session>&, restbed::Session*) (alloc_traits.h:542)
==45696==    by 0x5285222: std::_Sp_counted_ptr_inplace<restbed::Session, std::allocator<restbed::Session>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr_base.h:531)
==45696==    by 0x40EC91: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:150)
==45696==    by 0x40E6E8: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:659)
==45696==    by 0x40F081: std::__shared_ptr<restbed::Session, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:925)
==45696==    by 0x40F09D: std::shared_ptr<restbed::Session>::~shared_ptr() (shared_ptr.h:93)
==45696==    by 0x5275133: restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1}::~error_code() (session.cpp:238)
==45696==    by 0x527425B: std::_Function_base::_Base_manager<restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1}>::_M_destroy(std::_Any_data&, std::integral_constant<bool, false>) (functional:1726)
==45696==    by 0x527356E: std::_Function_base::_Base_manager<restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1}>::_M_manager(std::_Any_data&, restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1} const&, std::_Manager_operation) (functional:1750)
==45696==    by 0x40E296: std::_Function_base::~_Function_base() (functional:1830)
==45696==    by 0x5244441: std::function<void (std::error_code const&, unsigned long)>::~function() (functional:1974)
==45696==    by 0x529101B: restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}::~function() (socket_impl.cpp:334)
==45696==    by 0x52929B5: asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::basic_streambuf_ref<std::allocator<char> >, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}>::~read_until_delim_string_op() (read_until.hpp:668)
==45696==    by 0x5299D4B: asio::detail::binder2<asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::basic_streambuf_ref<std::allocator<char> >, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}>, std::error_code, unsigned long>::~binder2() (bind_handler.hpp:126)
==45696==    by 0x5299E5B: asio::detail::reactive_socket_recv_op<asio::mutable_buffers_1, asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::basic_streambuf_ref<std::allocator<char> >, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}> >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (reactive_socket_recv_op.hpp:107)
==45696==    by 0x524294F: asio::detail::scheduler_operation::complete(void*, std::error_code const&, unsigned long) (scheduler_operation.hpp:39)
==45696==    by 0x52A1562: asio::detail::epoll_reactor::descriptor_state::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (epoll_reactor.ipp:695)
==45696==    by 0x524294F: asio::detail::scheduler_operation::complete(void*, std::error_code const&, unsigned long) (scheduler_operation.hpp:39)
==45696==    by 0x5243609: asio::detail::scheduler::do_run_one(asio::detail::scoped_lock<asio::detail::posix_mutex>&, asio::detail::scheduler_thread_info&, std::error_code const&) (scheduler.ipp:375)
==45696==    by 0x52431F5: asio::detail::scheduler::run(std::error_code&) (scheduler.ipp:146)
==45696==    by 0x5243928: asio::io_context::run() (io_context.ipp:59)
==45696==    by 0x5257FBD: restbed::Service::start(std::shared_ptr<restbed::Settings const> const&) (service.cpp:197)
==45696==    by 0x40D9F0: main (testprogram.cpp:49)
greenday9 commented 8 years ago

In terms of generating load into the example. I use the following simple node script.

E.g. node load_test.js localhost 1000 2

var http = require('http');

var postData = "testing 123";

var requestCount = 0;

if(process.argv.length < 4)
{
  console.log('ERROR: Missing hostname and TPS!');
  process.exit(1);
}

var iterations = 1;
var tps = process.argv[3];
if(process.argv.length > 4)
{
  iterations = process.argv[4];
}

console.log(`connecting to ${process.argv[2]} at ${tps} TPS with ${iterations} iterations`);

var options = {
  host: process.argv[2],
  path: '/resource',
  port: '8010',
  method: 'POST',
  headers: {
   'Content-Type' : 'text/plain',
   'Content-Length' : Buffer.byteLength(postData)
  }
};

callback = function(response) {
  if(response.statusCode != 200)
    console.log(`STATUS: ${response.statusCode}`);
}

setInterval(function() {
  for(var i = 0; i < iterations; i++)
  {
    var req = http.request(options, callback);
    req.write(postData);
    req.end();
    requestCount++;
  }
}, 1000 / tps);

setInterval(function() {
  console.log(`TPS: ${requestCount}`);
  requestCount = 0;
}, 1000);
ben-crowhurst commented 8 years ago

Thanks.

ben-crowhurst commented 8 years ago
  for(var i = 0; i < iterations; i++)
  {
    var req = http.request(options, callback);
    req.write(postData);
    req.end();
    requestCount++;
  }

As stated previously. I believe each request (client) has a restricted scope and is being reclaimed after the end of the for-loop/req.end. This causes the client to close the connection and once the timeout is fired it creates the bad socket operation on the server.

Curious as to why ASIO would throw an error on socket->close and not just report success. Either way poor handling on our end.

callum-kirby commented 8 years ago

I pushed fix for this issue, further validation required.

ben-crowhurst commented 8 years ago

@callum-kirby please provide a regression test.

ben-crowhurst commented 8 years ago

@greenday9 We've managed to replicate this issue and resolve it via @callum-kirby's fix. Please can you confirm on your end.

greenday9 commented 8 years ago

It runs longer now. But I've hit a coredump. I'm running the same example program with the same load injection script.

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007fb5e866d318 in asio::detail::reactive_socket_service_base::close (this=0x28, impl=..., ec=...)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/impl/reactive_socket_service_base.ipp:105
105         reactor_.deregister_descriptor(impl.socket_, impl.reactor_data_,
(gdb) where
#0  0x00007fb5e866d318 in asio::detail::reactive_socket_service_base::close (this=0x28, impl=..., ec=...)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/impl/reactive_socket_service_base.ipp:105
#1  0x00007fb5e867057c in asio::stream_socket_service<asio::ip::tcp>::close (this=0x0, impl=..., ec=...)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/stream_socket_service.hpp:162
#2  0x00007fb5e866f86b in asio::basic_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >::close (this=0x2645730)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/basic_socket.hpp:352
#3  0x00007fb5e8659f0e in restbed::detail::SocketImpl::close (this=0x220ce80)
    at /home/user/extern/restbed/source/corvusoft/restbed/detail/socket_impl.cpp:93
#4  0x00007fb5e865b9a2 in restbed::detail::SocketImpl::connection_timeout_handler (this=0x220ce80, error=...)
    at /home/user/extern/restbed/source/corvusoft/restbed/detail/socket_impl.cpp:449
#5  0x00007fb5e867810c in std::_Mem_fn_base<void (restbed::detail::SocketImpl::*)(std::error_code const&), true>::operator()<std::error_code const&, void> (
    this=0x7ffdd413b490, __object=0x220ce80) at /usr/local/include/c++/5.4.0/functional:600
#6  0x00007fb5e8677f25 in std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>::__call<void, std::error_code const&, 0ul, 1ul>(std::tuple<std::error_code const&>&&, std::_Index_tuple<0ul, 1ul>) (this=0x7ffdd413b490,
    __args=<unknown type in /home/user/sdk/lib/librestbed.so, CU 0x29bac1, DIE 0x2f9106>) at /usr/local/include/c++/5.4.0/functional:1074
#7  0x00007fb5e8677b8d in std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>::operator()<std::error_code const&, void>(std::error_code const&) (this=0x7ffdd413b490) at /usr/local/include/c++/5.4.0/functional:1133
#8  0x00007fb5e867752b in asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>::operator()() (this=0x7ffdd413b490)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/bind_handler.hpp:64
#9  0x00007fb5e8676deb in asio::asio_handler_invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, ...) (function=...)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/handler_invoke_hook.hpp:68
#10 0x00007fb5e8676768 in asio_handler_invoke_helpers::invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>&) (function=..., context=...)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/handler_invoke_helpers.hpp:37
#11 0x00007fb5e8675dad in asio::detail::handler_work<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::system_executor>::complete<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>&) (this=0x7ffdd413b4bf, function=..., handler=...)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/handler_work.hpp:81
#12 0x00007fb5e8674beb in asio::detail::wait_handler<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (owner=0x1292f80, base=0x1671ea0)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/wait_handler.hpp:71
#13 0x00007fb5e860c950 in asio::detail::scheduler_operation::complete (this=0x1671ea0, owner=0x1292f80, ec=..., bytes_transferred=0)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/scheduler_operation.hpp:39
#14 0x00007fb5e860d60a in asio::detail::scheduler::do_run_one (this=0x1292f80, lock=..., this_thread=..., ec=...)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/impl/scheduler.ipp:375
#15 0x00007fb5e860d1f6 in asio::detail::scheduler::run (this=0x1292f80, ec=...)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/detail/impl/scheduler.ipp:146
#16 0x00007fb5e860d929 in asio::io_context::run (this=0x1292f10)
    at /home/user/extern/restbed/dependency/asio/asio/include/asio/impl/io_context.ipp:59
#17 0x00007fb5e8621fbe in restbed::Service::start (this=0x7ffdd413b7f0, settings=...)
    at /home/user/extern/restbed/source/corvusoft/restbed/service.cpp:197
#18 0x000000000040d9f1 in main () at /home/user/testprogram/testprogram.cpp:49

valgrind also reports a memory issue as well during runs. It appears in some cases SocketImpl is deleted (via Session::yield) and then the connect_timeout_handler callback fires on the the deleted SocketImpl.

==41342== Invalid read of size 8
==41342==    at 0x5260514: std::__shared_ptr<asio::basic_waitable_timer<std::chrono::_V2::steady_clock, asio::wait_traits<std::chrono::_V2::steady_clock>, asio::waitable_timer_service<std::chrono::_V2::steady_clock, asio::wait_traits<std::chrono::_V2::steady_clock> > >, (__gnu_cxx::_Lock_policy)2>::operator->() const (shared_ptr_base.h:1055)
==41342==    by 0x5291953: restbed::detail::SocketImpl::connection_timeout_handler(std::error_code const&) (socket_impl.cpp:442)
==41342==    by 0x52AE10B: void std::_Mem_fn_base<void (restbed::detail::SocketImpl::*)(std::error_code const&), true>::operator()<std::error_code const&, void>(restbed::detail::SocketImpl*, std::error_code const&) const (in /home/kgreenwell/adev/my-repo/sdk/lib/librestbed.so)
==41342==    by 0x52ADF24: void std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>::__call<void, std::error_code const&, 0ul, 1ul>(std::tuple<std::error_code const&>&&, std::_Index_tuple<0ul, 1ul>) (functional:1074)
==41342==    by 0x52ADB8C: void std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>::operator()<std::error_code const&, void>(std::error_code const&) (functional:1133)
==41342==    by 0x52AD52A: asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>::operator()() (bind_handler.hpp:64)
==41342==    by 0x52ACDEA: void asio::asio_handler_invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, ...) (handler_invoke_hook.hpp:68)
==41342==    by 0x52AC767: void asio_handler_invoke_helpers::invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>&) (handler_invoke_helpers.hpp:37)
==41342==    by 0x52ABDAC: void asio::detail::handler_work<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::system_executor>::complete<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>&) (handler_work.hpp:81)
==41342==    by 0x52AABEA: asio::detail::wait_handler<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (wait_handler.hpp:71)
==41342==    by 0x524294F: asio::detail::scheduler_operation::complete(void*, std::error_code const&, unsigned long) (scheduler_operation.hpp:39)
==41342==    by 0x5243609: asio::detail::scheduler::do_run_one(asio::detail::scoped_lock<asio::detail::posix_mutex>&, asio::detail::scheduler_thread_info&, std::error_code const&) (scheduler.ipp:375)
==41342==    by 0x52431F5: asio::detail::scheduler::run(std::error_code&) (scheduler.ipp:146)
==41342==    by 0x5243928: asio::io_context::run() (io_context.ipp:59)
==41342==    by 0x5257FBD: restbed::Service::start(std::shared_ptr<restbed::Settings const> const&) (service.cpp:197)
==41342==    by 0x40D9F0: main (testprogram.cpp:49)
==41342==  Address 0xe7bf808 is 72 bytes inside a block of size 136 free'd
==41342==    at 0x4C2B131: operator delete(void*) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==41342==    by 0x528E659: __gnu_cxx::new_allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> >::deallocate(std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2>*, unsigned long) (new_allocator.h:110)
==41342==    by 0x528E0A6: std::allocator_traits<std::allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> > >::deallocate(std::allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> >&, std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2>*, unsigned long) (alloc_traits.h:517)
==41342==    by 0x528D7FB: std::__allocated_ptr<std::allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> > >::~__allocated_ptr() (allocated_ptr.h:72)
==41342==    by 0x528F4E8: std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2>::_M_destroy() (shared_ptr_base.h:539)
==41342==    by 0x40ECC7: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:167)
==41342==    by 0x40E6E8: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:659)
==41342==    by 0x525372F: std::__shared_ptr<restbed::detail::SocketImpl, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:925)
==41342==    by 0x5253767: std::shared_ptr<restbed::detail::SocketImpl>::~shared_ptr() (shared_ptr.h:93)
==41342==    by 0x5254B85: restbed::detail::RequestImpl::~RequestImpl() (request_impl.hpp:38)
==41342==    by 0x5254C5D: std::default_delete<restbed::detail::RequestImpl>::operator()(restbed::detail::RequestImpl*) const (unique_ptr.h:76)
==41342==    by 0x5253E12: std::unique_ptr<restbed::detail::RequestImpl, std::default_delete<restbed::detail::RequestImpl> >::~unique_ptr() (unique_ptr.h:236)
==41342==    by 0x5252504: restbed::Request::~Request() (request.cpp:65)
==41342==    by 0x52CEC84: void __gnu_cxx::new_allocator<restbed::Request>::destroy<restbed::Request>(restbed::Request*) (new_allocator.h:124)
==41342==    by 0x52CEB6A: void std::allocator_traits<std::allocator<restbed::Request> >::destroy<restbed::Request>(std::allocator<restbed::Request>&, restbed::Request*) (alloc_traits.h:542)
==41342==    by 0x52CE5EC: std::_Sp_counted_ptr_inplace<restbed::Request, std::allocator<restbed::Request>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr_base.h:531)
==41342==    by 0x40EC91: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:150)
==41342==    by 0x40E6E8: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:659)
==41342==    by 0x40E307: std::__shared_ptr<restbed::Request const, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:925)
==41342==    by 0x40E323: std::shared_ptr<restbed::Request const>::~shared_ptr() (shared_ptr.h:93)
==41342==    by 0x52CEFFA: restbed::detail::SessionImpl::~SessionImpl() (session_impl.cpp:80)
==41342==    by 0x52CF039: restbed::detail::SessionImpl::~SessionImpl() (session_impl.cpp:83)
==41342==    by 0x5275D45: std::default_delete<restbed::detail::SessionImpl>::operator()(restbed::detail::SessionImpl*) const (unique_ptr.h:76)
==41342==    by 0x527557A: std::unique_ptr<restbed::detail::SessionImpl, std::default_delete<restbed::detail::SessionImpl> >::~unique_ptr() (unique_ptr.h:236)
==41342==    by 0x526F038: restbed::Session::~Session() (session.cpp:50)
==41342==    by 0x5285348: void __gnu_cxx::new_allocator<restbed::Session>::destroy<restbed::Session>(restbed::Session*) (new_allocator.h:124)
==41342==    by 0x5285314: void std::allocator_traits<std::allocator<restbed::Session> >::destroy<restbed::Session>(std::allocator<restbed::Session>&, restbed::Session*) (alloc_traits.h:542)
==41342==    by 0x5285222: std::_Sp_counted_ptr_inplace<restbed::Session, std::allocator<restbed::Session>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr_base.h:531)
==41342==    by 0x40EC91: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:150)
==41342==    by 0x40E6E8: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:659)
==41342==    by 0x40F081: std::__shared_ptr<restbed::Session, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:925)
==41342==    by 0x40F09D: std::shared_ptr<restbed::Session>::~shared_ptr() (shared_ptr.h:93)
==41342==    by 0x5275133: restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1}::~error_code() (session.cpp:238)
==41342==    by 0x527425B: std::_Function_base::_Base_manager<restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1}>::_M_destroy(std::_Any_data&, std::integral_constant<bool, false>) (functional:1726)
==41342==    by 0x527356E: std::_Function_base::_Base_manager<restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1}>::_M_manager(std::_Any_data&, restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1} const&, std::_Manager_operation) (functional:1750)
==41342==    by 0x40E296: std::_Function_base::~_Function_base() (functional:1830)
==41342==    by 0x5244441: std::function<void (std::error_code const&, unsigned long)>::~function() (functional:1974)
==41342==    by 0x529101B: restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}::~function() (socket_impl.cpp:336)
==41342==    by 0x5292A57: asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::basic_streambuf_ref<std::allocator<char> >, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}>::~read_until_delim_string_op() (read_until.hpp:668)
==41342==    by 0x5299DED: asio::detail::binder2<asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::basic_streambuf_ref<std::allocator<char> >, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}>, std::error_code, unsigned long>::~binder2() (bind_handler.hpp:126)
==41342==    by 0x5299EFD: asio::detail::reactive_socket_recv_op<asio::mutable_buffers_1, asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::basic_streambuf_ref<std::allocator<char> >, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}> >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (reactive_socket_recv_op.hpp:107)
==41342==    by 0x524294F: asio::detail::scheduler_operation::complete(void*, std::error_code const&, unsigned long) (scheduler_operation.hpp:39)
==41342==    by 0x52A1604: asio::detail::epoll_reactor::descriptor_state::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (epoll_reactor.ipp:695)
==41342==    by 0x524294F: asio::detail::scheduler_operation::complete(void*, std::error_code const&, unsigned long) (scheduler_operation.hpp:39)
==41342==    by 0x5243609: asio::detail::scheduler::do_run_one(asio::detail::scoped_lock<asio::detail::posix_mutex>&, asio::detail::scheduler_thread_info&, std::error_code const&) (scheduler.ipp:375)
==41342==    by 0x52431F5: asio::detail::scheduler::run(std::error_code&) (scheduler.ipp:146)
==41342==    by 0x5243928: asio::io_context::run() (io_context.ipp:59)
==41342==    by 0x5257FBD: restbed::Service::start(std::shared_ptr<restbed::Settings const> const&) (service.cpp:197)
==41342==    by 0x40D9F0: main (testprogram.cpp:49)
ben-crowhurst commented 8 years ago

Attempting to replicate.

A few notes:

a) Please make sure you do indeed close the session on your end.


session->sleep_for( std::chrono::milliseconds(5), []( const shared_ptr< Session > session )
{
    //session->yield( OK, "Hello, World!", { { "Content-Length", "13" }, { "Connection", "close" } } );
    session->close( OK, "Hello, World!", { { "Content-Length", "13" }, { "Connection", "close" } } );
} );

b) Do you in fact have 10 cores available where the machine is running?

Here is the suggested approach to setting the worker limit.

settings->set_worker_limit( thread::hardware_concurrency( ) );
callum-kirby commented 8 years ago

Yep confirmed on Ubuntu 10x threads on 4 core system; see what I can do.

greenday9 commented 8 years ago

@ben-crowhurst

a) How does close and yield differ? My assumption is that yield() leaves the connection open which is what we want in the case the client sends follow on requests. It is artificial in the test program (particularly since we return the "Connection: close" header).

b) No, we don't have 10 cores. This should be optimized for the number of cores/etc. However, the test program is simply replicating the issue. Thanks for the suggestion on using thread::hardware_concurrency().

ben-crowhurst commented 8 years ago

@greenday9 You are correct. The yield with 'Connection: close' caught my eye and decided to mention it.

Attempting to resolve timer issue now.

ben-crowhurst commented 8 years ago

Ok moving the SocketImpl::connection_timeout_handler to a static method resolves this issue.

Still attempting to locate how this situation arises in the first place.

callum-kirby commented 8 years ago

@greenday9 Not seeing a service crash anymore, please rerun your stress suite.

ignitenet-martynas commented 8 years ago

@ben-crowhurst The situation arises as follows, given presence of worker threads:

  1. A Keep-Alive connection times out and SocketImpl::connection_timeout_handler() is called
  2. connection_timeout_handler() calls close()
  3. A completion handler for asio::async_write() or asio::async_read() in one of the SocketImpl::write() or SocketImpl::read() implementations gets called from a different thread with error_code == "Operation aborted", the write()/read() returns, then one level above the shared_ptr goes out of scope and subsequently SocketImpl is destroyed and its tcp::socket gets closed.
  4. SocketImpl::close() in the original thread proceeds to call m_socket->close() but it's already closed, therefore Asio thrown the "close: bad file descriptor" exception.

I've confirmed the above scenario through painful testing with lots of debug messages with thread IDs :)

Just catching and silencing the exception that you pushed is definitely not a proper fix for this. This also explains why making connection_timeout_handler() static "fixed" the issue -- it was no longer operating on an already-destroyed object.

I tried fixing it by deriving SocketImpl from std::enable_shared_from_this and then in connection_timeout_handler() doing the following:

auto socket = shared_from_this(); socket->close();

That took care of the socket not getting destroyed (because the shared_ptr had its refcount upped) asynchronously.

However, that is still not a proper fix because a certain race condition remains: what if there are multiple worker threads and data becomes available for asio::async_read() at the very same precise moment that connection_timeout_handler() is called? The result will be that connection_timeout_handler() will close the socket that the asio::async_read() completion handler has already begun getting data from which is a nasty race condition.

Therefore the proper fix is to never allow async_read() / async_write() completion handlers run concurrently with connection_timeout_handler() by using an asio::io_service::strand.

I'll do a PR momentarily with that fix implemented.

ignitenet-martynas commented 8 years ago

My apologies – I am still quite new to Git. Will do a proper PR shortly.

greenday9 commented 8 years ago

It is definitely much more stable, but there is still an issue. If I kill the load injection tool while the test program is heavily loaded, I see the unhandled exception or coredump. This is easy to reproduce especially when the program is running with valgrind. See the following valgrind trace:

==48317== Thread 7:
==48317== Invalid read of size 8
==48317==    at 0x5270528: std::__shared_ptr<asio::basic_waitable_timer<std::chrono::_V2::steady_clock, asio::wait_traits<std::chrono::_V2::steady_clock>, asio::waitable_timer_service<std::chrono::_V2::steady_clock, asio::wait_traits<std::chrono::_V2::steady_clock> > >, (__gnu_cxx::_Lock_policy)2>::operator->() const (shared_ptr_base.h:1055)
==48317==    by 0x52A2005: restbed::detail::SocketImpl::connection_timeout_handler(std::error_code const&) (socket_impl.cpp:441)
==48317==    by 0x52CDE8B: void std::_Mem_fn_base<void (restbed::detail::SocketImpl::*)(std::error_code const&), true>::operator()<std::error_code const&, void>(restbed::detail::SocketImpl*, std::error_code const&) const (in /home/user/sdk/lib/librestbed.so)
==48317==    by 0x52CDA68: void std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>::__call<void, std::error_code const&, 0ul, 1ul>(std::tuple<std::error_code const&>&&, std::_Index_tuple<0ul, 1ul>) (functional:1074)
==48317==    by 0x52CD440: void std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>::operator()<std::error_code const&, void>(std::error_code const&) (functional:1133)
==48317==    by 0x52CCB9A: asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>::operator()() (bind_handler.hpp:64)
==48317==    by 0x52CC3B2: void asio::asio_handler_invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, ...) (handler_invoke_hook.hpp:68)
==48317==    by 0x52CBB4D: void asio_handler_invoke_helpers::invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>&) (handler_invoke_helpers.hpp:37)
==48317==    by 0x52CEA46: void asio::detail::asio_handler_invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>*) (bind_handler.hpp:105)
==48317==    by 0x52CE810: void asio_handler_invoke_helpers::invoke<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>, asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&, asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&) (handler_invoke_helpers.hpp:37)
==48317==    by 0x52CE61E: void asio::detail::strand_service::dispatch<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::strand_service::strand_impl*&, asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&) (strand_service.hpp:61)
==48317==    by 0x52CE4E2: asio::async_result<asio::handler_type<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>, void (), void>::type>::type asio::io_context::strand::dispatch<asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code> >(asio::detail::binder1<std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, std::error_code>&&) (io_context_strand.hpp:221)
==48317==    by 0x52CE46B: void asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>::operator()<std::error_code>(std::error_code const&) (wrapped_handler.hpp:86)
==48317==    by 0x52CE42A: asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>::operator()() (bind_handler.hpp:64)
==48317==    by 0x52CE3D5: asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >::operator()() (wrapped_handler.hpp:190)
==48317==    by 0x52CE2F9: void asio::asio_handler_invoke<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> > >(asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >&, ...) (handler_invoke_hook.hpp:68)
==48317==    by 0x52CE00E: void asio_handler_invoke_helpers::invoke<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >(asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >&, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>&) (handler_invoke_helpers.hpp:37)
==48317==    by 0x52CDBC6: void asio::detail::asio_handler_invoke<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >, asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >(asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >&, asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >*) (wrapped_handler.hpp:274)
==48317==    by 0x52CD738: void asio_handler_invoke_helpers::invoke<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >, asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> > >(asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >&, asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >&) (handler_invoke_helpers.hpp:37)
==48317==    by 0x52CDDDC: void asio::detail::handler_work<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >, asio::system_executor>::complete<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> > >(asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >&, asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >&) (handler_work.hpp:81)
==48317==    by 0x52CD8FF: asio::detail::completion_handler<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> > >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (completion_handler.hpp:69)
==48317==    by 0x52CD34F: void asio::detail::strand_service::dispatch<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> > >(asio::detail::strand_service::strand_impl*&, asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >&) (strand_service.hpp:87)
==48317==    by 0x52CCAEA: asio::async_result<asio::handler_type<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >, void (), void>::type>::type asio::io_context::strand::dispatch<asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> > >(asio::detail::rewrapped_handler<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)> >&&) (io_context_strand.hpp:221)
==48317==    by 0x52CC287: void asio::detail::asio_handler_invoke<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>(asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>&, asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>*) (wrapped_handler.hpp:231)
==48317==    by 0x52CBA4E: void asio_handler_invoke_helpers::invoke<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>, asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running> >(asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>&, asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>&) (handler_invoke_helpers.hpp:37)
==48317==    by 0x52CAE08: void asio::detail::handler_work<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, asio::system_executor>::complete<asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code> >(asio::detail::binder1<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>, std::error_code>&, asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running>&) (handler_work.hpp:81)
==48317==    by 0x52C9761: asio::detail::wait_handler<asio::detail::wrapped_handler<asio::io_context::strand, std::_Bind<std::_Mem_fn<void (restbed::detail::SocketImpl::*)(std::error_code const&)> (restbed::detail::SocketImpl*, std::_Placeholder<1>)>, asio::detail::is_continuation_if_running> >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (wait_handler.hpp:71)
==48317==    by 0x525282D: asio::detail::scheduler_operation::complete(void*, std::error_code const&, unsigned long) (scheduler_operation.hpp:39)
==48317==    by 0x52534E7: asio::detail::scheduler::do_run_one(asio::detail::scoped_lock<asio::detail::posix_mutex>&, asio::detail::scheduler_thread_info&, std::error_code const&) (scheduler.ipp:375)
==48317==    by 0x52530D3: asio::detail::scheduler::run(std::error_code&) (scheduler.ipp:146)
==48317==    by 0x5253806: asio::io_context::run() (io_context.ipp:59)
==48317==    by 0x526792A: restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2}::operator()() const (service.cpp:190)
==48317==    by 0x526CA5F: void std::_Bind_simple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2} ()>::_M_invoke<>(std::_Index_tuple<>) (functional:1531)
==48317==    by 0x526C9C8: std::_Bind_simple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2} ()>::operator()() (functional:1520)
==48317==    by 0x526C967: std::thread::_Impl<std::_Bind_simple<restbed::Service::start(std::shared_ptr<restbed::Settings const> const&)::{lambda()#2} ()> >::_M_run() (thread:115)
==48317==    by 0x5CC2C8F: execute_native_thread_routine (thread.cc:84)
==48317==    by 0x6869DC4: start_thread (in /usr/lib64/libpthread-2.17.so)
==48317==    by 0x659728C: clone (in /usr/lib64/libc-2.17.so)
==48317==  Address 0x12578058 is 72 bytes inside a block of size 152 free'd
==48317==    at 0x4C2B131: operator delete(void*) (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==48317==    by 0x529E791: __gnu_cxx::new_allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> >::deallocate(std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2>*, unsigned long) (new_allocator.h:110)
==48317==    by 0x529E1D8: std::allocator_traits<std::allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> > >::deallocate(std::allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> >&, std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2>*, unsigned long) (alloc_traits.h:517)
==48317==    by 0x529D92D: std::__allocated_ptr<std::allocator<std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2> > >::~__allocated_ptr() (allocated_ptr.h:72)
==48317==    by 0x529F620: std::_Sp_counted_ptr_inplace<restbed::detail::SocketImpl, std::allocator<restbed::detail::SocketImpl>, (__gnu_cxx::_Lock_policy)2>::_M_destroy() (shared_ptr_base.h:539)
==48317==    by 0x40EDD7: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:167)
==48317==    by 0x40E7F8: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:659)
==48317==    by 0x5263745: std::__shared_ptr<restbed::detail::SocketImpl, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:925)
==48317==    by 0x526377D: std::shared_ptr<restbed::detail::SocketImpl>::~shared_ptr() (shared_ptr.h:93)
==48317==    by 0x5264B9B: restbed::detail::RequestImpl::~RequestImpl() (request_impl.hpp:38)
==48317==    by 0x5264C73: std::default_delete<restbed::detail::RequestImpl>::operator()(restbed::detail::RequestImpl*) const (unique_ptr.h:76)
==48317==    by 0x5263E28: std::unique_ptr<restbed::detail::RequestImpl, std::default_delete<restbed::detail::RequestImpl> >::~unique_ptr() (unique_ptr.h:236)
==48317==    by 0x526248C: restbed::Request::~Request() (request.cpp:65)
==48317==    by 0x52EF81A: void __gnu_cxx::new_allocator<restbed::Request>::destroy<restbed::Request>(restbed::Request*) (new_allocator.h:124)
==48317==    by 0x52EF700: void std::allocator_traits<std::allocator<restbed::Request> >::destroy<restbed::Request>(std::allocator<restbed::Request>&, restbed::Request*) (alloc_traits.h:542)
==48317==    by 0x52EF182: std::_Sp_counted_ptr_inplace<restbed::Request, std::allocator<restbed::Request>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr_base.h:531)
==48317==    by 0x40EDA1: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:150)
==48317==    by 0x40E7F8: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:659)
==48317==    by 0x40E417: std::__shared_ptr<restbed::Request const, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:925)
==48317==    by 0x40E433: std::shared_ptr<restbed::Request const>::~shared_ptr() (shared_ptr.h:93)
==48317==    by 0x52EFB8E: restbed::detail::SessionImpl::~SessionImpl() (session_impl.cpp:80)
==48317==    by 0x52EFBCD: restbed::detail::SessionImpl::~SessionImpl() (session_impl.cpp:83)
==48317==    by 0x5285DE7: std::default_delete<restbed::detail::SessionImpl>::operator()(restbed::detail::SessionImpl*) const (unique_ptr.h:76)
==48317==    by 0x528561C: std::unique_ptr<restbed::detail::SessionImpl, std::default_delete<restbed::detail::SessionImpl> >::~unique_ptr() (unique_ptr.h:236)
==48317==    by 0x527F04C: restbed::Session::~Session() (session.cpp:50)
==48317==    by 0x52953EC: void __gnu_cxx::new_allocator<restbed::Session>::destroy<restbed::Session>(restbed::Session*) (new_allocator.h:124)
==48317==    by 0x52953B8: void std::allocator_traits<std::allocator<restbed::Session> >::destroy<restbed::Session>(std::allocator<restbed::Session>&, restbed::Session*) (alloc_traits.h:542)
==48317==    by 0x52952C6: std::_Sp_counted_ptr_inplace<restbed::Session, std::allocator<restbed::Session>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr_base.h:531)
==48317==    by 0x40EDA1: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:150)
==48317==    by 0x40E7F8: std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() (shared_ptr_base.h:659)
==48317==    by 0x40F191: std::__shared_ptr<restbed::Session, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() (shared_ptr_base.h:925)
==48317==    by 0x40F1AD: std::shared_ptr<restbed::Session>::~shared_ptr() (shared_ptr.h:93)
==48317==    by 0x52851D5: restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1}::~error_code() (session.cpp:238)
==48317==    by 0x528426F: std::_Function_base::_Base_manager<restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1}>::_M_destroy(std::_Any_data&, std::integral_constant<bool, false>) (functional:1726)
==48317==    by 0x5283582: std::_Function_base::_Base_manager<restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1}>::_M_manager(std::_Any_data&, restbed::Session::yield(restbed::Response const&, std::function<void (std::shared_ptr<restbed::Session>)> const&)::{lambda(std::error_code const&, unsigned long)#1}::operator()(std::error_code const&, unsigned long) const::{lambda(std::error_code const&, unsigned long)#1} const&, std::_Manager_operation) (functional:1750)
==48317==    by 0x40E3A6: std::_Function_base::~_Function_base() (functional:1830)
==48317==    by 0x525431F: std::function<void (std::error_code const&, unsigned long)>::~function() (functional:1974)
==48317==    by 0x52A158D: restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}::~function() (socket_impl.cpp:335)
==48317==    by 0x52A15E9: asio::detail::wrapped_handler<asio::io_context::strand, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}, asio::detail::is_continuation_if_running>::~wrapped_handler() (wrapped_handler.hpp:48)
==48317==    by 0x52A36B3: asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::basic_streambuf_ref<std::allocator<char> >, asio::detail::wrapped_handler<asio::io_context::strand, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}, asio::detail::is_continuation_if_running> >::~read_until_delim_string_op() (read_until.hpp:668)
==48317==    by 0x52ADE4B: asio::detail::binder2<asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::basic_streambuf_ref<std::allocator<char> >, asio::detail::wrapped_handler<asio::io_context::strand, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}, asio::detail::is_continuation_if_running> >, std::error_code, unsigned long>::~binder2() (bind_handler.hpp:126)
==48317==    by 0x52ADF5B: asio::detail::reactive_socket_recv_op<asio::mutable_buffers_1, asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp, asio::stream_socket_service<asio::ip::tcp> >, asio::basic_streambuf_ref<std::allocator<char> >, asio::detail::wrapped_handler<asio::io_context::strand, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::string const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}, asio::detail::is_continuation_if_running> > >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (reactive_socket_recv_op.hpp:107)
==48317==    by 0x525282D: asio::detail::scheduler_operation::complete(void*, std::error_code const&, unsigned long) (scheduler_operation.hpp:39)
==48317==    by 0x52BEDB8: asio::detail::epoll_reactor::descriptor_state::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) (epoll_reactor.ipp:695)
==48317==    by 0x525282D: asio::detail::scheduler_operation::complete(void*, std::error_code const&, unsigned long) (scheduler_operation.hpp:39)
==48317==    by 0x52534E7: asio::detail::scheduler::do_run_one(asio::detail::scoped_lock<asio::detail::posix_mutex>&, asio::detail::scheduler_thread_info&, std::error_code const&) (scheduler.ipp:375)
==48317==    by 0x52530D3: asio::detail::scheduler::run(std::error_code&) (scheduler.ipp:146)
==48317==    by 0x5253806: asio::io_context::run() (io_context.ipp:59)
==48317==    by 0x5267FD1: restbed::Service::start(std::shared_ptr<restbed::Settings const> const&) (service.cpp:197)
==48317==    by 0x40DB00: main (testprogram.cpp:49)
==48317== 
ignitenet-martynas commented 8 years ago

OK so now it's the other way around. I've done more debugging and the reason for this crash is this:

  1. Client closes the connection abruptly
  2. read_async() completion handler is called with error set to End of file, calls m_timer->cancel(), proceeds to return, SocketImpl shared_pointer goes out of scope, SocketImpl object is destroyed
  3. (3A) Under normal circumstances m_timer->cancel() returns 1 (meaning the timer has been canceled) and connection_timeout_handler() is invoked later with error set to Operation aborted and returns early, w/o accessing anything through this. That is perfectly fine, even on an already-destroyed object.
  4. (3B) Under rare extraordinary circumstances m_timer->cancel() returns 0 (meaning the timer has either expired (not our case) or its handler(s) have been queued for invocation in the near future) and after the async_read() completion handler returns and the SocketImpl object is destroyed, connection_timeout_handler() is still invoked w/ the error parameter not set. It then proceeds to access m_timer and crashes.

I have been able to reproduce case 3B quite a few times (not 100% reliably though due to its async nature) and it resulted in an identical crash every time.

The fix is to bring back your static connection_timeout_handler() and shared_from_this magic. I tried it and it does work fine, no more crashes after substantial abuse.

Note that the strand fix is still needed because the original race scenario is still valid.

I'll submit a PR shortly.

greenday9 commented 8 years ago

Looks good with the latest change. I haven't been able to reproduce the unhandled exception or coredump.

ignitenet-martynas commented 8 years ago

Yay! :)

ben-crowhurst commented 8 years ago

Great news. I'm looking into a regression test so as to avoid this being reintroduced during the next refactor.

clrusby commented 5 years ago

Hi @ben-crowhurst + others on this thread,

I believe I hit this issue earlier today using restbed 4.5 when returning a c. 9MB file to a web browser. I was running the service single threaded, and it appeared that a second request was coming in from the client while the data resulting from the first request was being transmitted.

(Historically I've run on Linux and not seen this issue; today I was running on Windows. The error code the socket was closed following data transmission was "invalid file descriptor".)

It seems the solution to this issue was to wrap the timeout handler in an asio strand?

My question is why was this only done for the call on lines 290 + 613 and not the calls on lines 415 + 464?

I added the calls into my codebase and it resolved my issue.

Thanks Chris