Crash after hitting timedOut in SentMessage

[jterrace]

Happened soon after typing "cont" from gdb--I was in the debugger long enough to cause a timeout. The timedOut function calls the callback function of the parent SentMessage, which has been deleted by that time.

[jterrace]

Author: PatrickHorn I suspect the problem is that I broke in the debugger between when a message was sent and when the response was received. When I hit cont, I suspect the order of actions was:

• Received response message, put on IOService queue
• Timeout occurred, put on IOService queue.
• Queue is processed:
• SentMessage deleted. and its timer is Cancelled, pushed on the IOService queue.
• Timeout is processed but not treated like a cancellation.
• Cancellation processed

The solution is probably to keep a boolean in SentMessage::TimerHandler whether it was cancelled instead of looking at the boost::asio::error_code.

Have only encountered this bug once. Only likely to happen when messages are sent over the network.

[jterrace]

Author: PatrickHorn Should be fixed in b7409b8. GIve it a try.

[jterrace]

Author: PatrickHorn and why does trac keep closing bugs when they aren't marked as fixed yet

[jterrace]

Author: danielrh I get a throw() in /home/daniel/sirikata/libcore/src/util/SentMessage.cpp:63 {{{ 48 void SentMessage::timedOut() { 49 RoutableMessageHeader msg; 50 if (this->header().has_destination_object()) { 51 msg.set_source_object(this->header().destination_object()); 52 } 53 if (this->header().has_destination_space()) { 54 msg.set_source_space(this->header().destination_space()); 55 } 56 msg.set_source_port(this->header().destination_port()); 57 msg.set_return_status(RoutableMessageHeader::TIMEOUT_FAILURE); 58 msg.set_reply_id(this->getId()); 59 this->mTimerHandle.reset(); 60 61 // Make sure this is on the same thread-- 62 // We can use mQueryTracker->sendMessage(msg) to send it via the same loop 63 this->mResponseCallback(this, msg, MemoryReference(NULL,0));//dies here 64 } }}}

9 0xf7f3c6da in Sirikata::SentMessage::timedOut (this=0x94400b0)

at /home/daniel/sirikata/libcore/src/util/SentMessage.cpp:63

63 this->mResponseCallback(this, msg, MemoryReference(NULL,0));

{{{

7 0xf7ecc509 in boost::throw_exceptionboost::bad_function_call (

e=@0xffdc9c98)
at /home/daniel/sirikata/dependencies/installed-boost/include/boost-1_37/boost/throw_exception.hpp:64

8 0xf7f3e020 in boost::function3<void, Sirikata::SentMessage, Sirikata::RoutableMessageHeader const&, Sirikata::DataReference<void const> >::operator() (this=0x9440120, a0=0x94400b0, a1=@0xffdc9ce4, a2=

 {first = 0x0, second = 0})
at /home/daniel/sirikata/dependencies/installed-boost/include/boost-1_37/boost/function/function_template.hpp:986

9 0xf7f3c6da in Sirikata::SentMessage::timedOut (this=0x94400b0)

at /home/daniel/sirikata/libcore/src/util/SentMessage.cpp:63

10 0xf7f3da2e in boost::_mfi::mf0<void, Sirikata::SentMessage>::operator()

(this=0x9402e70, p=0x94400b0)
at /home/daniel/sirikata/dependencies/installed-boost/include/boost-1_37/boost/bind/mem_fn_template.hpp:49

11 0xf7f3dc0c in boost::_bi::list1boost::_bi::value<Sirikata::SentMessage* >::operator()<boost::_mfi::mf0<void, Sirikata::SentMessage>, boost::_bi::list0> (this=0x9402e78, f=@0x9402e70, a=@0xffdc9db7)

at /home/daniel/sirikata/dependencies/installed-boost/include/boost-1_37/boost/bind.hpp:232

12 0xf7f3dc4f in boost::_bi::bind_t<void, boost::_mfi::mf0<void, Sirikata::SentMessage>, boost::_bi::list1boost::_bi::value<Sirikata::SentMessage* > >::operator() (this=0x9402e70)

at /home/daniel/sirikata/dependencies/installed-boost/include/boost-1_37/boost/bind/bind_template.hpp:20

13 0xf7f3dc69 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf0<void, Sirikata::SentMessage>, boost::_bi::list1boost::_bi::value<Sirikata::SentMessage* > >, void>::invoke (

function_obj_ptr=@0x9402e70)
at /home/daniel/sirikata/dependencies/installed-boost/include/boost-1_37/boost/function/function_template.hpp:152

14 0xf7f1356f in boost::function0::operator() (this=0x9402e6c)

at /home/daniel/sirikata/dependencies/installed-boost/include/boost-1_37/boost/function/function_template.hpp:989

15 0xf7f14a10 in Sirikata::Network::TimerHandle::TimedOut::timedOut (

error=@0xffdc9f28, wthis=@0xffdc9e84)
at /home/daniel/sirikata/libcore/src/network/IOServiceFactory.cpp:117

16 0xf7f0faa6 in boost::bi::list2boost::arg<1 ()(), boost::_bi::valueboost::sharedptr >::operator()<void ()(boost::system::error_code const&, boost::weak_ptrSirikata::Network::TimerHandle), boost::_bi::list1boost::system::error_code& > (this=0xffdc9f20,

f=@0xffdc9f1c, a=@0xffdc9ec4)
at /home/daniel/sirikata/dependencies/installed-boost/include/boost-1_37/boost/bind.hpp:292

17 0xf7f0fb1e in boost::_bi::bind_t<void, void (*)(boost::system::error_code const&, boost::weak_ptr), boost::bi::list2boost::arg<1 ()(), boost::_bi::valueboost::shared_ptr > >::operator()boost::system::error_code (

this=0xffdc9f1c, a1=@0xffdc9f28)
at /home/daniel/sirikata/dependencies/installed-boost/include/boost-1_37/boost/bind/bind_template.hpp:32

18 0xf7f0fb3b in boost::asio::detail::binder1<boost::_bi::bindt<void, void ()(boost::system::error_code const&, boost::weak_ptrSirikata::Network::TimerHandle), boost::bi::list2boost::arg<1 ()(), boost::_bi::valueboost::shared_ptr > >, boost::system::error_code>::operator() (this=0xffdc9f1c)

at /home/daniel/sirikata/dependencies/installed-boost/include/boost-1_37/boost/asio/detail/bind_handler.hpp:39

19 0xf7f0fb4f in boost::asio::asio_handler_invoke<boost::asio::detail::binder1<boost::_bi::bindt<void, void ()(boost::system::error_code const&, boost::weak_ptrSirikata::Network::TimerHandle), boost::bi::list2boost::arg<1 ()(), boost::_bi::valueboost::shared_ptr > >, boost::system::error_code> > (function=@0xffdc9f1c)

at /home/daniel/sirikata/dependencies/installed-boost/include/boost-1_37/boost/asio/handler_invoke_hook.hpp:63

20 0xf7f0fb89 in boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder1<boost::_bi::bindt<void, void ()(boost::system::error_code const&, boost::weak_ptrSirikata::Network::TimerHandle), boost::bi::list2boost::arg<1 ()(), boost::_bi::valueboost::shared_ptr > >, boost::system::error_code>, boost::_bi::bindt<void, void ()(boost::system::error_code const&, boost::weak_ptrSirikata::Network::TimerHandle), boost::bi::list2boost::arg<1 ()(), boost::_bi::valueboost::shared_ptr > > > (function=@0xffdc9f80,

context=0xffdc9f80)
at /home/daniel/sirikata/dependencies/installed-boost/include/boost-1_37/boost/asio/detail/handler_invoke_helpers.hpp:39

21 0xf7f0fbd4 in boost::asio::detail::asio_handler_invoke<boost::asio::detail::binder1<boost::_bi::bindt<void, void ()(boost::system::error_code const&, boost::weak_ptrSirikata::Network::TimerHandle), boost::bi::list2boost::arg<1 ()(), boost::_bi::valueboost::shared_ptr > >, boost::system::error_code>, boost::_bi::bindt<void, void ()(boost::system::error_code const&, boost::weak_ptrSirikata::Network::TimerHandle), boost::bi::list2boost::arg<1 ()(), boost::_bi::valueboost::shared_ptr > >, boost::system::error_code>

(function=@0xffdc9f80, this_handler=0xffdc9f80)
at /home/daniel/sirikata/dependencies/installed-boost/include/boost-1_37/boost/asio/detail/bind_handler.hpp:72

22 0xf7f0fbee in boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder1<boost::_bi::bindt<void, void ()(boost::system::error_code const&, boost::weak_ptrSirikata::Network::TimerHandle), boost::bi::list2boost::arg<1 ()(), boost::_bi::valueboost::shared_ptr > >, boost::system::error_code>, boost::asio::detail::binder1<boost::_bi::bindt<void, void ()(boost::system::error_code const&, boost::weak_ptrSirikata::Network::TimerHandle), boost::bi::list2boost::arg<1 ()(), boost::_bi::valueboost::shared_ptr > >, boost::system::error_code> > (function=@0xffdc9f80, context=0xffdc9f80)

at /home/daniel/sirikata/dependencies/installed-boost/include/boost-1_37/boost/asio/detail/handler_invoke_helpers.hpp:39

23 0xf7f1111f in boost::asio::detail::handler_queue::handler_wrapper<boost::asio::detail::binder1<boost::_bi::bindt<void, void ()(boost::system::error_code const&, boost::weak_ptrSirikata::Network::TimerHandle), boost::_bi::list2boost::arg<1 (*)(), boost::_bi::valueboost::shared_ptr > >, boost::system::error_code> >::do_call (

base=0x92950f0)
at /home/daniel/sirikata/dependencies/installed-boost/include/boost-1_37/boost/asio/detail/handler_queue.hpp:192

24 0xf7f0a216 in boost::asio::detail::handler_queue::handler::invoke (

this=0x92950f0)
at /home/daniel/sirikata/dependencies/installed-boost/include/boost-1_37/boost/asio/detail/handler_queue.hpp:39

25 0xf7f1c78a in boost::asio::detail::task_io_serviceboost::asio::detail::epoll_reactor::do_one (this=0x8211720, lock=@0xffdca05c,

this_idle_thread=0x0, ec=@0xffdca08c)
at /home/daniel/sirikata/dependencies/installed-boost/include/boost-1_37/boost/asio/detail/task_io_service.hpp:268

26 0xf7f1c9e7 in boost::asio::detail::task_io_serviceboost::asio::detail::epoll_reactor::poll (this=0x8211720, ec=@0xffdca08c)

at /home/daniel/sirikata/dependencies/installed-boost/include/boost-1_37/boost/asio/detail/task_io_service.hpp:130

27 0xf7f1ca7c in boost::asio::io_service::poll (this=0x82078a8)

at /home/daniel/sirikata/dependencies/installed-boost/include/boost-1_37/boost/asio/impl/io_service.ipp:84

28 0xf7f09cab in Sirikata::Network::IOServiceFactory::pollService (

ios=0x82078a8)
at /home/daniel/sirikata/libcore/src/network/IOServiceFactory.cpp:63

}}}

[jterrace]

Author: danielrh Fixed in c6096a23589c5bfc379d1a406c4ce5ab8309fc6b and bf728677436533e0d2b46cda2adbc3098c7aba2a Additional asserts in 6f81d687ffaf4e2798e9b120cc933b1617ee9b82 and 096ef251972a8d932201bac132d6c89e150e51de to verify usage is single threaded and proper