search

XRPLF / rippled

Decentralized cryptocurrency blockchain daemon implementing the XRP Ledger protocol in C++
https://xrpl.org
ISC License
4.59k stars 1.48k forks source link

Server stalled for 500 seconds (I/O starvation or deadlock?) #112

Closed alexdupre closed 9 years ago

alexdupre commented 11 years ago

The server is a dedicated machine with a fast SATA drive. DB files have mtime set to 22:07, when the first "server stalled" message appeared (

Last log lines:

2013-Jul-06 22:07:08 LedgerConsensus:NFO Have the consensus ledger D761B1D961B7CB9EE07EEB07B4AC20D3550BB153E3B7BAD8F3D7A5580E08D297 2013-Jul-06 22:07:08 LedgerConsensus:NFO Proposers:2 nw:95 thrV:2 thrC:1 2013-Jul-06 22:07:08 LedgerConsensus:DBG CCTime: seq1231426: 426463530 has 2, 2 required 2013-Jul-06 22:07:08 LedgerConsensus:DBG Close time consensus reached: 426463530 2013-Jul-06 22:07:08 LedgerConsensus:DBG AC716E14017457CDB5D987562F3B1FD01554F7B7 has A65975FF01C7590E2FE9225CBFD063A5161C50AE9A8A9114DCC01EDBDEBDAC07 2013-Jul-06 22:07:08 LedgerConsensus:DBG 9FD43FCC5B131E96266059B3D067C696A03168AB has C02FF871B4A183FD70D1971386C57419B28147934DCF7AC540FC09FA0F5730E0 2013-Jul-06 22:07:08 LedgerConsensus:DBG Checking for TX consensus: agree=0, disagree=2 2013-Jul-06 22:07:08 Transactor:NFO applyTransaction: future sequence number 2013-Jul-06 22:07:08 TransactionEngine:NFO applyTransaction: terResult=terPRE_SEQ : -92 : Missing/inapplicable prior transaction. 2013-Jul-06 22:07:08 TransactionEngine:DBG Not applying transaction 2013-Jul-06 22:07:08 NetworkOPs:DBG Transaction should be held: -92 2013-Jul-06 22:07:08 InboundLedger:WRN Timeout(3) pc=1 acquiring A65975FF01C7590E2FE9225CBFD063A5161C50AE9A8A9114DCC01EDBDEBDAC07 2013-Jul-06 22:07:08 InboundLedger:WRN Timeout(4) pc=1 acquiring A65975FF01C7590E2FE9225CBFD063A5161C50AE9A8A9114DCC01EDBDEBDAC07 2013-Jul-06 22:07:08 InboundLedger:WRN Timeout(5) pc=1 acquiring A65975FF01C7590E2FE9225CBFD063A5161C50AE9A8A9114DCC01EDBDEBDAC07 2013-Jul-06 22:07:09 InboundLedger:WRN Timeout(7) pc=1 acquiring D761B1D961B7CB9EE07EEB07B4AC20D3550BB153E3B7BAD8F3D7A5580E08D297 2013-Jul-06 22:07:09 InboundLedger:WRN Timeout(10) pc=1 acquiring 37099AFE09155F5807FC265527E61195DD66EACC6B3FE333BA97504CD51367BD 2013-Jul-06 22:07:09 InboundLedger:DBG No progress(1) for ledger 1D761B1D961B7CB9EE07EEB07B4AC20D3550BB153E3B7BAD8F3D7A5580E08D297 2013-Jul-06 22:07:09 InboundLedger:DBG No progress(1) for ledger 137099AFE09155F5807FC265527E61195DD66EACC6B3FE333BA97504CD51367BD 2013-Jul-06 22:07:09 InboundLedger:WRN Want: D761B1D961B7CB9EE07EEB07B4AC20D3550BB153E3B7BAD8F3D7A5580E08D297 2013-Jul-06 22:07:09 InboundLedger:WRN Want: 37099AFE09155F5807FC265527E61195DD66EACC6B3FE333BA97504CD51367BD 2013-Jul-06 22:07:09 InboundLedger:NFO Attempting by hash fetch for ledger D761B1D961B7CB9EE07EEB07B4AC20D3550BB153E3B7BAD8F3D7A5580E08D297 2013-Jul-06 22:07:09 InboundLedger:NFO Attempting by hash fetch for ledger 37099AFE09155F5807FC265527E61195DD66EACC6B3FE333BA97504CD51367BD 2013-Jul-06 22:07:18 LoadManager:WRN Server stalled for 10 seconds. 2013-Jul-06 22:07:23 LoadManager:NFO { "job_types" : [

            {
                    "in_progress" : 1,
                    "job_type" : "ledgerData"
            },

            {
                    "avg_time" : 9,
                    "in_progress" : 1,
                    "job_type" : "transaction",
                    "over_target" : true,
                    "peak_time" : 1099,
                    "per_second" : 1
            },

            {
                    "avg_time" : 2,
                    "job_type" : "peerCommand",
                    "peak_time" : 1058,
                    "per_second" : 1
            },

            {
                    "avg_time" : 8,
                    "job_type" : "processTransaction",
                    "peak_time" : 1098,
                    "per_second" : 1
            }
    ],
    "threads" : 6

}

2013-Jul-06 22:07:24 LoadManager:NFO { "job_types" : [

            {
                    "in_progress" : 1,
                    "job_type" : "ledgerData"
            },

            {
                    "avg_time" : 9,
                    "in_progress" : 1,
                    "job_type" : "transaction",
                    "over_target" : true,
                    "peak_time" : 1099
             },

            {
                    "avg_time" : 2,
                    "job_type" : "peerCommand",
                    "peak_time" : 1058
            },

            {
                    "avg_time" : 8,
                    "job_type" : "processTransaction",
                    "peak_time" : 1098
            }
    ],
    "threads" : 6

}

2013-Jul-06 22:07:24 LoadManager:DBG Local load fee raised from 256 to 272 2013-Jul-06 22:07:25 LoadManager:NFO { "job_types" : [

            {
                    "in_progress" : 1,
                    "job_type" : "ledgerData"
            },

            {
                    "avg_time" : 10,
                    "in_progress" : 1,
                    "job_type" : "transaction",
                    "over_target" : true,
                    "peak_time" : 1237
            },

            {
                    "avg_time" : 3,
                    "job_type" : "peerCommand",
                    "peak_time" : 1191
            },

            {
                    "avg_time" : 9,
                    "job_type" : "processTransaction",
                     "peak_time" : 1235
            }
    ],
    "threads" : 6

}

2013-Jul-06 22:07:25 LoadManager:DBG Local load fee raised from 272 to 289 2013-Jul-06 22:07:26 LoadManager:NFO { "job_types" : [

            {
                    "in_progress" : 1,
                    "job_type" : "ledgerData"
            },

            {
                    "avg_time" : 15,
                    "in_progress" : 1,
                    "job_type" : "transaction",
                    "over_target" : true,
                    "peak_time" : 1855
            },

            {
                    "avg_time" : 4,
                    "job_type" : "peerCommand",
                    "over_target" : true,
                    "peak_time" : 1787
            },

            {
                    "avg_time" : 15,
                    "job_type" : "processTransaction",
                    "peak_time" : 1853
            }
    ],
    "threads" : 6

}

2013-Jul-06 22:07:26 LoadManager:DBG Local load fee raised from 289 to 307 2013-Jul-06 22:07:27 LoadManager:DBG Local load fee lowered from 307 to 256 2013-Jul-06 22:07:28 LoadManager:WRN Server stalled for 20 seconds. ... 2013-Jul-06 22:15:28 LoadManager:WRN Server stalled for 500 seconds.

Backtraces:

Thread 16 (Thread 8050041c0 (LWP 100334)):

0 0x0000000801f8a64a in ?? () from /lib/libthr.so.3

1 0x0000000801f8a33c in ?? () from /lib/libthr.so.3

2 0x0000000801f8548c in ?? () from /lib/libthr.so.3

3 0x0000000000e3ab90 in boost::recursive_mutex::lock (this=0x80507e058) at /usr/local/include/boost/thread/pthread/recursive_mutex.hpp:105

4 0x0000000000e4626e in boost::unique_lockboost::recursive_mutex::lock (this=0x7fffffff86f0) at /usr/local/include/boost/thread/locks.hpp:508

5 0x0000000000e4dd59 in boost::unique_lockboost::recursive_mutex::uniquelock (this=0x7fffffff86f0, m=...) at /usr/local/include/boost/thread/locks.hpp:301

6 0x000000000125f523 in ripple::NetworkOPs::checkState (this=0x80507e478, result=...) at ./src/cpp/ripple/NetworkOPs.cpp:614

7 0x00000000012ae55b in boost::_mfi::mf1<void, ripple::NetworkOPs, boost::system::error_code const&>::operator() (this=0x7fffffff8d20, p=0x80507e478, a1=...)

at /usr/local/include/boost/bind/mem_fn_template.hpp:165

8 0x00000000012ad87d in boost::_bi::list2boost::_bi::value<ripple::NetworkOPs, boost::arg<1> ()()>::operator()<boost::_mfi::mf1<void, ripple::NetworkOPs, boost::system::error_code const&>, boost::_bi::list1<boost::system::error_code const&> > (this=0x7fffffff8d30, f=..., a=...) at /usr/local/include/boost/bind/bind.hpp:313

9 0x00000000012ac44a in boost::_bi::bind_t<void, boost::_mfi::mf1<void, ripple::NetworkOPs, boost::system::error_code const&>, boost::_bi::list2boost::_bi::value<ripple::NetworkOPs, boost::arg<1> ()()> >::operator()boost::system::error_code (this=0x7fffffff8d20, a1=...) at /usr/local/include/boost/bind/bind_template.hpp:47

10 0x00000000012ab3b9 in boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, ripple::NetworkOPs, boost::system::error_code const&>, boost::_bi::list2boost::_bi::value<ripple::NetworkOPs, boost::arg<1> ()()> >, boost::system::error_code>::operator() (this=0x7fffffff8d20) at /usr/local/include/boost/asio/detail/bind_handler.hpp:46

11 0x00000000012a9789 in boost::asio::asio_handler_invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, ripple::NetworkOPs, boost::system::error_code const&>, boost::_bi::list2boost::_bi::value<ripple::NetworkOPs, boost::arg<1> ()()> >, boost::system::error_code> > (function=...) at /usr/local/include/boost/asio/handler_invoke_hook.hpp:64

12 0x00000000012a5ca2 in boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder1<boost::_bi::bind_t<void, boost::_mfi::mf1<void, ripple::NetworkOPs, boost::system::error_code const&>, boost::_bi::list2boost::bi::value<ripple::NetworkOPs*, boost::arg<1> ()()> >, boost::system::error_code>, boost::_bi::bind_t<void, boost::_mfi::mf1<void, ripple::NetworkOPs, boost::system::error_code const&>, boost::_bi::list2<boost::bi::value<ripple::NetworkOPs>, boost::arg<1> (*)()> > > (function=..., context=...)

at /usr/local/include/boost/asio/detail/handler_invoke_helpers.hpp:39

13 0x00000000012a07ed in boost::asio::detail::wait_handler<boost::_bi::bind_t<void, boost::_mfi::mf1<void, ripple::NetworkOPs, boost::system::error_code const&>, boost::_bi::list2boost::_bi::value<ripple::NetworkOPs, boost::arg<1> ()()> > >::do_complete (owner=0x805023c80, base=0x8057fc7c0) at /usr/local/include/boost/asio/detail/wait_handler.hpp:69

14 0x00000000010cb4b4 in boost::asio::detail::task_io_service_operation::complete (this=0x8057fc7c0, owner=..., ec=..., bytes_transferred=0)

at /usr/local/include/boost/asio/detail/task_io_service_operation.hpp:37

15 0x00000000010cdedb in boost::asio::detail::task_io_service::do_run_one (this=0x805023c80, lock=..., this_thread=..., ec=...)

at /usr/local/include/boost/asio/detail/impl/task_io_service.ipp:419

16 0x00000000010cd88c in boost::asio::detail::task_io_service::run (this=0x805023c80, ec=...) at /usr/local/include/boost/asio/detail/impl/task_io_service.ipp:160

17 0x00000000010ce231 in boost::asio::io_service::run (this=0x80507e028) at /usr/local/include/boost/asio/impl/io_service.ipp:59

18 0x00000000011d119c in ripple::Application::run (this=0x80507e000) at ./src/cpp/ripple/ripple_Application.cpp:689

19 0x000000000102df13 in ripple::startServer () at ./src/cpp/ripple/ripple_Main.cpp:42

20 0x000000000102f294 in ripple::rippleMain (argc=3, argv=0x7fffffff9c70) at ./src/cpp/ripple/ripple_Main.cpp:295

21 0x00000000012d1d66 in main (argc=3, argv=0x7fffffff9c70) at modules/ripple_app/ripple_app.cpp:425

Thread 15 (Thread 805091540 (LWP 100119)):

0 0x0000000801f8a64c in ?? () from /lib/libthr.so.3

1 0x0000000801f886e1 in ?? () from /lib/libthr.so.3

2 0x0000000000dd811d in beast::WaitableEvent::wait (this=0x80508b368, timeOutMillisecs=1000) at Subtrees/beast/modules/beast_core/native/beast_posix_SharedCode.h:107

3 0x0000000000dc0876 in beast::Thread::wait (this=0x80508b328, timeOutMilliseconds=1000) at Subtrees/beast/modules/beast_core/threads/beast_Thread.cpp:231

4 0x0000000000dfe045 in beast::InterruptibleThread::wait (this=0x80508b328, milliSeconds=1000) at Subtrees/beast/modules/beast_basics/threads/beast_InterruptibleThread.cpp:102

5 0x0000000000e03195 in beast::OncePerSecond::TimerSingleton::threadRun (this=0x80508b300) at Subtrees/beast/modules/beast_basics/events/beast_OncePerSecond.cpp:44

6 0x0000000000dfe36d in beast::InterruptibleThread::run (this=0x80508b328) at Subtrees/beast/modules/beast_basics/threads/beast_InterruptibleThread.cpp:209

7 0x0000000000dfdda8 in beast::InterruptibleThread::ThreadHelper::run (this=0x80508b328) at Subtrees/beast/modules/beast_basics/threads/beast_InterruptibleThread.cpp:34

8 0x0000000000dc03f5 in beast::Thread::threadEntryPoint (this=0x80508b328) at Subtrees/beast/modules/beast_core/threads/beast_Thread.cpp:89

9 0x0000000000dc045e in beast::beast_threadEntryPoint (userData=0x80508b328) at Subtrees/beast/modules/beast_core/threads/beast_Thread.cpp:101

10 0x0000000000dd9da3 in beast::threadEntryProc (userData=0x80508b328) at Subtrees/beast/modules/beast_core/native/beast_posix_SharedCode.h:829

11 0x0000000801f805e1 in ?? () from /lib/libthr.so.3

12 0x0000000000000000 in ?? ()

Thread 14 (Thread 805091380 (LWP 100281)):

0 0x0000000801f8a64c in ?? () from /lib/libthr.so.3

1 0x0000000801f886e1 in ?? () from /lib/libthr.so.3

2 0x0000000000dd8050 in beast::WaitableEvent::wait (this=0x80504fdc8, timeOutMillisecs=-1) at Subtrees/beast/modules/beast_core/native/beast_posix_SharedCode.h:86

3 0x0000000000dc0876 in beast::Thread::wait (this=0x80504fd88, timeOutMilliseconds=-1) at Subtrees/beast/modules/beast_core/threads/beast_Thread.cpp:231

4 0x0000000000dfe045 in beast::InterruptibleThread::wait (this=0x80504fd88, milliSeconds=-1) at Subtrees/beast/modules/beast_basics/threads/beast_InterruptibleThread.cpp:102

5 0x0000000000e024cf in beast::ThreadWithCallQueue::threadRun (this=0x80504fd30) at Subtrees/beast/modules/beast_basics/threads/beast_ThreadWithCallQueue.cpp:148

6 0x0000000000dfe36d in beast::InterruptibleThread::run (this=0x80504fd88) at Subtrees/beast/modules/beast_basics/threads/beast_InterruptibleThread.cpp:209

7 0x0000000000dfdda8 in beast::InterruptibleThread::ThreadHelper::run (this=0x80504fd88) at Subtrees/beast/modules/beast_basics/threads/beast_InterruptibleThread.cpp:34

8 0x0000000000dc03f5 in beast::Thread::threadEntryPoint (this=0x80504fd88) at Subtrees/beast/modules/beast_core/threads/beast_Thread.cpp:89

9 0x0000000000dc045e in beast::beast_threadEntryPoint (userData=0x80504fd88) at Subtrees/beast/modules/beast_core/threads/beast_Thread.cpp:101

10 0x0000000000dd9da3 in beast::threadEntryProc (userData=0x80504fd88) at Subtrees/beast/modules/beast_core/native/beast_posix_SharedCode.h:829

11 0x0000000801f805e1 in ?? () from /lib/libthr.so.3

12 0x0000000000000000 in ?? ()

Thread 13 (Thread 8050911c0 (LWP 100292)):

0 0x0000000801f8a64c in ?? () from /lib/libthr.so.3

1 0x0000000801f886e1 in ?? () from /lib/libthr.so.3

2 0x0000000000eb9a41 in boost::condition_variable::wait (this=0x80507ea50, m=...) at /usr/local/include/boost/thread/pthread/condition_variable.hpp:62

3 0x0000000000eb68c3 in ripple::JobQueue::threadEntry (this=0x80507ea48) at modules/ripple_core/functional/ripple_JobQueue.cpp:246

4 0x0000000000ec605a in std::tr1::_Mem_fn<void (ripple::JobQueue::*)()>::operator() (this=0x8050aa6d0, __object=0x80507ea48) at /usr/local/lib/gcc46/include/c++/tr1/functional:572

5 0x0000000000ec5f4e in std::tr1::_Bind<std::tr1::_Mem_fn<void (ripple::JobQueue::)()> (ripple::JobQueue)>::__call<, 0>(std::tr1::tuple<> const&, std::tr1::_Index_tuple<0>) (

this=0x8050aa6d0, __args=...) at /usr/local/lib/gcc46/include/c++/tr1/functional:1153

6 0x0000000000ec5c9e in std::tr1::_Bind<std::tr1::_Mem_fn<void (ripple::JobQueue::)()> (ripple::JobQueue)>::operator()<>() (this=0x8050aa6d0)

at /usr/local/lib/gcc46/include/c++/tr1/functional:1207

7 0x0000000000ec588c in boost::detail::thread_data<std::tr1::_Bind<std::tr1::_Mem_fn<void (ripple::JobQueue::)()> (ripple::JobQueue)> >::run() (this=0x8050aa600)

at /usr/local/include/boost/thread/detail/thread.hpp:78

8 0x000000080285fac3 in thread_proxy () from /usr/local/lib/libboost_thread.so.5

9 0x0000000801f805e1 in ?? () from /lib/libthr.so.3

10 0x0000000000000000 in ?? ()

Thread 12 (Thread 80500ae40 (LWP 100354)):

0 0x0000000801f8a64a in ?? () from /lib/libthr.so.3

1 0x0000000801f8a33c in ?? () from /lib/libthr.so.3

2 0x0000000801f8548c in ?? () from /lib/libthr.so.3

3 0x0000000000e3ab90 in boost::recursive_mutex::lock (this=0x80507e058) at /usr/local/include/boost/thread/pthread/recursive_mutex.hpp:105

4 0x0000000000e4626e in boost::unique_lockboost::recursive_mutex::lock (this=0x7fffff5fb460) at /usr/local/include/boost/thread/locks.hpp:508

5 0x0000000000e4dd59 in boost::unique_lockboost::recursive_mutex::uniquelock (this=0x7fffff5fb460, m=...) at /usr/local/include/boost/thread/locks.hpp:301

6 0x000000000127c166 in ripple::TransactionAcquire::onTimer (this=0x89b7f4f20, progress=false) at ./src/cpp/ripple/ripple_TransactionAcquire.cpp:53

7 0x00000000012c76b3 in ripple::PeerSet::invokeOnTimer (this=0x89b7f4f20) at ./src/cpp/ripple/ripple_PeerSet.cpp:56

8 0x00000000012c7a22 in ripple::PeerSet::TimerJobEntry (ptr=...) at ./src/cpp/ripple/ripple_PeerSet.cpp:91

9 0x00000000012eda70 in std::tr1::_Bind<void (*(std::tr1::_Placeholder<1>, boost::shared_ptrripple::PeerSet))(ripple::Job&, boost::shared_ptrripple::PeerSet)>::call<ripple::Job&, 0, 1>(std::tr1::tupleripple::Job& const&, std::tr1::_Index_tuple<0, 1>) (this=0x87c2324e0, args=...) at /usr/local/lib/gcc46/include/c++/tr1/functional:1153

10 0x00000000012e8434 in std::tr1::_Bind<void (*(std::tr1::_Placeholder<1>, boost::shared_ptrripple::PeerSet))(ripple::Job&, boost::shared_ptrripple::PeerSet)>::operator()ripple::Job(ripple::Job&) (this=0x87c2324e0) at /usr/local/lib/gcc46/include/c++/tr1/functional:1207

11 0x00000000012e0ede in std::tr1::_Function_handler<void (ripple::Job&), std::tr1::_Bind<void (*(std::tr1::_Placeholder<1>, boost::shared_ptrripple::PeerSet))(ripple::Job&, boost::shared_ptrripple::PeerSet)> >::_M_invoke(std::tr1::_Any_data const&, ripple::Job&) (functor=..., args#0=...) at /usr/local/lib/gcc46/include/c++/tr1/functional:1684

12 0x0000000000ebca34 in std::tr1::function<void (ripple::Job&)>::operator()(ripple::Job&) const (this=0x7fffff5fbce0, __args#0=...)

at /usr/local/lib/gcc46/include/c++/tr1/functional:2040

13 0x0000000000eb4f0e in ripple::Job::doJob (this=0x7fffff5fbcd0) at modules/ripple_core/functional/ripple_Job.cpp:41

14 0x0000000000eb6aa2 in ripple::JobQueue::threadEntry (this=0x80507ea48) at modules/ripple_core/functional/ripple_JobQueue.cpp:268

15 0x0000000000ec605a in std::tr1::_Mem_fn<void (ripple::JobQueue::*)()>::operator() (this=0x8050aa5d0, __object=0x80507ea48) at /usr/local/lib/gcc46/include/c++/tr1/functional:572

16 0x0000000000ec5f4e in std::tr1::_Bind<std::tr1::_Mem_fn<void (ripple::JobQueue::)()> (ripple::JobQueue)>::__call<, 0>(std::tr1::tuple<> const&, std::tr1::_Index_tuple<0>) (

this=0x8050aa5d0, __args=...) at /usr/local/lib/gcc46/include/c++/tr1/functional:1153

17 0x0000000000ec5c9e in std::tr1::_Bind<std::tr1::_Mem_fn<void (ripple::JobQueue::)()> (ripple::JobQueue)>::operator()<>() (this=0x8050aa5d0)

at /usr/local/lib/gcc46/include/c++/tr1/functional:1207

18 0x0000000000ec588c in boost::detail::thread_data<std::tr1::_Bind<std::tr1::_Mem_fn<void (ripple::JobQueue::)()> (ripple::JobQueue)> >::run() (this=0x8050aa500)

at /usr/local/include/boost/thread/detail/thread.hpp:78

19 0x000000080285fac3 in thread_proxy () from /usr/local/lib/libboost_thread.so.5

20 0x0000000801f805e1 in ?? () from /lib/libthr.so.3

21 0x0000000000000000 in ?? ()

Thread 11 (Thread 80500ac80 (LWP 100398)):

0 0x0000000801f8a64c in ?? () from /lib/libthr.so.3

1 0x0000000801f886e1 in ?? () from /lib/libthr.so.3

2 0x0000000000eb9a41 in boost::condition_variable::wait (this=0x80507ea50, m=...) at /usr/local/include/boost/thread/pthread/condition_variable.hpp:62

3 0x0000000000eb68c3 in ripple::JobQueue::threadEntry (this=0x80507ea48) at modules/ripple_core/functional/ripple_JobQueue.cpp:246

4 0x0000000000ec605a in std::tr1::_Mem_fn<void (ripple::JobQueue::*)()>::operator() (this=0x8050aa4d0, __object=0x80507ea48) at /usr/local/lib/gcc46/include/c++/tr1/functional:572

5 0x0000000000ec5f4e in std::tr1::_Bind<std::tr1::_Mem_fn<void (ripple::JobQueue::)()> (ripple::JobQueue)>::__call<, 0>(std::tr1::tuple<> const&, std::tr1::_Index_tuple<0>) (

this=0x8050aa4d0, __args=...) at /usr/local/lib/gcc46/include/c++/tr1/functional:1153

6 0x0000000000ec5c9e in std::tr1::_Bind<std::tr1::_Mem_fn<void (ripple::JobQueue::)()> (ripple::JobQueue)>::operator()<>() (this=0x8050aa4d0)

at /usr/local/lib/gcc46/include/c++/tr1/functional:1207

7 0x0000000000ec588c in boost::detail::thread_data<std::tr1::_Bind<std::tr1::_Mem_fn<void (ripple::JobQueue::)()> (ripple::JobQueue)> >::run() (this=0x8050aa400)

at /usr/local/include/boost/thread/detail/thread.hpp:78

8 0x000000080285fac3 in thread_proxy () from /usr/local/lib/libboost_thread.so.5

9 0x0000000801f805e1 in ?? () from /lib/libthr.so.3

10 0x0000000000000000 in ?? ()

Thread 10 (Thread 80500aac0 (LWP 100455)):

0 0x0000000801f8a64c in ?? () from /lib/libthr.so.3

1 0x0000000801f886e1 in ?? () from /lib/libthr.so.3

2 0x0000000000eb9a41 in boost::condition_variable::wait (this=0x80507ea50, m=...) at /usr/local/include/boost/thread/pthread/condition_variable.hpp:62

3 0x0000000000eb68c3 in ripple::JobQueue::threadEntry (this=0x80507ea48) at modules/ripple_core/functional/ripple_JobQueue.cpp:246

4 0x0000000000ec605a in std::tr1::_Mem_fn<void (ripple::JobQueue::*)()>::operator() (this=0x8050aa3d0, __object=0x80507ea48) at /usr/local/lib/gcc46/include/c++/tr1/functional:572

5 0x0000000000ec5f4e in std::tr1::_Bind<std::tr1::_Mem_fn<void (ripple::JobQueue::)()> (ripple::JobQueue)>::__call<, 0>(std::tr1::tuple<> const&, std::tr1::_Index_tuple<0>) (

this=0x8050aa3d0, __args=...) at /usr/local/lib/gcc46/include/c++/tr1/functional:1153

6 0x0000000000ec5c9e in std::tr1::_Bind<std::tr1::_Mem_fn<void (ripple::JobQueue::)()> (ripple::JobQueue)>::operator()<>() (this=0x8050aa3d0)

at /usr/local/lib/gcc46/include/c++/tr1/functional:1207

7 0x0000000000ec588c in boost::detail::thread_data<std::tr1::_Bind<std::tr1::_Mem_fn<void (ripple::JobQueue::)()> (ripple::JobQueue)> >::run() (this=0x8050aa300)

at /usr/local/include/boost/thread/detail/thread.hpp:78

8 0x000000080285fac3 in thread_proxy () from /usr/local/lib/libboost_thread.so.5

9 0x0000000801f805e1 in ?? () from /lib/libthr.so.3

10 0x0000000000000000 in ?? ()

Thread 9 (Thread 80500a900 (LWP 100457)):

0 0x0000000801f8a64c in ?? () from /lib/libthr.so.3

1 0x0000000801f886e1 in ?? () from /lib/libthr.so.3

2 0x0000000000eb9a41 in boost::condition_variable::wait (this=0x80507ea50, m=...) at /usr/local/include/boost/thread/pthread/condition_variable.hpp:62

3 0x0000000000eb68c3 in ripple::JobQueue::threadEntry (this=0x80507ea48) at modules/ripple_core/functional/ripple_JobQueue.cpp:246

4 0x0000000000ec605a in std::tr1::_Mem_fn<void (ripple::JobQueue::*)()>::operator() (this=0x8050aa2d0, __object=0x80507ea48) at /usr/local/lib/gcc46/include/c++/tr1/functional:572

5 0x0000000000ec5f4e in std::tr1::_Bind<std::tr1::_Mem_fn<void (ripple::JobQueue::)()> (ripple::JobQueue)>::__call<, 0>(std::tr1::tuple<> const&, std::tr1::_Index_tuple<0>) (

this=0x8050aa2d0, __args=...) at /usr/local/lib/gcc46/include/c++/tr1/functional:1153

6 0x0000000000ec5c9e in std::tr1::_Bind<std::tr1::_Mem_fn<void (ripple::JobQueue::)()> (ripple::JobQueue)>::operator()<>() (this=0x8050aa2d0)

at /usr/local/lib/gcc46/include/c++/tr1/functional:1207

7 0x0000000000ec588c in boost::detail::thread_data<std::tr1::_Bind<std::tr1::_Mem_fn<void (ripple::JobQueue::)()> (ripple::JobQueue)> >::run() (this=0x8050aa200)

at /usr/local/include/boost/thread/detail/thread.hpp:78

8 0x000000080285fac3 in thread_proxy () from /usr/local/lib/libboost_thread.so.5

9 0x0000000801f805e1 in ?? () from /lib/libthr.so.3

10 0x0000000000000000 in ?? ()

Thread 8 (Thread 80500a740 (LWP 100471)):

0 0x0000000801f8a64a in ?? () from /lib/libthr.so.3

1 0x0000000801f8a33c in ?? () from /lib/libthr.so.3

2 0x0000000801f8548c in ?? () from /lib/libthr.so.3

3 0x0000000000e3ab90 in boost::recursive_mutex::lock (this=0x80507e058) at /usr/local/include/boost/thread/pthread/recursive_mutex.hpp:105

4 0x0000000000e4626e in boost::unique_lockboost::recursive_mutex::lock (this=0x7ffffedf7230) at /usr/local/include/boost/thread/locks.hpp:508

5 0x0000000000e4dd59 in boost::unique_lockboost::recursive_mutex::uniquelock (this=0x7ffffedf7230, m=...) at /usr/local/include/boost/thread/locks.hpp:301

6 0x000000000125dc69 in ripple::NetworkOPs::processTransaction(boost::shared_ptrripple::Transaction, bool, bool, std::tr1::function<void (boost::shared_ptr, ripple::TER)>) (this=0x80507e478, trans=..., bAdmin=false, bFailHard=false, callback=...) at ./src/cpp/ripple/NetworkOPs.cpp:334

7 0x000000000107b70e in ripple::NetworkOPs::processTransaction (this=0x80507e478, transaction=..., bAdmin=false, bFailHard=false) at ./src/cpp/ripple/NetworkOPs.h:142

8 0x00000000011c4ec0 in ripple::checkTransaction (flags=0, stx=..., peer=...) at ./src/cpp/ripple/ripple_Peer.cpp:1082

9 0x0000000001202ef0 in std::tr1::_Bind<void (*(std::tr1::_Placeholder<1>, int, boost::shared_ptrripple::SerializedTransaction, boost::weak_ptrripple::Peer))(ripple::Job&, int, boost::shared_ptrripple::SerializedTransaction, boost::weak_ptrripple::Peer)>::__call<ripple::Job&, 0, 1, 2, 3>(std::tr1::tupleripple::Job& const&, std::tr1::_Index_tuple<0, 1, 2, 3>) (

this=0x8050e9520, __args=...) at /usr/local/lib/gcc46/include/c++/tr1/functional:1153

10 0x00000000011fac74 in std::tr1::_Bind<void (*(std::tr1::_Placeholder<1>, int, boost::shared_ptrripple::SerializedTransaction, boost::weak_ptrripple::Peer))(ripple::Job&, int, boost::shared_ptrripple::SerializedTransaction, boost::weak_ptrripple::Peer)>::operator()ripple::Job(ripple::Job&) (this=0x8050e9520)

at /usr/local/lib/gcc46/include/c++/tr1/functional:1207

11 0x00000000011f1ada in std::tr1::_Function_handler<void (ripple::Job&), std::tr1::_Bind<void (*(std::tr1::_Placeholder<1>, int, boost::shared_ptrripple::SerializedTransaction, boost::weak_ptrripple::Peer))(ripple::Job&, int, boost::shared_ptrripple::SerializedTransaction, boost::weak_ptrripple::Peer)> >::_M_invoke(std::tr1::_Any_data const&, ripple::Job&) (

__functor=..., __args#0=...) at /usr/local/lib/gcc46/include/c++/tr1/functional:1684

12 0x0000000000ebca34 in std::tr1::function<void (ripple::Job&)>::operator()(ripple::Job&) const (this=0x7ffffedf7ce0, __args#0=...)

at /usr/local/lib/gcc46/include/c++/tr1/functional:2040

13 0x0000000000eb4f0e in ripple::Job::doJob (this=0x7ffffedf7cd0) at modules/ripple_core/functional/ripple_Job.cpp:41

14 0x0000000000eb6aa2 in ripple::JobQueue::threadEntry (this=0x80507ea48) at modules/ripple_core/functional/ripple_JobQueue.cpp:268

15 0x0000000000ec605a in std::tr1::_Mem_fn<void (ripple::JobQueue::*)()>::operator() (this=0x8050aa1d0, __object=0x80507ea48) at /usr/local/lib/gcc46/include/c++/tr1/functional:572

16 0x0000000000ec5f4e in std::tr1::_Bind<std::tr1::_Mem_fn<void (ripple::JobQueue::)()> (ripple::JobQueue)>::__call<, 0>(std::tr1::tuple<> const&, std::tr1::_Index_tuple<0>) (

this=0x8050aa1d0, __args=...) at /usr/local/lib/gcc46/include/c++/tr1/functional:1153

17 0x0000000000ec5c9e in std::tr1::_Bind<std::tr1::_Mem_fn<void (ripple::JobQueue::)()> (ripple::JobQueue)>::operator()<>() (this=0x8050aa1d0)

at /usr/local/lib/gcc46/include/c++/tr1/functional:1207

18 0x0000000000ec588c in boost::detail::thread_data<std::tr1::_Bind<std::tr1::_Mem_fn<void (ripple::JobQueue::)()> (ripple::JobQueue)> >::run() (this=0x8050aa100)

at /usr/local/include/boost/thread/detail/thread.hpp:78

19 0x000000080285fac3 in thread_proxy () from /usr/local/lib/libboost_thread.so.5

20 0x0000000801f805e1 in ?? () from /lib/libthr.so.3

21 0x0000000000000000 in ?? ()

Thread 7 (Thread 80500a580 (LWP 100473)):

0 0x000000080331c04c in thr_kill () from /lib/libc.so.7

1 0x00000008033b902b in abort () from /lib/libc.so.7

2 0x00000008033a2355 in __assert () from /lib/libc.so.7

3 0x000000000107cfed in ripple::LoadManager::threadRun (this=0x80504fc00) at ./src/cpp/ripple/ripple_LoadManager.cpp:355

4 0x0000000000dfe36d in beast::InterruptibleThread::run (this=0x80504fc10) at Subtrees/beast/modules/beast_basics/threads/beast_InterruptibleThread.cpp:209

5 0x0000000000dfdda8 in beast::InterruptibleThread::ThreadHelper::run (this=0x80504fc10) at Subtrees/beast/modules/beast_basics/threads/beast_InterruptibleThread.cpp:34

6 0x0000000000dc03f5 in beast::Thread::threadEntryPoint (this=0x80504fc10) at Subtrees/beast/modules/beast_core/threads/beast_Thread.cpp:89

7 0x0000000000dc045e in beast::beast_threadEntryPoint (userData=0x80504fc10) at Subtrees/beast/modules/beast_core/threads/beast_Thread.cpp:101

8 0x0000000000dd9da3 in beast::threadEntryProc (userData=0x80504fc10) at Subtrees/beast/modules/beast_core/native/beast_posix_SharedCode.h:829

9 0x0000000801f805e1 in ?? () from /lib/libthr.so.3

10 0x0000000000000000 in ?? ()

Thread 6 (Thread 80500a3c0 (LWP 100476)):

0 0x00000008033a238c in kevent () from /lib/libc.so.7

1 0x00000000010ccae5 in boost::asio::detail::kqueue_reactor::run (this=0x805020da0, block=true, ops=...) at /usr/local/include/boost/asio/detail/impl/kqueue_reactor.ipp:365

2 0x00000000010cde57 in boost::asio::detail::task_io_service::do_run_one (this=0x805023d00, lock=..., this_thread=..., ec=...)

at /usr/local/include/boost/asio/detail/impl/task_io_service.ipp:403

3 0x00000000010cd88c in boost::asio::detail::task_io_service::run (this=0x805023d00, ec=...) at /usr/local/include/boost/asio/detail/impl/task_io_service.ipp:160

4 0x00000000010ce231 in boost::asio::io_service::run (this=0x80507e038) at /usr/local/include/boost/asio/impl/io_service.ipp:59

5 0x00000000011cf4a5 in ripple::runAux (svc=...) at ./src/cpp/ripple/ripple_Application.cpp:383

6 0x0000000001226c63 in std::tr1::_Bind<void (*(boost::reference_wrapper))(boost::asio::io_service&)>::call<, 0>(std::tr1::tuple<> const&, std::tr1::_Index_tuple<0>) (this=0x805013fd0, args=...) at /usr/local/lib/gcc46/include/c++/tr1/functional:1153

7 0x0000000001226b32 in std::tr1::_Bind<void (*(boost::reference_wrapper))(boost::asio::io_service&)>::operator()<>() (this=0x805013fd0)

at /usr/local/lib/gcc46/include/c++/tr1/functional:1207

8 0x0000000001226700 in boost::detail::thread_data<std::tr1::_Bind<void (*(boost::reference_wrapper))(boost::asio::io_service&)> >::run() (this=0x805013f00)

at /usr/local/include/boost/thread/detail/thread.hpp:78

9 0x000000080285fac3 in thread_proxy () from /usr/local/lib/libboost_thread.so.5

10 0x0000000801f805e1 in ?? () from /lib/libthr.so.3

11 0x0000000000000000 in ?? ()

Thread 5 (Thread 80500a200 (LWP 100486)):

0 0x0000000801f8a64c in ?? () from /lib/libthr.so.3

1 0x0000000801f886e1 in ?? () from /lib/libthr.so.3

2 0x00000000010d8d6b in boost::asio::detail::posix_event::waitboost::asio::detail::scoped_lock (this=0x7ffffe7f4eb0, lock=...)

at /usr/local/include/boost/asio/detail/posix_event.hpp:80

3 0x00000000010cdf36 in boost::asio::detail::task_io_service::do_run_one (this=0x805023d80, lock=..., this_thread=..., ec=...)

at /usr/local/include/boost/asio/detail/impl/task_io_service.ipp:430

4 0x00000000010cd88c in boost::asio::detail::task_io_service::run (this=0x805023d80, ec=...) at /usr/local/include/boost/asio/detail/impl/task_io_service.ipp:160

5 0x00000000010ce231 in boost::asio::io_service::run (this=0x8050258f0) at /usr/local/include/boost/asio/impl/io_service.ipp:59

6 0x0000000001170a95 in boost::asio::detail::resolver_service_base::work_io_service_runner::operator() (this=0x805025d28)

at /usr/local/include/boost/asio/detail/impl/resolver_service_base.ipp:32

7 0x00000000011bd26c in boost::asio::detail::posix_thread::funcboost::asio::detail::resolver_service_base::work_io_service_runner::run (this=0x805025d20)

at /usr/local/include/boost/asio/detail/posix_thread.hpp:82

9 0x0000000801f805e1 in ?? () from /lib/libthr.so.3

10 0x0000000000000000 in ?? ()

Thread 4 (Thread 80500a040 (LWP 100487)):

0 0x00000008033a238c in kevent () from /lib/libc.so.7

1 0x00000000010ccae5 in boost::asio::detail::kqueue_reactor::run (this=0x80543da50, block=true, ops=...) at /usr/local/include/boost/asio/detail/impl/kqueue_reactor.ipp:365

2 0x00000000010cde57 in boost::asio::detail::task_io_service::do_run_one (this=0x80551d180, lock=..., this_thread=..., ec=...)

at /usr/local/include/boost/asio/detail/impl/task_io_service.ipp:403

3 0x00000000010cd88c in boost::asio::detail::task_io_service::run (this=0x80551d180, ec=...) at /usr/local/include/boost/asio/detail/impl/task_io_service.ipp:160

4 0x00000000010ce231 in boost::asio::io_service::run (this=0x8055f88c0) at /usr/local/include/boost/asio/impl/io_service.ipp:59

5 0x00000000010d5ed3 in websocketpp::endpoint_base::run_internal (this=0x8055f88c0) at Subtrees/websocket/src/endpoint.hpp:53

6 0x00000000010e92fc in websocketpp::role::server<websocketpp::endpoint<websocketpp::role::server, websocketpp::socket::autotls, websocketpp::log::logger> >::start_listen (

this=0x8055f88d0, e=..., num_threads=0) at Subtrees/websocket/src/roles/server.hpp:347

7 0x00000000010e0851 in websocketpp::role::server<websocketpp::endpoint<websocketpp::role::server, websocketpp::socket::autotls, websocketpp::log::logger> >::listen (this=0x8055f88d0,

e=..., num_threads=1) at Subtrees/websocket/src/roles/server.hpp:266

8 0x00000000010ca390 in ripple::WSDoor::startListening (this=0x8051a2610) at ./src/cpp/ripple/WSDoor.cpp:51

9 0x00000000011488ea in std::tr1::_Mem_fn<void (ripple::WSDoor::*)()>::operator() (this=0x805013ad0, __object=0x8051a2610) at /usr/local/lib/gcc46/include/c++/tr1/functional:572

10 0x00000000011471ee in std::tr1::_Bind<std::tr1::_Mem_fn<void (ripple::WSDoor::)()> (ripple::WSDoor)>::__call<, 0>(std::tr1::tuple<> const&, std::tr1::_Index_tuple<0>) (

this=0x805013ad0, __args=...) at /usr/local/lib/gcc46/include/c++/tr1/functional:1153

11 0x0000000001145a68 in std::tr1::_Bind<std::tr1::_Mem_fn<void (ripple::WSDoor::)()> (ripple::WSDoor)>::operator()<>() (this=0x805013ad0)

at /usr/local/lib/gcc46/include/c++/tr1/functional:1207

12 0x0000000001143b48 in boost::detail::thread_data<std::tr1::_Bind<std::tr1::_Mem_fn<void (ripple::WSDoor::)()> (ripple::WSDoor)> >::run() (this=0x805013a00)

at /usr/local/include/boost/thread/detail/thread.hpp:78

13 0x000000080285fac3 in thread_proxy () from /usr/local/lib/libboost_thread.so.5

14 0x0000000801f805e1 in ?? () from /lib/libthr.so.3

15 0x00007ffffe3f4000 in ?? ()

Cannot access memory at address 0x7ffffe5f4000

rec commented 9 years ago

Can't reproduce, feel free to re-open with a new test case or report!