Closed nathanielhourt closed 10 years ago
had the same crash yesterday:
--- there are now 31 active connections to the p2p network
--- there are now 30 active connections to the p2p network
--- there are now 29 active connections to the p2p network
--- there are now 30 active connections to the p2p network
myhost (unlocked) >>>
Program received signal SIGSEGV, Segmentation fault.
fc::mutex::unlock() () at /home/admin/.BitSharesX_build/libraries/fc/src/thread/mutex.cpp:33
33 while( n->next_blocked_mutex ) {
(gdb) bt
#0 fc::mutex::unlock() () at /home/admin/.BitSharesX_build/libraries/fc/src/thread/mutex.cpp:33
#1 0x00000000007aadcd in bts::net::detail::message_oriented_connection_impl::send_message(bts::net::message const&) ()
at /home/admin/.BitSharesX_build/libraries/fc/include/fc/thread/scoped_lock.hpp:8
#2 0x00000000007a0980 in bts::net::peer_connection::send_queued_messages_task() () at /home/admin/.BitSharesX_build/libraries/net/peer_connection.cpp:178
#3 0x00000000007a107c in fc::detail::void_functor_run<bts::net::peer_connection::send_message(bts::net::message const&)::{lambda()#1}>::run(void*, fc::detail::void_functor_run<bts::net::peer_connection::send_message(bts::net::message const&)::{lambda()#1}>) () at /home/admin/.BitSharesX_build/libraries/net/peer_connection.cpp:222
#4 0x0000000000662880 in fc::task_base::run_impl() () at /home/admin/.BitSharesX_build/libraries/fc/src/thread/task.cpp:42
#5 0x00000000006612ab in fc::thread_d::process_tasks() () at /home/admin/.BitSharesX_build/libraries/fc/src/thread/thread_d.hpp:377
#6 0x00000000006614f3 in fc::thread_d::start_process_tasks(long) () at /home/admin/.BitSharesX_build/libraries/fc/src/thread/thread_d.hpp:357
#7 0x0000000000d226be in make_fcontext ()
#8 0x00000000018292e0 in ?? ()
#9 0x0000000000000000 in ?? ()
I got another seemingly network-related crash on 0.4.4:
--- there are now 60 active connections to the p2p network
--- there are now 59 active connections to the p2p network
myhost (unlocked) >>>
Program received signal SIGSEGV, Segmentation fault.
fc::tcp_socket::close() () at /usr/include/boost/asio/basic_io_object.hpp:214
214 return *service_;
(gdb) bt
#0 fc::tcp_socket::close() () at /usr/include/boost/asio/basic_io_object.hpp:214
#1 0x00000000007cde0a in bts::net::stcp_socket::close() () at /home/admin/.BitSharesX_build/libraries/net/stcp_socket.cpp:107
#2 0x00000000007a565c in bts::net::peer_connection::send_queued_messages_task() () at /home/admin/.BitSharesX_build/libraries/net/peer_connection.cpp:185
#3 0x00000000007a5abc in fc::detail::void_functor_run<bts::net::peer_connection::send_message(bts::net::message const&)::{lambda()#1}>::run(void*, fc::detail::void_functor_run<bts::net::peer_connection::send_message(bts::net::message const&)::{lambda()#1}>) () at /home/admin/.BitSharesX_build/libraries/net/peer_connection.cpp:222
#4 0x0000000000667550 in fc::task_base::run_impl() () at /home/admin/.BitSharesX_build/libraries/fc/src/thread/task.cpp:42
#5 0x0000000000665f7b in fc::thread_d::process_tasks() () at /home/admin/.BitSharesX_build/libraries/fc/src/thread/thread_d.hpp:377
#6 0x00000000006661c3 in fc::thread_d::start_process_tasks(long) () at /home/admin/.BitSharesX_build/libraries/fc/src/thread/thread_d.hpp:357
#7 0x0000000000d2f91e in make_fcontext ()
#8 0x00000000018322e0 in ?? ()
#9 0x0000000000000000 in ?? ()
Can we get a copy of the P2P log file after this crash?
added assertions to aid in debugging this problem (particularly the first two stack traces in this issue) f75e4b490b871a67aea7608d13386f279693b9f4 and c88976b22d98a7e5dbe8563f244f7d795081dff1
Here's a link: https://www.dropbox.com/s/t1avmr9cgxzwt6p/p2p.log.20140821T130000.lzma?dl=0 Sorry there's a lot of noise in the logs as I haven't solved this issue yet: https://github.com/BitShares/bitshares_toolkit/issues/585 I did upgrade my kernel though, which didn't fix it. Haven't upgraded my libc yet so I suspect this might be the issue...
some crash here, version 0.4.4, version 0.4.6 too
(wallet closed) >>>
Program received signal SIGSEGV, Segmentation fault.
fc::tcp_socket::close (this=0x6b63617473222c8a) at /root/tmp/bitshares_toolkit/libraries/fc/src/network/tcp_socket.cpp:64
64 if( is_open() )
(gdb) bt
#0 fc::tcp_socket::close (this=0x6b63617473222c8a) at /root/tmp/bitshares_toolkit/libraries/fc/src/network/tcp_socket.cpp:64
#1 0x00000000007d33ba in bts::net::stcp_socket::close (this=<optimized out>) at /root/tmp/bitshares_toolkit/libraries/net/stcp_socket.cpp:107
#2 0x00000000007b4cac in close_connection (this=<optimized out>) at /root/tmp/bitshares_toolkit/libraries/net/message_oriented_connection.cpp:205
#3 bts::net::message_oriented_connection::close_connection (this=this@entry=0x22cdcd68) at /root/tmp/bitshares_toolkit/libraries/net/message_oriented_connection.cpp:272
#4 0x00000000007a98fe in bts::net::peer_connection::close_connection (this=this@entry=0x22cdcd00) at /root/tmp/bitshares_toolkit/libraries/net/peer_connection.cpp:230
#5 0x00000000007aa81c in bts::net::peer_connection::send_queued_messages_task (this=0x22cdcd00) at /root/tmp/bitshares_toolkit/libraries/net/peer_connection.cpp:185
#6 0x00000000007aac7c in operator() (__closure=<optimized out>) at /root/tmp/bitshares_toolkit/libraries/net/peer_connection.cpp:222
#7 fc::detail::void_functor_run<bts::net::peer_connection::send_message(const bts::net::message&)::__lambda2>::run(void *, void *) (functor=<optimized out>, prom=0x24cfd490)
at /root/tmp/bitshares_toolkit/libraries/fc/include/fc/thread/task.hpp:56
#8 0x000000000066c5a3 in fc::task_base::run_impl (this=this@entry=0x24cfd4a0) at /root/tmp/bitshares_toolkit/libraries/fc/src/thread/task.cpp:42
#9 0x000000000066cc55 in fc::task_base::run (this=this@entry=0x24cfd4a0) at /root/tmp/bitshares_toolkit/libraries/fc/src/thread/task.cpp:31
#10 0x000000000066addb in run_next_task (this=0x184a2c0) at /root/tmp/bitshares_toolkit/libraries/fc/src/thread/thread_d.hpp:377
#11 fc::thread_d::process_tasks (this=this@entry=0x184a2c0) at /root/tmp/bitshares_toolkit/libraries/fc/src/thread/thread_d.hpp:400
#12 0x000000000066b026 in fc::thread_d::start_process_tasks (my=25469632) at /root/tmp/bitshares_toolkit/libraries/fc/src/thread/thread_d.hpp:357
#13 0x0000000000d45d2e in make_fcontext ()
#14 0x000000000184a2c0 in ?? ()
#15 0x0000000000000000 in ?? ()
This issue may be fixed in the latest code; we're not certain yet because it's so difficult to reproduce. When DacSun drops 0.4.5, be sure to update to it and post back here if these crashes persist.
just got same error on 0.4.5
(gdb) bt
#0 get_tail (next=<synthetic pointer>, h=0x907676b313d3a84b) at /home/user/bitsharesx/libraries/fc/src/thread/mutex.cpp:33
#1 fc::mutex::unlock (this=this@entry=0x302fe4c) at /home/user/bitsharesx/libraries/fc/src/thread/mutex.cpp:160
#2 0x00000000007b5226 in ~scoped_lock (this=<synthetic pointer>, __in_chrg=<optimized out>) at /home/user/bitsharesx/libraries/fc/include/fc/thread/scoped_lock.hpp:8
#3 bts::net::detail::message_oriented_connection_impl::send_message (this=0x83b8f563388ca909, message_to_send=...) at /home/user/bitsharesx/libraries/net/message_oriented_connection.cpp:195
#4 0x00000000007b5928 in bts::net::message_oriented_connection::send_message (this=this@entry=0x10f8ef98, message_to_send=...)
at /home/user/bitsharesx/libraries/net/message_oriented_connection.cpp:267
#5 0x00000000007ab220 in bts::net::peer_connection::send_queued_messages_task (this=0x10f8ef30) at /home/user/bitsharesx/libraries/net/peer_connection.cpp:178
#6 0x00000000007ab91c in operator() (__closure=<optimized out>) at /home/user/bitsharesx/libraries/net/peer_connection.cpp:222
#7 fc::detail::void_functor_run<bts::net::peer_connection::send_message(const bts::net::message&)::__lambda2>::run(void *, void *) (functor=<optimized out>, prom=0xed599f0)
at /home/user/bitsharesx/libraries/fc/include/fc/thread/task.hpp:56
#8 0x000000000066c483 in fc::task_base::run_impl (this=this@entry=0xed59a00) at /home/user/bitsharesx/libraries/fc/src/thread/task.cpp:42
#9 0x000000000066cb35 in fc::task_base::run (this=this@entry=0xed59a00) at /home/user/bitsharesx/libraries/fc/src/thread/task.cpp:31
#10 0x000000000066acbb in run_next_task (this=0x184f350) at /home/user/bitsharesx/libraries/fc/src/thread/thread_d.hpp:379
#11 fc::thread_d::process_tasks (this=this@entry=0x184f350) at /home/user/bitsharesx/libraries/fc/src/thread/thread_d.hpp:402
#12 0x000000000066af06 in fc::thread_d::start_process_tasks (my=25490256) at /home/user/bitsharesx/libraries/fc/src/thread/thread_d.hpp:359
#13 0x0000000000d4a85e in make_fcontext ()
#14 0x000000000184f350 in ?? ()
#15 0x0000000000000000 in ?? ()
(gdb)
let me know if you need anything else.
--- there are now 98 active connections to the p2p network
default (unlocked) >>>
Program received signal SIGSEGV, Segmentation fault.
get_tail (next=<synthetic pointer>, h=0x20657564206e6f69) at /root/bitsharesx/libraries/fc/src/thread/mutex.cpp:33
33 while( n->next_blocked_mutex ) {
(gdb) bt
#0 get_tail (next=<synthetic pointer>, h=0x20657564206e6f69) at /root/bitsharesx/libraries/fc/src/thread/mutex.cpp:33
#1 fc::mutex::unlock (this=this@entry=0x13589e4f) at /root/bitsharesx/libraries/fc/src/thread/mutex.cpp:160
#2 0x00000000007b52a6 in ~scoped_lock (this=<synthetic pointer>, __in_chrg=<optimized out>) at /root/bitsharesx/libraries/fc/include/fc/thread/scoped_lock.hpp:8
#3 bts::net::detail::message_oriented_connection_impl::send_message (this=0xd7c714ba8ffde62c, message_to_send=...) at /root/bitsharesx/libraries/net/message_oriented_connection.cpp:195
#4 0x00000000007b59a8 in bts::net::message_oriented_connection::send_message (this=this@entry=0x1339d148, message_to_send=...) at /root/bitsharesx/libraries/net/message_oriented_connection.cpp:267
#5 0x00000000007ab2a0 in bts::net::peer_connection::send_queued_messages_task (this=0x1339d0e0) at /root/bitsharesx/libraries/net/peer_connection.cpp:178
#6 0x00000000007ab99c in operator() (__closure=<optimized out>) at /root/bitsharesx/libraries/net/peer_connection.cpp:222
#7 fc::detail::void_functor_run<bts::net::peer_connection::send_message(const bts::net::message&)::__lambda2>::run(void *, void *) (functor=<optimized out>, prom=0x12c2c430) at /root/bitsharesx/libraries/fc/include/fc/thread/task.hpp:56
#8 0x000000000066c503 in fc::task_base::run_impl (this=this@entry=0x12c2c440) at /root/bitsharesx/libraries/fc/src/thread/task.cpp:42
#9 0x000000000066cbb5 in fc::task_base::run (this=this@entry=0x12c2c440) at /root/bitsharesx/libraries/fc/src/thread/task.cpp:31
#10 0x000000000066ad3b in run_next_task (this=0x184f350) at /root/bitsharesx/libraries/fc/src/thread/thread_d.hpp:379
#11 fc::thread_d::process_tasks (this=this@entry=0x184f350) at /root/bitsharesx/libraries/fc/src/thread/thread_d.hpp:402
#12 0x000000000066af86 in fc::thread_d::start_process_tasks (my=25490256) at /root/bitsharesx/libraries/fc/src/thread/thread_d.hpp:359
#13 0x0000000000d4a8fe in make_fcontext ()
#14 0x000000000184f350 in ?? ()
#15 0x0000000000000000 in ?? ()
(gdb)
Stack Trace from riverhead-del-server-1
P2P Logs: https://www.dropbox.com/s/ybrli5ruc2xdn3y/riverhead-del-server-1.p2p_logs.tar.gz
Python Exception <class 'ImportError'> No module named 'gdb':
#0 get_tail (next=<synthetic pointer>, h=0x1e0) at /home/riverhead/github/bitsharesx/libraries/fc/src/thread/mutex.cpp:33
#1 fc::mutex::unlock (this=this@entry=0x2539a18) at /home/riverhead/github/bitsharesx/libraries/fc/src/thread/mutex.cpp:160
#2 0x00000000007b5290 in ~scoped_lock (this=<synthetic pointer>, __in_chrg=<optimized out>)
at /home/riverhead/github/bitsharesx/libraries/fc/include/fc/thread/scoped_lock.hpp:8
#3 bts::net::detail::message_oriented_connection_impl::send_message (this=0x2539850, message_to_send=...)
at /home/riverhead/github/bitsharesx/libraries/net/message_oriented_connection.cpp:195
#4 0x00000000007b5928 in bts::net::message_oriented_connection::send_message (this=this@entry=0x26c0948, message_to_send=...)
at /home/riverhead/github/bitsharesx/libraries/net/message_oriented_connection.cpp:267
#5 0x00000000007ab220 in bts::net::peer_connection::send_queued_messages_task (this=0x26c08e0)
at /home/riverhead/github/bitsharesx/libraries/net/peer_connection.cpp:178
#6 0x00000000007ab91c in operator() (__closure=<optimized out>)
at /home/riverhead/github/bitsharesx/libraries/net/peer_connection.cpp:222
#7 fc::detail::void_functor_run<bts::net::peer_connection::send_message(const bts::net::message&)::__lambda2>::run(void *, void *) (
functor=<optimized out>, prom=0x1d4fca0) at /home/riverhead/github/bitsharesx/libraries/fc/include/fc/thread/task.hpp:56
#8 0x000000000066c483 in fc::task_base::run_impl (this=this@entry=0x1d4fcb0)
at /home/riverhead/github/bitsharesx/libraries/fc/src/thread/task.cpp:42
#9 0x000000000066cb35 in fc::task_base::run (this=this@entry=0x1d4fcb0)
at /home/riverhead/github/bitsharesx/libraries/fc/src/thread/task.cpp:31
#10 0x000000000066acbb in run_next_task (this=0x184f350) at /home/riverhead/github/bitsharesx/libraries/fc/src/thread/thread_d.hpp:379
#11 fc::thread_d::process_tasks (this=this@entry=0x184f350)
at /home/riverhead/github/bitsharesx/libraries/fc/src/thread/thread_d.hpp:402
#12 0x000000000066af06 in fc::thread_d::start_process_tasks (my=25490256)
at /home/riverhead/github/bitsharesx/libraries/fc/src/thread/thread_d.hpp:359
#13 0x0000000000d4a85e in make_fcontext ()
#14 0x000000000184f350 in ?? ()
#15 0x0000000000000000 in ?? ()
There is still crash in version 0.4.6:
rocess 54815 stopped
* thread #1: tid = 0x1af6d8, 0x0000000100568b34 bitshares_client`fc::tcp_socket::close() [inlined] boost::asio::detail::reactive_socket_service_base::is_open(boost::asio::detail::reactive_socket_service_base::base_implementation_type const&) const + 16 at reactive_socket_service_base.hpp:89, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x88)
frame #0: 0x0000000100568b34 bitshares_client`fc::tcp_socket::close() [inlined] boost::asio::detail::reactive_socket_service_base::is_open(boost::asio::detail::reactive_socket_service_base::base_implementation_type const&) const + 16 at reactive_socket_service_base.hpp:89
86 // Determine whether the socket is open.
87 bool is_open(const base_implementation_type& impl) const
88 {
-> 89 return impl.socket_ != invalid_socket;
90 }
91
92 // Destroy a socket implementation.
(lldb) bt
* thread #1: tid = 0x1af6d8, 0x0000000100568b34 bitshares_client`fc::tcp_socket::close() [inlined] boost::asio::detail::reactive_socket_service_base::is_open(boost::asio::detail::reactive_socket_service_base::base_implementation_type const&) const + 16 at reactive_socket_service_base.hpp:89, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x88)
* frame #0: 0x0000000100568b34 bitshares_client`fc::tcp_socket::close() [inlined] boost::asio::detail::reactive_socket_service_base::is_open(boost::asio::detail::reactive_socket_service_base::base_implementation_type const&) const + 16 at reactive_socket_service_base.hpp:89
frame #1: 0x0000000100568b24 bitshares_client`fc::tcp_socket::close() [inlined] boost::asio::stream_socket_service<boost::asio::ip::tcp>::is_open(boost::asio::detail::reactive_socket_service<boost::asio::ip::tcp>::implementation_type const&) const at stream_socket_service.hpp:163
frame #2: 0x0000000100568b24 bitshares_client`fc::tcp_socket::close() [inlined] boost::asio::basic_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >::is_open() const at basic_socket.hpp:338
frame #3: 0x0000000100568b24 bitshares_client`fc::tcp_socket::close() [inlined] fc::tcp_socket::is_open(this=0x0000000000000068) const at tcp_socket.cpp:54
frame #4: 0x0000000100568b24 bitshares_client`fc::tcp_socket::close(this=0x0000000000000068) + 4 at tcp_socket.cpp:64
frame #5: 0x000000010042abdd bitshares_client`bts::net::stcp_socket::close(this=<unavailable>) + 29 at stcp_socket.cpp:107
frame #6: 0x000000010043a29b bitshares_client`bts::net::peer_connection::send_queued_messages_task(this=<unavailable>) + 731 at peer_connection.cpp:230
frame #7: 0x000000010043b781 bitshares_client`fc::detail::void_functor_run<bts::net::peer_connection::send_message(bts::net::message const&)::$_0>::run(void*, void*) [inlined] operator() + 17 at peer_connection.cpp:223
frame #8: 0x000000010043b77c bitshares_client`fc::detail::void_functor_run<bts::net::peer_connection::send_message(functor=<unavailable>, prom=0x0000000130a1ee00)::$_0>::run(void*, void*) + 12 at task.hpp:56
frame #9: 0x00000001004fa461 bitshares_client`fc::task_base::run_impl(this=0x0000000130a1ee10) + 481 at task.cpp:42
frame #10: 0x00000001004fd5c1 bitshares_client`fc::thread_d::run_next_task(this=0x0000000101b00c00) + 65 at thread_d.hpp:379
frame #11: 0x00000001004fd360 bitshares_client`fc::thread_d::process_tasks(this=0x0000000101b00c00) + 144 at thread_d.hpp:402
frame #12: 0x00000001004fcdaa bitshares_client`fc::thread_d::start_process_tasks(my=4323281920) + 26 at thread_d.hpp:359
My crashes seem to have gone away after wiping the bitsharesx folder and cloning. 0.4.5 is now running stable for me. Is it possible that these crashes are caused by git not syncing properly, or by some file that a previous version built that is still hanging out?
edit NVM seed node is still crashing.
Mine crashes went away when I moved to a computer behind a router (vs Digital Ocean). Both Ubuntu 14.04 systems.
--- there are now 31 active connections to the p2p network
(wallet closed) >>>
Program received signal SIGSEGV, Segmentation fault.
fc::tcp_socket::close (this=0x3a2265646f6322e3) at /home/user/bitsharesx/libraries/fc/src/network/tcp_socket.cpp:64
64 if( is_open() )
(gdb) bt
#0 fc::tcp_socket::close (this=0x3a2265646f6322e3) at /home/user/bitsharesx/libraries/fc/src/network/tcp_socket.cpp:64
#1 0x00000000007d6f9a in bts::net::stcp_socket::close (this=<optimized out>) at /home/user/bitsharesx/libraries/net/stcp_socket.cpp:107
#2 0x00000000007b888c in close_connection (this=<optimized out>) at /home/user/bitsharesx/libraries/net/message_oriented_connection.cpp:222
#3 bts::net::message_oriented_connection::close_connection (this=this@entry=0xd69eb38) at /home/user/bitsharesx/libraries/net/message_oriented_connection.cpp:289
#4 0x00000000007ad5be in bts::net::peer_connection::close_connection (this=this@entry=0xd69ead0) at /home/user/bitsharesx/libraries/net/peer_connection.cpp:231
#5 0x00000000007ae3c9 in bts::net::peer_connection::send_queued_messages_task (this=0xd69ead0) at /home/user/bitsharesx/libraries/net/peer_connection.cpp:186
#6 0x00000000007ae82c in operator() (__closure=<optimized out>) at /home/user/bitsharesx/libraries/net/peer_connection.cpp:223
#7 fc::detail::void_functor_run<bts::net::peer_connection::send_message(const bts::net::message&)::__lambda2>::run(void *, void *) (functor=<optimized out>, prom=0xd2fd6e0)
at /home/user/bitsharesx/libraries/fc/include/fc/thread/task.hpp:56
#8 0x000000000066ea13 in fc::task_base::run_impl (this=this@entry=0xd2fd6f0) at /home/user/bitsharesx/libraries/fc/src/thread/task.cpp:42
#9 0x000000000066f0c5 in fc::task_base::run (this=this@entry=0xd2fd6f0) at /home/user/bitsharesx/libraries/fc/src/thread/task.cpp:31
#10 0x000000000066d24b in run_next_task (this=0x185a350) at /home/user/bitsharesx/libraries/fc/src/thread/thread_d.hpp:379
#11 fc::thread_d::process_tasks (this=this@entry=0x185a350) at /home/user/bitsharesx/libraries/fc/src/thread/thread_d.hpp:402
#12 0x000000000066d496 in fc::thread_d::start_process_tasks (my=25535312) at /home/user/bitsharesx/libraries/fc/src/thread/thread_d.hpp:359
#13 0x0000000000d53ace in make_fcontext ()
#14 0x000000000185a350 in ?? ()
#15 0x0000000000000000 in ?? ()
P2P Log: https://www.dropbox.com/s/ij8drowc8kv24zd/p2p.log.20140822T230000?dl=0
Should be fixed by:
But at least #693 seems to persist.
This issue appears to only affect active/signing delegate nodes, but like all ephemeral crashes, it's hard to say for certain.