Closed rgarver closed 6 years ago
iFoggz
commented
6 years ago since u compiled with debug symbols you should run this with gdb. the back trace will be more precise. There is many moving parts involving that function. The cause would be a call to it passing the argument in question that led to failure in ThreadMessageHandler2().
gdb --args src/gridcoinresearchd -testnet -printtoconsole -debug=true -debugnet=1
if you have experience with gdb then your a step ahead.
rgarver
commented
6 years ago Here's what I got:
#0 0x00007ffff5a35428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007ffff5a3702a in __GI_abort () at abort.c:89
#2 0x00007ffff659584d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007ffff65936b6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007ffff6593701 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007ffff6593969 in __cxa_rethrow () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00005555558ca0c7 in PrintException (pex=0x7fffd4000a00, pszThread=0x555555a0746a "ThreadSocketHandler()") at util.cpp:1039
#7 0x000055555577e81c in ThreadSocketHandler (parg=0x0) at net.cpp:985
#8 0x00005555558dd521 in boost::_bi::list1<boost::_bi::value<void*> >::operator()<void (*)(void*), boost::_bi::list0> (this=0x7fffdc001da0, f=@0x7fffdc001d98: 0x55555577e7a5 <ThreadSocketHandler(void*)>, a=...) at /usr/include/boost/bind/bind.hpp:253
#9 0x00005555558dd2c8 in boost::_bi::bind_t<void, void (*)(void*), boost::_bi::list1<boost::_bi::value<void*> > >::operator() (this=0x7fffdc001d98) at /usr/include/boost/bind/bind.hpp:893
#10 0x00005555558dd00f in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(void*), boost::_bi::list1<boost::_bi::value<void*> > > >::run (this=0x7fffdc001be0) at /usr/include/boost/thread/detail/thread.hpp:116
#11 0x00007ffff75285d5 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
#12 0x00007ffff5dd16ba in start_thread (arg=0x7fffe880f700) at pthread_create.c:333
#13 0x00007ffff5b073dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
denravonska
commented
6 years ago Can you break on util.cpp:1039 and print message.c_str()?
rgarver
commented
6 years ago Exiting AppInit2
[New Thread 0x7fffe880f700 (LWP 17040)]
[Thread 0x7fffe9010700 (LWP 17039) exited]
************************
[New Thread 0x7fffdbfff700 (LWP 17041)]
EXCEPTION: N5boost16exception_detail10clone_implINS0_19error_info_injectorINS_10lock_errorEEEEE
boost: mutex lock failed in pthread_mutex_lock: Invalid argument
gridcoin in ThreadSocketHandler()
************************
EXCEPTION: N5boost16exception_detail10clone_implINS0_19error_info_injectorINS_10lock_errorEEEEE
boost: mutex lock failed in pthread_mutex_lock: Invalid argument
gridcoin in ThreadSocketHandler()
************************
EXCEPTION: N5boost16exception_detail10clone_implINS0_19error_info_injectorINS_10lock_errorEEEEE
boost: mutex lock failed in pthread_mutex_lock: Invalid argument
gridcoin in ThreadOpenAddedConnections()
************************
EXCEPTION: N5boost16exception_detail10clone_implINS0_19error_info_injectorINS_10lock_errorEEEEE
boost: mutex lock failed in pthread_mutex_lock: Invalid argument
gridcoin in ThreadOpenAddedConnections()
[New Thread 0x7fffdb7fe700 (LWP 17042)]
************************
EXCEPTION: N5boost16exception_detail10clone_implINS0_19error_info_injectorINS_10lock_errorEEEEE
boost: mutex lock failed in pthread_mutex_lock: Invalid argument
gridcoin in ThreadOpenConnections()
************************
EXCEPTION: N5boost16exception_detail10clone_implINS0_19error_info_injectorINS_10lock_errorEEEEE
boost: mutex lock failed in pthread_mutex_lock: Invalid argument
gridcoin in ThreadOpenConnections()
[New Thread 0x7fffdaffd700 (LWP 17043)]
************************
[Switching to Thread 0x7fffe880f700 (LWP 17040)]
Thread 7 "grc-net" hit Breakpoint 1, PrintException (pex=0x7fffd4000a00, pszThread=0x555555a0746a "ThreadSocketHandler()") at util.cpp:1039
1039 throw;
(gdb) set scheduler-locking on
(gdb) p message.c_str()
$1 = 0x7fffd4000b50 "EXCEPTION: N5boost16exception_detail10clone_implINS0_19error_info_injectorINS_10lock_errorEEEEE \nboost: mutex lock failed in pthread_mutex_lock: Invalid argument \ngridcoin in ThreadSocketH"...
(gdb) printf "%s\n", message.c_str()
EXCEPTION: N5boost16exception_detail10clone_implINS0_19error_info_injectorINS_10lock_errorEEEEE
boost: mutex lock failed in pthread_mutex_lock: Invalid argument
gridcoin in ThreadSocketHandler() I'm happy to dig deeper here, but my C++ and GDB knowledge is 10+ years old at this point and it looks like this is a threading issue, which makes tracking down that much harder. If you have some thoughts about what I can do to track this down let me know and I'll see what I can do.
iFoggz
commented
6 years ago its in that function i think it may do with the lock there.
rgarver
commented
6 years ago Here's a little more detail on all of the threads. It looks like there are two of them failing. Or perhaps one failing and taking out the other.
...
TallyResearchAverages_v9: Superblock Loaded {c088bfa769aa7d774cbaae9a86da58e08714dc8606977d392eabf4a4b3916506 439698}
TallyMagnitudesInSuperblock: Extracted 15 magnitude entries from cached superblock 439698
[New Thread 0x7fffea012700 (LWP 16649)]
Using 8 OutboundConnections with a MaxConnections of 125
[New Thread 0x7fffe9811700 (LWP 16650)]
[New Thread 0x7fffe9010700 (LWP 16651)]
Done loading
Exiting AppInit2
[New Thread 0x7fffe880f700 (LWP 16652)]
[Thread 0x7fffe9010700 (LWP 16651) exited]
************************
EXCEPTION: N5boost16exception_detail10clone_implINS0_19error_info_injectorINS_10lock_errorEEEEE
boost: mutex lock failed in pthread_mutex_lock: Invalid argument
gridcoin in ThreadSocketHandler()
************************
EXCEPTION: N5boost16exception_detail10clone_implINS0_19error_info_injectorINS_10lock_errorEEEEE
boost: mutex lock failed in pthread_mutex_lock: Invalid argument
gridcoin in ThreadSocketHandler()
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >'
what(): boost: mutex lock failed in pthread_mutex_lock: Invalid argument
[New Thread 0x7fffdbfff700 (LWP 16654)]
************************
EXCEPTION: N5boost16exception_detail10clone_implINS0_19error_info_injectorINS_10lock_errorEEEEE
boost: mutex lock failed in pthread_mutex_lock: Invalid argument
gridcoin in ThreadOpenAddedConnections()
************************
EXCEPTION: N5boost16exception_detail10clone_implINS0_19error_info_injectorINS_10lock_errorEEEEE
boost: mutex lock failed in pthread_mutex_lock: Invalid argument
gridcoin in ThreadOpenAddedConnections()
terminate called recursively
[New Thread 0x7fffdb7fe700 (LWP 16655)]
************************
Thread 7 "grc-net" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffe880f700 (LWP 16652)]
0x00007ffff5a35428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) info threads
Id Target Id Frame
1 Thread 0x7ffff7fe3740 (LWP 16612) "gridcoinresearc" 0x00007ffff5a7fdb3 in _int_free (av=0x7ffff5dc4b20 <main_arena>, p=0x555561ec8140, have_lock=0) at malloc.c:3896
2 Thread 0x7fffeb014700 (LWP 16641) "grc-wallet" pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
3 Thread 0x7fffea813700 (LWP 16642) "grc-cpids" 0x00007ffff5b01737 in mprotect () at ../sysdeps/unix/syscall-template.S:84
4 Thread 0x7fffea012700 (LWP 16649) "grc-start" clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:81
5 Thread 0x7fffe9811700 (LWP 16650) "grc-rpclist" __memset_sse2 () at ../sysdeps/x86_64/multiarch/../memset.S:93
* 7 Thread 0x7fffe880f700 (LWP 16652) "grc-net" 0x00007ffff5a35428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
8 Thread 0x7fffdbfff700 (LWP 16654) "grc-opencon" 0x00007ffff5a35428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
9 Thread 0x7fffdb7fe700 (LWP 16655) "grc-opencon" 0x00007ffff5af72ad in write () at ../sysdeps/unix/syscall-template.S:84
(gdb) thread apply all bt
Thread 9 (Thread 0x7fffdb7fe700 (LWP 16655)):
#0 0x00007ffff5af72ad in write () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007ffff5a78bff in _IO_new_file_write (f=0x7ffff5dc5620 <_IO_2_1_stdout_>, data=0x555555e35fa0, n=27) at fileops.c:1263
#2 0x00007ffff5a7a409 in new_do_write (to_do=27,
data=0x555555e35fa0 "\n\n", '*' <repeats 24 times>, "\non_detail10clone_implINS0_19error_info_injectorINS_10lock_errorEEEEE \nboost: mutex lock failed in pthread_mutex_lock: Invalid argument \ngridcoin in ThreadOpenAdd"..., fp=0x7ffff5dc5620 <_IO_2_1_stdout_>)
at fileops.c:518
#3 _IO_new_do_write (fp=0x7ffff5dc5620 <_IO_2_1_stdout_>,
data=0x555555e35fa0 "\n\n", '*' <repeats 24 times>, "\non_detail10clone_implINS0_19error_info_injectorINS_10lock_errorEEEEE \nboost: mutex lock failed in pthread_mutex_lock: Invalid argument \ngridcoin in ThreadOpenAdd"..., to_do=27) at fileops.c:494
#4 0x00007ffff5a7947d in _IO_new_file_xsputn (f=0x7ffff5dc5620 <_IO_2_1_stdout_>, data=<optimized out>, n=27) at fileops.c:1331
#5 0x00007ffff5a4d241 in _IO_vfprintf_internal (s=0x7ffff5dc5620 <_IO_2_1_stdout_>, format=0x555555a33110 "\n\n", '*' <repeats 24 times>, "\n%s\n", ap=0x7fffdb7fdc70) at vfprintf.c:1320
#6 0x00005555558cf34f in OutputDebugStringF (pszFormat=0x555555a33110 "\n\n", '*' <repeats 24 times>, "\n%s\n") at util.cpp:232
#7 0x00005555558ca085 in PrintException (pex=0x7fffd0000a00, pszThread=0x555555a07d2f "ThreadOpenConnections()") at util.cpp:1036
#8 0x0000555555781a86 in ThreadOpenConnections (parg=0x0) at net.cpp:1697
#9 0x00005555558dd521 in boost::_bi::list1<boost::_bi::value<void*> >::operator()<void (*)(void*), boost::_bi::list0> (this=0x7fffdc0025a0, f=@0x7fffdc002598: 0x555555781a0f <ThreadOpenConnections(void*)>, a=...) at /usr/include/boost/bind/bind.hpp:253
#10 0x00005555558dd2c8 in boost::_bi::bind_t<void, void (*)(void*), boost::_bi::list1<boost::_bi::value<void*> > >::operator() (this=0x7fffdc002598) at /usr/include/boost/bind/bind.hpp:893
#11 0x00005555558dd00f in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(void*), boost::_bi::list1<boost::_bi::value<void*> > > >::run (this=0x7fffdc0023e0) at /usr/include/boost/thread/detail/thread.hpp:116
#12 0x00007ffff75285d5 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
#13 0x00007ffff5dd16ba in start_thread (arg=0x7fffdb7fe700) at pthread_create.c:333
#14 0x00007ffff5b073dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 8 (Thread 0x7fffdbfff700 (LWP 16654)):
#0 0x00007ffff5a35428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007ffff5a3702a in __GI_abort () at abort.c:89
#2 0x00007ffff65957dd in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007ffff65936b6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007ffff6593701 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007ffff6593969 in __cxa_rethrow () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00005555558ca0c7 in PrintException (pex=0x7fffcc000a00, pszThread=0x555555a07e44 "ThreadOpenAddedConnections()") at util.cpp:1039
#7 0x0000555555782989 in ThreadOpenAddedConnections (parg=0x0) at net.cpp:1908
#8 0x00005555558dd521 in boost::_bi::list1<boost::_bi::value<void*> >::operator()<void (*)(void*), boost::_bi::list0> (this=0x7fffdc002190, f=@0x7fffdc002188: 0x555555782912 <ThreadOpenAddedConnections(void*)>, a=...) at /usr/include/boost/bind/bind.hpp:253
#9 0x00005555558dd2c8 in boost::_bi::bind_t<void, void (*)(void*), boost::_bi::list1<boost::_bi::value<void*> > >::operator() (this=0x7fffdc002188) at /usr/include/boost/bind/bind.hpp:893
#10 0x00005555558dd00f in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(void*), boost::_bi::list1<boost::_bi::value<void*> > > >::run (this=0x7fffdc001fd0) at /usr/include/boost/thread/detail/thread.hpp:116
#11 0x00007ffff75285d5 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
#12 0x00007ffff5dd16ba in start_thread (arg=0x7fffdbfff700) at pthread_create.c:333
#13 0x00007ffff5b073dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 7 (Thread 0x7fffe880f700 (LWP 16652)):
#0 0x00007ffff5a35428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007ffff5a3702a in __GI_abort () at abort.c:89
#2 0x00007ffff659584d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007ffff65936b6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007ffff6593701 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007ffff6593969 in __cxa_rethrow () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00005555558ca0c7 in PrintException (pex=0x7fffd4000a00, pszThread=0x555555a0746a "ThreadSocketHandler()") at util.cpp:1039
#7 0x000055555577e81c in ThreadSocketHandler (parg=0x0) at net.cpp:985
#8 0x00005555558dd521 in boost::_bi::list1<boost::_bi::value<void*> >::operator()<void (*)(void*), boost::_bi::list0> (this=0x7fffdc001da0, f=@0x7fffdc001d98: 0x55555577e7a5 <ThreadSocketHandler(void*)>, a=...) at /usr/include/boost/bind/bind.hpp:253
#9 0x00005555558dd2c8 in boost::_bi::bind_t<void, void (*)(void*), boost::_bi::list1<boost::_bi::value<void*> > >::operator() (this=0x7fffdc001d98) at /usr/include/boost/bind/bind.hpp:893
#10 0x00005555558dd00f in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(void*), boost::_bi::list1<boost::_bi::value<void*> > > >::run (this=0x7fffdc001be0) at /usr/include/boost/thread/detail/thread.hpp:116
#11 0x00007ffff75285d5 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
#12 0x00007ffff5dd16ba in start_thread (arg=0x7fffe880f700) at pthread_create.c:333
#13 0x00007ffff5b073dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 5 (Thread 0x7fffe9811700 (LWP 16650)):
#0 __memset_sse2 () at ../sysdeps/x86_64/multiarch/../memset.S:93
#1 0x00005555555d9208 in std::__fill_a<unsigned char> (__first=0x7fffe00203d0 "", __last=0x7fffe00247d0 "", __c=@0x7fffe9810617: 0 '\000') at /usr/include/c++/5/bits/stl_algobase.h:723
#2 0x00005555555d9261 in std::__fill_n_a<unsigned long, unsigned char> (__first=0x7fffe00203d0 "", __n=17408, __c=@0x7fffe9810617: 0 '\000') at /usr/include/c++/5/bits/stl_algobase.h:779
#3 0x00005555555cf123 in std::fill_n<unsigned char*, unsigned long, unsigned char> (__first=0x7fffe00203d0 "", __n=17408, __value=@0x7fffe9810617: 0 '\000') at /usr/include/c++/5/bits/stl_algobase.h:805
#4 0x00005555555e19e8 in std::__uninitialized_default_n_1<true>::__uninit_default_n<unsigned char*, unsigned long> (__first=0x7fffe00203d0 "", __n=17408) at /usr/include/c++/5/bits/stl_uninitialized.h:540
#5 0x00005555555d8aca in std::__uninitialized_default_n<unsigned char*, unsigned long> (__first=0x7fffe00203d0 "", __n=17408) at /usr/include/c++/5/bits/stl_uninitialized.h:575
#6 0x00005555555ce67e in std::__uninitialized_default_n_a<unsigned char*, unsigned long, unsigned char> (__first=0x7fffe00203d0 "", __n=17408) at /usr/include/c++/5/bits/stl_uninitialized.h:637
#7 0x00005555555c45f6 in std::vector<unsigned char, std::allocator<unsigned char> >::_M_default_initialize (this=0x7fffe0016c78, __n=17408) at /usr/include/c++/5/bits/stl_vector.h:1311
#8 0x00005555555b8af8 in std::vector<unsigned char, std::allocator<unsigned char> >::vector (this=0x7fffe0016c78, __n=17408, __a=...) at /usr/include/c++/5/bits/stl_vector.h:279
#9 0x00005555555b6552 in boost::asio::ssl::detail::stream_core::stream_core (this=0x7fffe0016be8, context=0x7fffe0001140, io_service=...) at /usr/include/boost/asio/ssl/detail/stream_core.hpp:52
#10 0x00005555555c08c3 in boost::asio::ssl::stream<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> > >::stream<boost::asio::io_service> (this=0x7fffe0016bc8, arg=..., ctx=...)
at /usr/include/boost/asio/ssl/stream.hpp:109
#11 0x00005555555ca4e8 in AcceptedConnectionImpl<boost::asio::ip::tcp>::AcceptedConnectionImpl (this=0x7fffe0016ba0, io_service=..., context=..., fUseSSL=false) at bitcoinrpc.cpp:641
#12 0x00005555555a3495 in RPCListen<boost::asio::ip::tcp, boost::asio::socket_acceptor_service<boost::asio::ip::tcp> > (acceptor=..., context=..., fUseSSL=false) at bitcoinrpc.cpp:718
---Type <return> to continue, or q <return> to quit---
#13 0x000055555559e105 in ThreadRPCServer2 (parg=0x0) at bitcoinrpc.cpp:879
#14 0x000055555559cead in ThreadRPCServer (parg=0x0) at bitcoinrpc.cpp:687
#15 0x00005555558dd521 in boost::_bi::list1<boost::_bi::value<void*> >::operator()<void (*)(void*), boost::_bi::list0> (this=0x55556204a4d0, f=@0x55556204a4c8: 0x55555559ce79 <ThreadRPCServer(void*)>, a=...) at /usr/include/boost/bind/bind.hpp:253
#16 0x00005555558dd2c8 in boost::_bi::bind_t<void, void (*)(void*), boost::_bi::list1<boost::_bi::value<void*> > >::operator() (this=0x55556204a4c8) at /usr/include/boost/bind/bind.hpp:893
#17 0x00005555558dd00f in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(void*), boost::_bi::list1<boost::_bi::value<void*> > > >::run (this=0x55556204a310) at /usr/include/boost/thread/detail/thread.hpp:116
#18 0x00007ffff75285d5 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
#19 0x00007ffff5dd16ba in start_thread (arg=0x7fffe9811700) at pthread_create.c:333
#20 0x00007ffff5b073dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 4 (Thread 0x7fffea012700 (LWP 16649)):
#0 clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:81
#1 0x00007ffff5dd014a in create_thread (pd=pd@entry=0x7fffdb7fe700, attr=attr@entry=0x7fffea011b80, stopped_start=<optimized out>, stopped_start@entry=false, stackaddr=<optimized out>, thread_ran=0x7fffea011b7f) at ../sysdeps/unix/sysv/linux/createthread.c:102
#2 0x00007ffff5dd1e84 in __pthread_create_2_1 (newthread=<optimized out>, attr=<optimized out>, start_routine=<optimized out>, arg=<optimized out>) at pthread_create.c:679
#3 0x00007ffff75272e9 in boost::thread::start_thread_noexcept() () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
#4 0x00005555556d5f48 in boost::thread::start_thread (this=0x7fffdc0023c0) at /usr/include/boost/thread/detail/thread.hpp:179
#5 0x00005555558d1878 in boost::thread::thread<void (*)(void*), void*> (this=0x7fffdc0023c0, f=0x555555781a0f <ThreadOpenConnections(void*)>, a1=0x0) at /usr/include/boost/thread/detail/thread.hpp:392
#6 0x00005555558cd40d in ThreadHandler::createThread (this=0x555555e030c0, pfn=0x555555781a0f <ThreadOpenConnections(void*)>, parg=0x0, tname="ThreadOpenConnections") at util.cpp:1633
#7 0x0000555555784c2d in StartNode (parg=0x0) at net.cpp:2304
#8 0x00005555558dd521 in boost::_bi::list1<boost::_bi::value<void*> >::operator()<void (*)(void*), boost::_bi::list0> (this=0x55556203d970, f=@0x55556203d968: 0x555555784726 <StartNode(void*)>, a=...) at /usr/include/boost/bind/bind.hpp:253
#9 0x00005555558dd2c8 in boost::_bi::bind_t<void, void (*)(void*), boost::_bi::list1<boost::_bi::value<void*> > >::operator() (this=0x55556203d968) at /usr/include/boost/bind/bind.hpp:893
#10 0x00005555558dd00f in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(void*), boost::_bi::list1<boost::_bi::value<void*> > > >::run (this=0x55556203d7b0) at /usr/include/boost/thread/detail/thread.hpp:116
#11 0x00007ffff75285d5 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
#12 0x00007ffff5dd16ba in start_thread (arg=0x7fffea012700) at pthread_create.c:333
#13 0x00007ffff5b073dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 3 (Thread 0x7fffea813700 (LWP 16642)):
#0 0x00007ffff5b01737 in mprotect () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007ffff5a8150d in grow_heap (diff=8192, h=0x7fffe4000000) at arena.c:493
#2 sysmalloc (nb=nb@entry=634736, av=av@entry=0x7fffe4000020) at malloc.c:2409
#3 0x00007ffff5a82743 in _int_malloc (av=av@entry=0x7fffe4000020, bytes=bytes@entry=634717) at malloc.c:3827
#4 0x00007ffff5a84184 in __GI___libc_malloc (bytes=634717) at malloc.c:2913
#5 0x00007ffff6593e78 in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007ffff6625499 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#7 0x00007ffff6626833 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_append(char const*, unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#8 0x00005555556ec710 in std::operator+<char, std::char_traits<char>, std::allocator<char> > (
__lhs="<client_state>\r\n<host_info>\r\n <timezone>0</timezone>\r\n <domain_name>ubuntu</domain_name>\r\n <ip_addr>127.0.1.1</ip_addr>\r\n <host_cpid>cabb9e79a98f6ca95b44d0379e700694</host_cpid>\r\n <p_nc"..., __rhs=" <name>bl_800.jpg</name>")
at /usr/include/c++/5/bits/basic_string.h:4787
#9 0x00005555556b94ca in getfilecontents (filename="/var/lib/boinc-client/client_state.xml") at main.cpp:4995
#10 0x00005555556c9d6e in HarvestCPIDs (cleardata=true) at main.cpp:7402
#11 0x00005555556cc103 in ThreadCPIDs () at main.cpp:7582
#12 0x000055555575ce37 in boost::_bi::list0::operator()<void (*)(), boost::_bi::list0> (this=0x555562018370, f=@0x555562018368: 0x5555556cc0cc <ThreadCPIDs()>) at /usr/include/boost/bind/bind.hpp:192
#13 0x000055555575cc8a in boost::_bi::bind_t<void, void (*)(), boost::_bi::list0>::operator() (this=0x555562018368) at /usr/include/boost/bind/bind.hpp:893
#14 0x000055555575cacf in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(), boost::_bi::list0> >::run (this=0x5555620181b0) at /usr/include/boost/thread/detail/thread.hpp:116
#15 0x00007ffff75285d5 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
#16 0x00007ffff5dd16ba in start_thread (arg=0x7fffea813700) at pthread_create.c:333
#17 0x00007ffff5b073dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 2 (Thread 0x7fffeb014700 (LWP 16641)):
#0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225
#1 0x00007ffff75287e2 in boost::this_thread::hiden::sleep_for(timespec const&) () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
#2 0x00005555558ce395 in boost::this_thread::sleep_for (ns=...) at /usr/include/boost/thread/pthread/thread_data.hpp:243
#3 0x00005555558d090c in boost::this_thread::sleep_for<long, boost::ratio<1l, 1000l> > (d=...) at /usr/include/boost/thread/v2/thread.hpp:112
#4 0x00005555558c71be in MilliSleep (n=500) at util.cpp:96
#5 0x000055555592ddd5 in ThreadFlushWalletDB (parg=0x555561f919d8) at walletdb.cpp:639
#6 0x00005555558dd521 in boost::_bi::list1<boost::_bi::value<void*> >::operator()<void (*)(void*), boost::_bi::list0> (this=0x555561f9e4d0, f=@0x555561f9e4c8: 0x55555592dcf3 <ThreadFlushWalletDB(void*)>, a=...) at /usr/include/boost/bind/bind.hpp:253
#7 0x00005555558dd2c8 in boost::_bi::bind_t<void, void (*)(void*), boost::_bi::list1<boost::_bi::value<void*> > >::operator() (this=0x555561f9e4c8) at /usr/include/boost/bind/bind.hpp:893
#8 0x00005555558dd00f in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(void*), boost::_bi::list1<boost::_bi::value<void*> > > >::run (this=0x555561f9e310) at /usr/include/boost/thread/detail/thread.hpp:116
#9 0x00007ffff75285d5 in ?? () from /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.58.0
#10 0x00007ffff5dd16ba in start_thread (arg=0x7fffeb014700) at pthread_create.c:333
#11 0x00007ffff5b073dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
Thread 1 (Thread 0x7ffff7fe3740 (LWP 16612)):
#0 0x00007ffff5a7fdb3 in _int_free (av=0x7ffff5dc4b20 <main_arena>, p=0x555561ec8140, have_lock=0) at malloc.c:3896
#1 0x00007ffff5a8453c in __GI___libc_free (mem=<optimized out>) at malloc.c:2968
#2 0x0000555555742adb in __gnu_cxx::new_allocator<std::_Rb_tree_node<uint256> >::deallocate (this=0x555561d5f530, __p=0x555561ec8150) at /usr/include/c++/5/ext/new_allocator.h:110
#3 0x0000555555736584 in std::allocator_traits<std::allocator<std::_Rb_tree_node<uint256> > >::deallocate (__a=..., __p=0x555561ec8150, __n=1) at /usr/include/c++/5/bits/alloc_traits.h:517
#4 0x0000555555721e52 in std::_Rb_tree<uint256, uint256, std::_Identity<uint256>, std::less<uint256>, std::allocator<uint256> >::_M_put_node (this=0x555561d5f530, __p=0x555561ec8150) at /usr/include/c++/5/bits/stl_tree.h:495
#5 0x0000555555706bb5 in std::_Rb_tree<uint256, uint256, std::_Identity<uint256>, std::less<uint256>, std::allocator<uint256> >::_M_drop_node (this=0x555561d5f530, __p=0x555561ec8150) at /usr/include/c++/5/bits/stl_tree.h:562
---Type <return> to continue, or q <return> to quit---
#6 0x00005555556f402d in std::_Rb_tree<uint256, uint256, std::_Identity<uint256>, std::less<uint256>, std::allocator<uint256> >::_M_erase (this=0x555561d5f530, __x=0x555561ec8150) at /usr/include/c++/5/bits/stl_tree.h:1614
#7 0x00005555556f400a in std::_Rb_tree<uint256, uint256, std::_Identity<uint256>, std::less<uint256>, std::allocator<uint256> >::_M_erase (this=0x555561d5f530, __x=0x555561e9dc10) at /usr/include/c++/5/bits/stl_tree.h:1612
#8 0x00005555556f400a in std::_Rb_tree<uint256, uint256, std::_Identity<uint256>, std::less<uint256>, std::allocator<uint256> >::_M_erase (this=0x555561d5f530, __x=0x555561df01f0) at /usr/include/c++/5/bits/stl_tree.h:1612
#9 0x00005555556f400a in std::_Rb_tree<uint256, uint256, std::_Identity<uint256>, std::less<uint256>, std::allocator<uint256> >::_M_erase (this=0x555561d5f530, __x=0x555561d74520) at /usr/include/c++/5/bits/stl_tree.h:1612
#10 0x00005555556f400a in std::_Rb_tree<uint256, uint256, std::_Identity<uint256>, std::less<uint256>, std::allocator<uint256> >::_M_erase (this=0x555561d5f530, __x=0x555561d5f700) at /usr/include/c++/5/bits/stl_tree.h:1612
#11 0x00005555556e537f in std::_Rb_tree<uint256, uint256, std::_Identity<uint256>, std::less<uint256>, std::allocator<uint256> >::~_Rb_tree (this=0x555561d5f530, __in_chrg=<optimized out>) at /usr/include/c++/5/bits/stl_tree.h:858
#12 0x00005555556d7a5d in std::set<uint256, std::less<uint256>, std::allocator<uint256> >::~set (this=0x555561d5f530, __in_chrg=<optimized out>) at /usr/include/c++/5/bits/stl_set.h:90
#13 0x0000555555748b15 in std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > >::~pair (this=0x555561d5f510, __in_chrg=<optimized out>)
at /usr/include/c++/5/bits/stl_pair.h:96
#14 0x0000555555748b63 in __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > > > >::destroy<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > > > (this=0x555555dca700 <mvCPIDBlockHashes[abi:cxx11]>, __p=0x555561d5f510) at /usr/include/c++/5/ext/new_allocator.h:124
#15 0x000055555573eeb2 in std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > > > > >::destroy<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > > > (__a=..., __p=0x555561d5f510) at /usr/include/c++/5/bits/alloc_traits.h:542
#16 0x0000555555730d04 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > > > >::_M_destroy_node (this=0x555555dca700 <mvCPIDBlockHashes[abi:cxx11]>, __p=0x555561d5f4f0)
at /usr/include/c++/5/bits/stl_tree.h:553
#17 0x000055555571a286 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > > > >::_M_drop_node (this=0x555555dca700 <mvCPIDBlockHashes[abi:cxx11]>, __p=0x555561d5f4f0)
at /usr/include/c++/5/bits/stl_tree.h:561
#18 0x000055555570deb7 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > > > >::_M_erase (this=0x555555dca700 <mvCPIDBlockHashes[abi:cxx11]>, __x=0x555561d5f4f0)
at /usr/include/c++/5/bits/stl_tree.h:1614
#19 0x000055555570de94 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > > > >::_M_erase (this=0x555555dca700 <mvCPIDBlockHashes[abi:cxx11]>, __x=0x55556176a100)
at /usr/include/c++/5/bits/stl_tree.h:1612
#20 0x000055555570de94 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > > > >::_M_erase (this=0x555555dca700 <mvCPIDBlockHashes[abi:cxx11]>, __x=0x5555618ffc00)
at /usr/include/c++/5/bits/stl_tree.h:1612
#21 0x000055555570de94 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > > > >::_M_erase (this=0x555555dca700 <mvCPIDBlockHashes[abi:cxx11]>, __x=0x555561763ff0)
at /usr/include/c++/5/bits/stl_tree.h:1612
#22 0x000055555570de94 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > > > >::_M_erase (this=0x555555dca700 <mvCPIDBlockHashes[abi:cxx11]>, __x=0x555561763bd0)
at /usr/include/c++/5/bits/stl_tree.h:1612
#23 0x00005555556f924d in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > > > >::~_Rb_tree (this=0x555555dca700 <mvCPIDBlockHashes[abi:cxx11]>, __in_chrg=<optimized out>)
at /usr/include/c++/5/bits/stl_tree.h:858
#24 0x000055555575c7fd in std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::set<uint256, std::less<uint256>, std::allocator<uint256> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::set<uint256, std::less<uint256>, std::allocator<uint256> > > > >::~map (this=0x555555dca700 <mvCPIDBlockHashes[abi:cxx11]>, __in_chrg=<optimized out>)
at /usr/include/c++/5/bits/stl_map.h:96
#25 0x00007ffff5a39ff8 in __run_exit_handlers (status=1, listp=0x7ffff5dc45f8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:82
#26 0x00007ffff5a3a045 in __GI_exit (status=<optimized out>) at exit.c:104
#27 0x00007ffff5a20837 in __libc_start_main (main=0x55555557eb90 <main(int, char**)>, argc=6, argv=0x7fffffffe048, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe038) at ../csu/libc-start.c:325
#28 0x000055555557da89 in _start ()@Foggyx420 any suggestions on how to approach fixing. I'm happy to hack on this, but there's a lot of context to pick up.
iFoggz
commented
6 years ago i will look around. i noticed a mac osx user had similar issue with bitcoin forked wallet. but nothing said about how it was resolved it just didnt happen after that point. will look at if anything was changed there
rgarver
commented
6 years ago I thought I had a "fix" by disabling debug mode, but it just changed the output and caused some deadlock:
Setting bpk to 9705630d47a01ed5ba53bd5422b22e6f
No neural network nodes online.Getting first projectFinished getting first projectStopping...
Stopping...POTENTIAL DEADLOCK DETECTED
Previous lock order was:
cs_main main.cpp:7672 (TRY)
(1) pto->cs_vSend main.cpp:7676 (TRY)
(2) cs_vNodes main.cpp:7727
Current lock order is:
(2) cs_vNodes net.cpp:1104
(1) pnode->cs_vSend net.cpp:1110 (TRY)
gridcoinresearchd: sync.cpp:97: void potential_deadlock_detected(const std::pair<void*, void*>&, const LockStack&, const LockStack&): Assertion `false' failed.
AbortedFixed with #835.
rgarver
commented
6 years ago Those errors were with the latest code. I'm doing a make clean and recompile, but I think this may be something else
iFoggz
commented
6 years ago did u fetch the latest from development branch? #835 was jsut merged not too long ago
rgarver
commented
6 years ago Yeah, I had. Rebuilding didn't change the outcome.
denravonska
commented
6 years ago Are you referring to the deadlock or the boost::thread exception errors?
rgarver
commented
6 years ago Hmm, I thought that they were the same error just getting caught at different levels. I guess I can't recreate the boost::thread issue and it's been replaced by the deadlock error.
I'm happy to open a new issue to track the new problem
rgarver
commented
6 years ago Nevermind this is covered by #779
I'm looking at commit
889adaa, but this has been happening for prior versions recently on the development branch. I'm building with the following commands on Ubuntu 16.04:The configure and make stages appear to run smoothly and running
test_gridcoinreported everything working fine. The wallet was created on the staging branch, but no other warnings show up.Here is the log from running
src/gridcoinresearchd -testnet -printtoconsole -debug=true -debugnet=1