search

moneroexamples / openmonero

Fully open sourced implementation of MyMonero backend
Other
163 stars 135 forks source link

Segmentation faults when stress testing #163

Open skubakdj opened 5 years ago

skubakdj commented 5 years ago

I've been stress testing OpenMonero to try and understand the max limits of (1) concurrent imports and (2) search threads. When testing either scenario, I've sporadically run into OM crashing. Here's a snippet grabbed while running 50 imports:

2019-07-26 14:49:02,753    I Current blockchain height: 1887004, pool size: 0 txs, no of TxSearch threads: 50
2019-07-26 14:49:13,727    I Current blockchain height: 1887004, pool size: 1 txs, no of TxSearch threads: 50
Segmentation fault (core dumped)
ubuntu@ip:~$ echo $?
139

No other errors seem to be produced and none of the system resources seem to be maxed out. Interestingly, I'm not always able to replicate under the same conditions -- spinning up 5000 search threads for (2) will sometimes run successfully for hours, while other times OM will crash after 20 minutes.

I'm running from commit 1e6c13f on the devel branch, synced to mainnet. Please let me know what I can do to help debug!

moneroexamples commented 5 years ago

Could you produce backtrace, by running OM through gdb? This should indicate where the segmentation fault occurs.

skubakdj commented 5 years ago

I was able to produce four distinct errors that crashed OM while stress testing. I think only the third is related to this issue specifically, but I wanted to include the others in case they're useful.

First: 

[OpenMonero]

2019-07-30 19:53:11,452 I Current blockchain height: 1890039, pool size: 12 txs, no of TxSearch threads: 200
2019-07-30 19:53:21,519 I Current blockchain height: 1890039, pool size: 12 txs, no of TxSearch threads: 200
2019-07-30 19:53:31,634 I Current blockchain height: 1890039, pool size: 12 txs, no of TxSearch threads: 200
2019-07-30 19:53:40,370 E Pinging mysql failed. Stoping mysql pinging thread.
2019-07-30 19:53:40,370 I Exiting Mysql ping thread loop.
# ERR: SQLException in /home/ubuntu/openmonero/src/db/MySqlAccounts.cpp(select) on line 372
# ERR: Lost connection to MySQL server during query
free(): invalid pointer
2019-07-30 23:37:30,190 I Request to finish the openmonero received
2019-07-30 23:37:30,202 I Stopping restbed service.
2019-07-30 23:37:30.204 E HTTP_CLIENT: Failed to SEND
2019-07-30 23:37:30.204 I Failed to invoke http request to  /getheight
2019-07-30 23:37:30,204 E Error getting blockchain height. 
2019-07-30 23:37:30,205 E rpc->get_current_height() failed!
Aborted (core dumped)
ubuntu@ip:~$ 

[gdb]

[New Thread 0x7f1a85ffb700 (LWP 6914)]
[New Thread 0x7f1a857fa700 (LWP 6915)]
[New Thread 0x7f1a84ff9700 (LWP 6916)]
[Thread 0x7f1ccb644700 (LWP 6682) exited]

Thread 8 "openmonero" received signal SIGABRT, Aborted.
[Switching to Thread 0x7f1ccae43700 (LWP 6683)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) 
Continuing.

Thread 8 "openmonero" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51  in ../sysdeps/unix/sysv/linux/raise.c
(gdb) 
Continuing.
Couldn't get registers: No such process.

Second:

[OpenMonero]

2019-08-01 16:30:15,077 I 42F2eY: address does not exists
2019-08-01 16:30:15,077 I create_if_notfound is false
2019-08-01 16:30:15,095 I 42F2eY: TxSearch thread created.
2019-08-01 16:30:15,232 E Pinging mysql failed. Stoping mysql pinging thread.
2019-08-01 16:30:15,232 I Exiting Mysql ping thread loop.
# ERR: SQLException in /home/ubuntu/openmonero/src/db/MySqlAccounts.cpp(select) on line 269
# ERR: Lost connection to MySQL server during query
free(): corrupted unsorted chunks
2019-08-01 16:30:15,372 I Request to finish the openmonero received
Aborted (core dumped)
ubuntu@ip:~$

[gdb omitted, no unusual output]

Third:

[OpenMonero]

2019-08-01 19:37:49,938 I Current blockchain height: 1891452, pool size: 29 txs, no of TxSearch threads: 50
2019-08-01 19:38:00,847 I Current blockchain height: 1891452, pool size: 31 txs, no of TxSearch threads: 50
2019-08-01 19:38:11,863 I Current blockchain height: 1891452, pool size: 31 txs, no of TxSearch threads: 50
2019-08-01 19:38:22,657 I Current blockchain height: 1891452, pool size: 32 txs, no of TxSearch threads: 50
2019-08-01 19:38:33,481 I Current blockchain height: 1891452, pool size: 32 txs, no of TxSearch threads: 50
2019-08-01 19:38:44,253 I Current blockchain height: 1891452, pool size: 34 txs, no of TxSearch threads: 50
Segmentation fault (core dumped)
ubuntu@ip:~$ 

[gdb]

[New Thread 0x7f3f83fff700 (LWP 3551)]
[New Thread 0x7f3f837fe700 (LWP 3552)]
[New Thread 0x7f3f82ffd700 (LWP 3553)]
[New Thread 0x7f3f827fc700 (LWP 3554)]

Thread 7 "openmonero" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f408322a700 (LWP 3427)]
__memset_avx2_erms () at ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S:141
141 ../sysdeps/x86_64/multiarch/memset-vec-unaligned-erms.S: No such file or directory.
(gdb)

Fourth:

[OpenMonero]

2019-08-01 20:51:34,859 I Current blockchain height: 1891485, pool size: 11 txs, no of TxSearch threads: 50
2019-08-01 20:51:44,968 I Current blockchain height: 1891485, pool size: 14 txs, no of TxSearch threads: 50
2019-08-01 20:51:55,031 I Current blockchain height: 1891485, pool size: 15 txs, no of TxSearch threads: 50
2019-08-01 20:51:58,588 E Pinging mysql failed. Stoping mysql pinging thread.
2019-08-01 20:51:58,588 I Exiting Mysql ping thread loop.
Segmentation fault (core dumped)
ubuntu@ip:~$ 

[gdb]

[New Thread 0x7f2be57fa700 (LWP 4590)]
[New Thread 0x7f2be4ff9700 (LWP 4591)]
[New Thread 0x7f2bbbfff700 (LWP 4592)]
[New Thread 0x7f2bbb7fe700 (LWP 4593)]
[Thread 0x7f2ccc8aa700 (LWP 4509) exited]

Thread 8 "openmonero" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f2cc7fff700 (LWP 4510)]
0x00007f40e44f9881 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
moneroexamples commented 5 years ago

Thanks will look. In case you get the error again in gdb, you need to call bt command for backtrace (https://ftp.gnu.org/old-gnu/Manuals/gdb/html_node/gdb_42.html)

Also, do you use OM pyton script from stress testing? https://github.com/moneroexamples/openmonero/tree/master/scripts I could try to reproduce it.

Just a stupid question, if you run 5000 search tasks, are you sure you are not running out of ram or mysql connections, as I see some errors ERR: Lost connection to MySQL server during query and Pinging mysql failed. Stoping mysql pinging thread..

skubakdj commented 5 years ago

Thanks will look. In case you get the error again in gdb, you need to call bt command for backtrace

Thank you for the suggestion! I hadn't used gdb before. I'll try to reproduce the error and provide some backtraces.

Also, do you use OM pyton script from stress testing?

I wrote a script in TypeScript for testing. I'd be happy to share it as a gist if that would help out.

Just a stupid question, if you run 5000 search tasks, are you sure you are not running out of ram or mysql connections, as I see some errors ERR: Lost connection to MySQL server during query and Pinging mysql failed. Stoping mysql pinging thread..

I initially ran out of mysql connections and had to up the max limit. I'm not so sure about the RAM usage though. I don't think the server was capped, but I can't yet confirm.

moneroexamples commented 5 years ago

The backtrace would be most helpful, as it should directly indicate line which results in the seg fault. For now its difficult for me to replicate this issue which makes it rather hard to fix.

If you can share the gist it could help out. This way I could run it with same set of parameters you have and try to pin point the issue.

skubakdj commented 5 years ago

My availability this week is rather poor, but I'll post the gist and backtraces as soon as possible!

skubakdj commented 5 years ago

Here's the gist, instructions to run are commented at the top:

https://gist.github.com/skubakdj/668bbc9f186213f17026163567566071

I was only able to recreate and obtain backtraces for the second and fourth errors described above.

Second:

[OpenMonero]

2019-08-12 15:24:20,453 I Current blockchain height: 1899288, pool size: 3 txs, no of TxSearch threads: 5000
2019-08-12 15:24:30,501 I Current blockchain height: 1899288, pool size: 3 txs, no of TxSearch threads: 5000
2019-08-12 15:24:40,549 I Current blockchain height: 1899288, pool size: 3 txs, no of TxSearch threads: 5000
2019-08-12 15:24:50,669 I Current blockchain height: 1899288, pool size: 3 txs, no of TxSearch threads: 5000
2019-08-12 15:24:51,154 E Pinging mysql failed. Stoping mysql pinging thread.
2019-08-12 15:24:51,154 I Exiting Mysql ping thread loop.
# ERR: SQLException in /home/ubuntu/openmonero/src/db/MySqlAccounts.cpp(select) on line 372
# ERR: Lost connection to MySQL server during query
free(): corrupted unsorted chunks
2019-08-12 15:29:45,685 I Request to finish the openmonero received
2019-08-12 15:29:45,685 I 479Ujr: search thread stopped.
2019-08-12 15:29:45,685 I 479Ujr: stopping the thread
2019-08-12 15:29:45,685 I 479Ujr: stopping the thread
2019-08-12 15:29:45,685 I 458Pec: search thread stopped.
2019-08-12 15:29:45,685 I 458Pec: stopping the thread
2019-08-12 15:29:45,685 I 458Pec: stopping the thread
2019-08-12 15:29:45,685 I 45VuVW: search thread stopped.
2019-08-12 15:29:45,685 I 45VuVW: stopping the thread
2019-08-12 15:29:45,685 I 45VuVW: stopping the thread
fish: './openmonero/build/openmonero -…' terminated by signal SIGABRT (Abort)

[gdb]

[New Thread 0x7faad2454700 (LWP 24121)]
[New Thread 0x7faad1c53700 (LWP 24122)]
[New Thread 0x7faad1452700 (LWP 24123)]
[Thread 0x7fb59bfff700 (LWP 19055) exited]

Thread 8 "openmonero" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fb59b7fe700 (LWP 19056)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51  ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007fc9b700d801 in __GI_abort () at abort.c:79
#2  0x00007fc9b7056897 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fc9b7183b9a "%s\n")
    at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007fc9b705d90a in malloc_printerr (str=str@entry=0x7fc9b71858e0 "free(): corrupted unsorted chunks") at malloc.c:5350
#4  0x00007fc9b7065346 in _int_free (have_lock=0, p=<optimized out>, av=0x7fc9b73b8c40 <main_arena>) at malloc.c:4318
#5  __GI___libc_free (mem=0x55616486d870) at malloc.c:3124
#6  0x00007fc9b7d895be in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#7  0x00007fc9b7d806a9 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#8  0x00007fc9b7d73264 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#9  0x00007fc9b7d2c8f3 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#10 0x00007fc9b7d2cc1c in mysql_close () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#11 0x00007fc9b7d3065a in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#12 0x00007fc9b7d30928 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#13 0x00007fc9b7d2ccf7 in mysql_send_query () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#14 0x00007fc9b7d2cd61 in mysql_real_query () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#15 0x00007fc9b833890e in mysqlpp::Query::use(char const*, unsigned long) () from /usr/lib/libmysqlpp.so.3
#16 0x00007fc9b8338d9f in mysqlpp::Query::use(mysqlpp::SQLTypeAdapter const&) () from /usr/lib/libmysqlpp.so.3
#17 0x00007fc9b8338f0f in mysqlpp::Query::use(mysqlpp::SQLQueryParms&) () from /usr/lib/libmysqlpp.so.3
#18 0x00007fc9b8338e0f in mysqlpp::Query::use(mysqlpp::SQLTypeAdapter const&) () from /usr/lib/libmysqlpp.so.3
#19 0x000055616343d918 in mysqlpp::Query::storein_sequence<std::vector<xmreg::XmrAccount, std::allocator<xmreg::XmrAccount> > > (this=0x7fb59b7fafe0, con=std::vector of length 0, capacity 0, s=...) at /usr/include/mysql++/query.h:756
#20 0x000055616343a029 in mysqlpp::Query::storein<xmreg::XmrAccount> (this=0x7fb59b7fafe0, 
    con=std::vector of length 0, capacity 0, s=...) at /usr/include/mysql++/query.h:909
#21 0x0000556163431884 in xmreg::MySqlAccounts::select (this=0x5561648bb260, 
---Type <return> to continue, or q <return> to quit---
    address="492LBXjNzqSNz9xNVUcdDajQaLfabGat6gDqHKx3uHXX3rqYS1w8aUAiV3z3fNNnewBAG76aFDVwehZ3M4FpC64a7uLDk1y", account=...)
    at /home/ubuntu/openmonero/src/db/MySqlAccounts.cpp:256
#22 0x0000556163475325 in xmreg::OpenMoneroRequests::login_and_start_search_thread (this=0x7faca93427d0, 
    xmr_address="492LBXjNzqSNz9xNVUcdDajQaLfabGat6gDqHKx3uHXX3rqYS1w8aUAiV3z3fNNnewBAG76aFDVwehZ3M4FpC64a7uLDk1y", 
    view_key="18297e223e4403f803b70c632b26443cbd9b83f61775f1f2727449ef5b9dab3d", acc=..., j_response=...)
    at /home/ubuntu/openmonero/src/OpenMoneroRequests.cpp:2004
#23 0x0000556163466992 in xmreg::OpenMoneroRequests::get_address_info (this=0x7faca93427d0, session=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<restbed::Session, std::allocator<restbed::Session>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<restbed::Session, std::allocator<restbed::Session>, (__gnu_cxx::_Lock_policy)2>'

std::shared_ptr<restbed::Session> (use count 19, weak count 1) = {...}, body=std::vector of length 187, capacity 187 = {...})
    at /home/ubuntu/openmonero/src/OpenMoneroRequests.cpp:489
#24 0x00005561633b04ac in std::__invoke_impl<void, void (xmreg::OpenMoneroRequests::*&)(std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&), xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&> (__f=
    @0x7faca93427b0: (void (xmreg::OpenMoneroRequests::*)(xmreg::OpenMoneroRequests * const, std::shared_ptr<restbed::Session>, const std::vector<unsigned char, std::allocator<unsigned char> > &)) 0x556163466008 <xmreg::OpenMoneroRequests::get_address_info(std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)>, __t=..., 
    __args#0=..., __args#1=std::vector of length 187, capacity 187 = {...}) at /usr/include/c++/7/bits/invoke.h:66
#25 0x00005561633ad708 in std::__invoke<void (xmreg::OpenMoneroRequests::*&)(std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&), xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&> (__fn=
    @0x7faca93427b0: (void (xmreg::OpenMoneroRequests::*)(xmreg::OpenMoneroRequests * const, std::shared_ptr<restbed::Session>, const std::vector<unsigned char, std::allocator<unsigned char> > &)) 0x556163466008 <xmreg::OpenMoneroRequests::get_address_info(std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)>, __args#0=..., 
    __args#1=..., __args#2=std::vector of length 187, capacity 187 = {...}) at /usr/include/c++/7/bits/invoke.h:95
#26 0x00005561633aa602 in std::_Function_handler<void (xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&), void (xmreg::OpenMoneroRequests::*)(std::shared_ptr<restbed::Sessi---Type <return> to continue, or q <return> to quit---
on>, std::vector<unsigned char, std::allocator<unsigned char> > const&)>::_M_invoke(std::_Any_data const&, xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>&&, std::vector<unsigned char, std::allocator<unsigned char> > const&) (
    __functor=..., __args#0=..., __args#1=..., __args#2=std::vector of length 187, capacity 187 = {...})
    at /usr/include/c++/7/bits/std_function.h:373
#27 0x0000556163499b06 in std::function<void (xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)>::operator()(xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&) const (this=0x7faca93427b0, __args#0=..., 
    __args#1=std::shared_ptr<restbed::Session> (empty) = {...}, __args#2=std::vector of length 187, capacity 187 = {...})
    at /usr/include/c++/7/bits/std_function.h:706
#28 0x0000556163495dfb in std::__invoke_impl<void, std::function<void (xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)>&, xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&>(std::__invoke_other, std::function<void (xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)>&, xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>&&, std::vector<unsigned char, std::allocator<unsigned char> > const&) (__f=..., __args#0=..., __args#1=..., __args#2=std::vector of length 187, capacity 187 = {...})
    at /usr/include/c++/7/bits/invoke.h:60
#29 0x0000556163492f14 in std::__invoke<std::function<void (xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)>&, xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&>(std::function<void (xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)>&, xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>&&, std::vector<unsigned char, std::allocator<unsigned char> > const&) (__fn=..., 
    __args#0=..., __args#1=..., __args#2=std::vector of length 187, capacity 187 = {...})
    at /usr/include/c++/7/bits/invoke.h:95
#30 0x000055616348c32d in std::_Bind<std::function<void (xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)> (xmreg::OpenMoneroRequests, std::_Placeholder<1>, std::_Placehol---Type <return> to continue, or q <return> to quit---
der<2>)>::__call<void, std::shared_ptr<restbed::Session>&&, std::vector<unsigned char, std::allocator<unsigned char> > const&, 0ul, 1ul, 2ul>(std::tuple<std::shared_ptr<restbed::Session>&&, std::vector<unsigned char, std::allocator<unsigned char> > const&>&&, std::_Index_tuple<0ul, 1ul, 2ul>) (this=0x7faca93427b0, __args=...) at /usr/include/c++/7/functional:467
#31 0x00005561634887ab in std::_Bind<std::function<void (xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)> (xmreg::OpenMoneroRequests, std::_Placeholder<1>, std::_Placeholder<2>)>::operator()<std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&, void>(std::shared_ptr<restbed::Session>&&, std::vector<unsigned char, std::allocator<unsigned char> > const&) (
    this=0x7faca93427b0, __args#0=..., __args#1=std::vector of length 187, capacity 187 = {...})
    at /usr/include/c++/7/functional:551
#32 0x0000556163481e31 in std::_Function_handler<void (std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&), std::_Bind<std::function<void (xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)> (xmreg::OpenMoneroRequests, std::_Placeholder<1>, std::_Placeholder<2>)> >::_M_invoke(std::_Any_data const&, std::shared_ptr<restbed::Session>&&, std::vector<unsigned char, std::allocator<unsigned char> > const&) (__functor=..., __args#0=..., __args#1=std::vector of length 187, capacity 187 = {...})
    at /usr/include/c++/7/bits/std_function.h:316
#33 0x00005561637fddaf in restbed::detail::SessionImpl::fetch_body(unsigned long, std::shared_ptr<restbed::Session>, std::function<void (std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)> const&) const ()
#34 0x00005561637d142e in restbed::Session::fetch(unsigned long, std::function<void (std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)> const&) ()
#35 0x0000556163460b2a in xmreg::handel_::operator() (this=0x7fb59305a250, session=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<restbed::Session, std::allocator<restbed::Session>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<restbed::Session, std::allocator<restbed::Session>, (__gnu_cxx::_Lock_policy)2>'

std::shared_ptr<restbed::Session> (use count 19, weak count 1) = {...})
    at /home/ubuntu/openmonero/src/OpenMoneroRequests.cpp:28
#36 0x0000556163482035 in std::_Function_handler<void (std::shared_ptr<restbed::Session>), xmreg::handel_>::_M_invoke(std::_Any_data const&, std::shared_ptr<restbed::Session>&&) (__functor=..., __args#0=...)
---Type <return> to continue, or q <return> to quit---
    at /usr/include/c++/7/bits/std_function.h:316
#37 0x00005561637f57bf in std::_Function_handler<void (std::shared_ptr<restbed::Session>), restbed::detail::ServiceImpl::router(std::shared_ptr<restbed::Session>) const::{lambda(std::shared_ptr<restbed::Session>)#1}::operator()(std::shared_ptr<restbed::Session>) const::{lambda(std::shared_ptr<restbed::Session>)#1}::operator()(std::shared_ptr<restbed::Session>) const::{lambda(std::shared_ptr<restbed::Session>)#1}>::_M_invoke(std::_Any_data const&, std::shared_ptr<restbed::Session>&&) ()
#38 0x00005561637f26dd in restbed::detail::rule_engine(std::shared_ptr<restbed::Session>, std::vector<std::shared_ptr<restbed::Rule>, std::allocator<std::shared_ptr<restbed::Rule> > > const&, std::function<void (std::shared_ptr<restbed::Session>)> const&, unsigned long) ()
#39 0x00005561637f7065 in std::_Function_handler<void (std::shared_ptr<restbed::Session>), restbed::detail::ServiceImpl::router(std::shared_ptr<restbed::Session>) const::{lambda(std::shared_ptr<restbed::Session>)#1}>::_M_invoke(std::_Any_data const&, std::shared_ptr<restbed::Session>&&) ()
#40 0x00005561637f26dd in restbed::detail::rule_engine(std::shared_ptr<restbed::Session>, std::vector<std::shared_ptr<restbed::Rule>, std::allocator<std::shared_ptr<restbed::Rule> > > const&, std::function<void (std::shared_ptr<restbed::Session>)> const&, unsigned long) ()
#41 0x00005561637f2c17 in restbed::detail::ServiceImpl::router(std::shared_ptr<restbed::Session>) const ()
#42 0x00005561637fc763 in std::_Function_handler<void (std::shared_ptr<restbed::Session>), std::_Bind<void (restbed::detail::ServiceImpl::*(restbed::detail::ServiceImpl const*, std::_Placeholder<1>))(std::shared_ptr<restbed::Session>) const> >::_M_invoke(std::_Any_data const&, std::shared_ptr<restbed::Session>&&) ()
#43 0x00005561637df37c in restbed::SessionManager::save(std::shared_ptr<restbed::Session>, std::function<void (std::shared_ptr<restbed::Session>)> const&) ()
#44 0x00005561637efb95 in restbed::detail::ServiceImpl::authenticate(std::shared_ptr<restbed::Session>) const ()
#45 0x00005561637f898b in restbed::detail::ServiceImpl::parse_request(std::error_code const&, unsigned long, std::shared_ptr<restbed::Session>) const ()
#46 0x00005561637fc6be in std::_Function_handler<void (std::error_code const&, unsigned long), std::_Bind<void (restbed::detail::ServiceImpl::*(restbed::detail::ServiceImpl const*, std::_Placeholder<1>, std::_Placeholder<2>, std::shared_ptr<restbed::Se---Type <return> to continue, or q <return> to quit---
ssion>))(std::error_code const&, unsigned long, std::shared_ptr<restbed::Session>) const> >::_M_invoke(std::_Any_data const&, std::error_code const&, unsigned long&&) ()
#47 0x00005561637e64d1 in asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp>, asio::basic_streambuf_ref<std::allocator<char> >, asio::detail::wrapped_handler<asio::io_context::strand, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}, asio::detail::is_continuation_if_running> >::operator()(std::error_code const&, unsigned long, int)
    ()
#48 0x00005561637e6d2b in asio::detail::completion_handler<asio::detail::rewrapped_handler<asio::detail::binder2<asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp>, asio::basic_streambuf_ref<std::allocator<char> >, asio::detail::wrapped_handler<asio::io_context::strand, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}, asio::detail::is_continuation_if_running> >, std::error_code, unsigned long>, {lambda(std::error_code const&, unsigned long)#1}> >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) ()
#49 0x00005561637e73bf in asio::detail::reactive_socket_recv_op<asio::mutable_buffers_1, asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp>, asio::basic_streambuf_ref<std::allocator<char> >, asio::detail::wrapped_handler<asio::io_context::strand, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}, asio::detail::is_continuation_if_running> > >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) ()
#50 0x00005561637ea655 in asio::detail::epoll_reactor::descriptor_state::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) ()
#51 0x00005561637c65f0 in restbed::Service::start(std::shared_ptr<restbed::Settings const> const&) ()
#52 0x000055616339ac20 in <lambda()>::operator()(void) const (__closure=0x55616489ec18)
---Type <return> to continue, or q <return> to quit---
    at /home/ubuntu/openmonero/main.cpp:339
#53 0x000055616339ec8b in std::__invoke_impl<void, main(int, char const**)::<lambda()> >(std::__invoke_other, <lambda()> &&)
    (__f=...) at /usr/include/c++/7/bits/invoke.h:60
#54 0x000055616339e6ac in std::__invoke<main(int, char const**)::<lambda()> >(<lambda()> &&) (__fn=...)
    at /usr/include/c++/7/bits/invoke.h:95
#55 0x000055616339f8ac in std::thread::_Invoker<std::tuple<main(int, char const**)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x55616489ec18) at /usr/include/c++/7/thread:234
#56 0x000055616339f7d8 in std::thread::_Invoker<std::tuple<main(int, char const**)::<lambda()> > >::operator()(void) (
    this=0x55616489ec18) at /usr/include/c++/7/thread:243
#57 0x000055616339f768 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main(int, char const**)::<lambda()> > > >::_M_run(void) (this=0x55616489ec10) at /usr/include/c++/7/thread:186
#58 0x00007fc9b7a3166f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#59 0x00007fc9b976d6db in start_thread (arg=0x7fb59b7fe700) at pthread_create.c:463
#60 0x00007fc9b70ee88f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)

Fourth:

[OpenMonero]

2019-08-12 13:54:08,933 I 46UVew: TxSearch thread created.
2019-08-12 13:54:08,950 I 46kuGZ: TxSearch thread created.
2019-08-12 13:54:08,970 I 4B1jS3: TxSearch thread created.
2019-08-12 13:54:09,065 I 41j9tT: TxSearch thread created.
2019-08-12 13:54:09,081 I 446LoF: TxSearch thread created.
2019-08-12 13:54:14,140 I Current blockchain height: 1899239, pool size: 30 txs, no of TxSearch threads: 5000
2019-08-12 13:54:24,216 I Current blockchain height: 1899239, pool size: 30 txs, no of TxSearch threads: 5000
2019-08-12 13:54:34,288 I Current blockchain height: 1899239, pool size: 30 txs, no of TxSearch threads: 5000
2019-08-12 13:54:44,367 I Current blockchain height: 1899239, pool size: 32 txs, no of TxSearch threads: 5000
2019-08-12 13:54:54,423 I Current blockchain height: 1899240, pool size: 6 txs, no of TxSearch threads: 5000
2019-08-12 13:55:08,389 I Current blockchain height: 1899240, pool size: 6 txs, no of TxSearch threads: 5000
2019-08-12 13:55:13,729 E Pinging mysql failed. Stoping mysql pinging thread.
2019-08-12 13:55:13,729 I Exiting Mysql ping thread loop.
fish: './openmonero/build/openmonero -…' terminated by signal SIGSEGV (Address boundary error)
ubuntu@ip ~>

[gdb]

[New Thread 0x7f2805442700 (LWP 13826)]
[New Thread 0x7f2804c41700 (LWP 13827)]
[New Thread 0x7f2804440700 (LWP 13828)]
[Thread 0x7f32d2e64700 (LWP 3654) exited]

Thread 8 "openmonero" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f32d2663700 (LWP 3655)]
0x00007f46eaac5bdd in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
(gdb) bt
#0  0x00007f46eaac5bdd in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#1  0x00007f46eaab01f6 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#2  0x00007f46eaab7a8c in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#3  0x00007f46eaaa8f2b in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#4  0x00007f46eaa8f94b in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#5  0x00007f46eaa5dcea in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#6  0x00007f46eaa5dfff in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#7  0x00007f46eaa5efa8 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#8  0x00007f46eaa60d71 in mysql_real_query () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
#9  0x00007f46eb06c90e in mysqlpp::Query::use(char const*, unsigned long) () from /usr/lib/libmysqlpp.so.3
#10 0x00007f46eb06cd9f in mysqlpp::Query::use(mysqlpp::SQLTypeAdapter const&) () from /usr/lib/libmysqlpp.so.3
#11 0x00007f46eb06cf0f in mysqlpp::Query::use(mysqlpp::SQLQueryParms&) () from /usr/lib/libmysqlpp.so.3
#12 0x00007f46eb06ce0f in mysqlpp::Query::use(mysqlpp::SQLTypeAdapter const&) () from /usr/lib/libmysqlpp.so.3
#13 0x00005557e8775918 in mysqlpp::Query::storein_sequence<std::vector<xmreg::XmrAccount, std::allocator<xmreg::XmrAccount> > > (this=0x7f32d2660070, con=std::vector of length 0, capacity 0, s=...) at /usr/include/mysql++/query.h:756
#14 0x00005557e8772029 in mysqlpp::Query::storein<xmreg::XmrAccount> (this=0x7f32d2660070, 
    con=std::vector of length 0, capacity 0, s=...) at /usr/include/mysql++/query.h:909
#15 0x00005557e8769884 in xmreg::MySqlAccounts::select (this=0x5557e941b1e0, 
    address="47uWiU7CJ7EAqfunzaDYZ5hsiR65VAKvr8YZTAG5ivfgHsw3JZoxpp2UEVyXeg6jfNMD6LqTS6PLBUh3f8F8aoJG4JArheX", account=...)
    at /home/ubuntu/openmonero/src/db/MySqlAccounts.cpp:256
#16 0x00005557e87ad325 in xmreg::OpenMoneroRequests::login_and_start_search_thread (this=0x7f29e60f9920, 
    xmr_address="47uWiU7CJ7EAqfunzaDYZ5hsiR65VAKvr8YZTAG5ivfgHsw3JZoxpp2UEVyXeg6jfNMD6LqTS6PLBUh3f8F8aoJG4JArheX", 
    view_key="e8634326e9e8fba2d0610b36a2127178d192109607e188a58ce40934ad42791d", acc=..., j_response=...)
    at /home/ubuntu/openmonero/src/OpenMoneroRequests.cpp:2004
#17 0x00005557e879e992 in xmreg::OpenMoneroRequests::get_address_info (this=0x7f29e60f9920, session=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<restbed::Session, std::allocator<restbed::Session>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<restbed::Session, std::allocator<restbed::Session>, (__gnu_cxx::_Lock_policy)2>'

---Type <return> to continue, or q <return> to quit---
std::shared_ptr<restbed::Session> (use count 19, weak count 1) = {...}, body=std::vector of length 187, capacity 187 = {...})
    at /home/ubuntu/openmonero/src/OpenMoneroRequests.cpp:489
#18 0x00005557e86e84ac in std::__invoke_impl<void, void (xmreg::OpenMoneroRequests::*&)(std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&), xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&> (__f=
    @0x7f29e60f9900: (void (xmreg::OpenMoneroRequests::*)(xmreg::OpenMoneroRequests * const, std::shared_ptr<restbed::Session>, const std::vector<unsigned char, std::allocator<unsigned char> > &)) 0x5557e879e008 <xmreg::OpenMoneroRequests::get_address_info(std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)>, __t=..., 
    __args#0=..., __args#1=std::vector of length 187, capacity 187 = {...}) at /usr/include/c++/7/bits/invoke.h:66
#19 0x00005557e86e5708 in std::__invoke<void (xmreg::OpenMoneroRequests::*&)(std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&), xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&> (__fn=
    @0x7f29e60f9900: (void (xmreg::OpenMoneroRequests::*)(xmreg::OpenMoneroRequests * const, std::shared_ptr<restbed::Session>, const std::vector<unsigned char, std::allocator<unsigned char> > &)) 0x5557e879e008 <xmreg::OpenMoneroRequests::get_address_info(std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)>, __args#0=..., 
    __args#1=..., __args#2=std::vector of length 187, capacity 187 = {...}) at /usr/include/c++/7/bits/invoke.h:95
#20 0x00005557e86e2602 in std::_Function_handler<void (xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&), void (xmreg::OpenMoneroRequests::*)(std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)>::_M_invoke(std::_Any_data const&, xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>&&, std::vector<unsigned char, std::allocator<unsigned char> > const&) (
    __functor=..., __args#0=..., __args#1=..., __args#2=std::vector of length 187, capacity 187 = {...})
    at /usr/include/c++/7/bits/std_function.h:373
#21 0x00005557e87d1b06 in std::function<void (xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)>::operator()(xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&) const (this=0x7f29e60f9900, __args#0=..., 
---Type <return> to continue, or q <return> to quit---
    __args#1=std::shared_ptr<restbed::Session> (empty) = {...}, __args#2=std::vector of length 187, capacity 187 = {...})
    at /usr/include/c++/7/bits/std_function.h:706
#22 0x00005557e87cddfb in std::__invoke_impl<void, std::function<void (xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)>&, xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&>(std::__invoke_other, std::function<void (xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)>&, xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>&&, std::vector<unsigned char, std::allocator<unsigned char> > const&) (__f=..., __args#0=..., __args#1=..., __args#2=std::vector of length 187, capacity 187 = {...})
    at /usr/include/c++/7/bits/invoke.h:60
#23 0x00005557e87caf14 in std::__invoke<std::function<void (xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)>&, xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&>(std::function<void (xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)>&, xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>&&, std::vector<unsigned char, std::allocator<unsigned char> > const&) (__fn=..., 
    __args#0=..., __args#1=..., __args#2=std::vector of length 187, capacity 187 = {...})
    at /usr/include/c++/7/bits/invoke.h:95
#24 0x00005557e87c432d in std::_Bind<std::function<void (xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)> (xmreg::OpenMoneroRequests, std::_Placeholder<1>, std::_Placeholder<2>)>::__call<void, std::shared_ptr<restbed::Session>&&, std::vector<unsigned char, std::allocator<unsigned char> > const&, 0ul, 1ul, 2ul>(std::tuple<std::shared_ptr<restbed::Session>&&, std::vector<unsigned char, std::allocator<unsigned char> > const&>&&, std::_Index_tuple<0ul, 1ul, 2ul>) (this=0x7f29e60f9900, __args=...) at /usr/include/c++/7/functional:467
#25 0x00005557e87c07ab in std::_Bind<std::function<void (xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)> (xmreg::OpenMoneroRequests, std::_Placeholder<1>, std::_Placeholder<2>)>::operator()<std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&, void>(std::shared_ptr<restbed::Session>&&, std::vector<unsigned char, std::allocator<unsigned char> > const&) (
---Type <return> to continue, or q <return> to quit---
    this=0x7f29e60f9900, __args#0=..., __args#1=std::vector of length 187, capacity 187 = {...})
    at /usr/include/c++/7/functional:551
#26 0x00005557e87b9e31 in std::_Function_handler<void (std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&), std::_Bind<std::function<void (xmreg::OpenMoneroRequests&, std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)> (xmreg::OpenMoneroRequests, std::_Placeholder<1>, std::_Placeholder<2>)> >::_M_invoke(std::_Any_data const&, std::shared_ptr<restbed::Session>&&, std::vector<unsigned char, std::allocator<unsigned char> > const&) (__functor=..., __args#0=..., __args#1=std::vector of length 187, capacity 187 = {...})
    at /usr/include/c++/7/bits/std_function.h:316
#27 0x00005557e8b35daf in restbed::detail::SessionImpl::fetch_body(unsigned long, std::shared_ptr<restbed::Session>, std::function<void (std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)> const&) const ()
#28 0x00005557e8b0942e in restbed::Session::fetch(unsigned long, std::function<void (std::shared_ptr<restbed::Session>, std::vector<unsigned char, std::allocator<unsigned char> > const&)> const&) ()
#29 0x00005557e8798b2a in xmreg::handel_::operator() (this=0x7f29e60d42e0, session=warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<restbed::Session, std::allocator<restbed::Session>, (__gnu_cxx::_Lock_policy)2>'
warning: RTTI symbol not found for class 'std::_Sp_counted_ptr_inplace<restbed::Session, std::allocator<restbed::Session>, (__gnu_cxx::_Lock_policy)2>'

std::shared_ptr<restbed::Session> (use count 19, weak count 1) = {...})
    at /home/ubuntu/openmonero/src/OpenMoneroRequests.cpp:28
#30 0x00005557e87ba035 in std::_Function_handler<void (std::shared_ptr<restbed::Session>), xmreg::handel_>::_M_invoke(std::_Any_data const&, std::shared_ptr<restbed::Session>&&) (__functor=..., __args#0=...)
    at /usr/include/c++/7/bits/std_function.h:316
#31 0x00005557e8b2d7bf in std::_Function_handler<void (std::shared_ptr<restbed::Session>), restbed::detail::ServiceImpl::router(std::shared_ptr<restbed::Session>) const::{lambda(std::shared_ptr<restbed::Session>)#1}::operator()(std::shared_ptr<restbed::Session>) const::{lambda(std::shared_ptr<restbed::Session>)#1}::operator()(std::shared_ptr<restbed::Session>) const::{lambda(std::shared_ptr<restbed::Session>)#1}>::_M_invoke(std::_Any_data const&, std::shared_ptr<restbed::Session>&&) ()
#32 0x00005557e8b2a6dd in restbed::detail::rule_engine(std::shared_ptr<restbed::Session>, std::vector<std::shared_ptr<restbed::Rule>, std::allocator<std::shared_ptr<restbed::Rule> > > const&, std::function<void (std::shared_ptr<restbed::Session>)> cons---Type <return> to continue, or q <return> to quit---
t&, unsigned long) ()
#33 0x00005557e8b2f065 in std::_Function_handler<void (std::shared_ptr<restbed::Session>), restbed::detail::ServiceImpl::router(std::shared_ptr<restbed::Session>) const::{lambda(std::shared_ptr<restbed::Session>)#1}>::_M_invoke(std::_Any_data const&, std::shared_ptr<restbed::Session>&&) ()
#34 0x00005557e8b2a6dd in restbed::detail::rule_engine(std::shared_ptr<restbed::Session>, std::vector<std::shared_ptr<restbed::Rule>, std::allocator<std::shared_ptr<restbed::Rule> > > const&, std::function<void (std::shared_ptr<restbed::Session>)> const&, unsigned long) ()
#35 0x00005557e8b2ac17 in restbed::detail::ServiceImpl::router(std::shared_ptr<restbed::Session>) const ()
#36 0x00005557e8b34763 in std::_Function_handler<void (std::shared_ptr<restbed::Session>), std::_Bind<void (restbed::detail::ServiceImpl::*(restbed::detail::ServiceImpl const*, std::_Placeholder<1>))(std::shared_ptr<restbed::Session>) const> >::_M_invoke(std::_Any_data const&, std::shared_ptr<restbed::Session>&&) ()
#37 0x00005557e8b1737c in restbed::SessionManager::save(std::shared_ptr<restbed::Session>, std::function<void (std::shared_ptr<restbed::Session>)> const&) ()
#38 0x00005557e8b27b95 in restbed::detail::ServiceImpl::authenticate(std::shared_ptr<restbed::Session>) const ()
#39 0x00005557e8b3098b in restbed::detail::ServiceImpl::parse_request(std::error_code const&, unsigned long, std::shared_ptr<restbed::Session>) const ()
#40 0x00005557e8b346be in std::_Function_handler<void (std::error_code const&, unsigned long), std::_Bind<void (restbed::detail::ServiceImpl::*(restbed::detail::ServiceImpl const*, std::_Placeholder<1>, std::_Placeholder<2>, std::shared_ptr<restbed::Session>))(std::error_code const&, unsigned long, std::shared_ptr<restbed::Session>) const> >::_M_invoke(std::_Any_data const&, std::error_code const&, unsigned long&&) ()
#41 0x00005557e8b1e4d1 in asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp>, asio::basic_streambuf_ref<std::allocator<char> >, asio::detail::wrapped_handler<asio::io_context::strand, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}, asio::detail::is_continuation_if_running> >::operator()(std::error_code const&, unsigned long, int)
---Type <return> to continue, or q <return> to quit---
    ()
#42 0x00005557e8b1ed2b in asio::detail::completion_handler<asio::detail::rewrapped_handler<asio::detail::binder2<asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp>, asio::basic_streambuf_ref<std::allocator<char> >, asio::detail::wrapped_handler<asio::io_context::strand, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}, asio::detail::is_continuation_if_running> >, std::error_code, unsigned long>, {lambda(std::error_code const&, unsigned long)#1}> >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) ()
#43 0x00005557e8b1f3bf in asio::detail::reactive_socket_recv_op<asio::mutable_buffers_1, asio::detail::read_until_delim_string_op<asio::basic_stream_socket<asio::ip::tcp>, asio::basic_streambuf_ref<std::allocator<char> >, asio::detail::wrapped_handler<asio::io_context::strand, restbed::detail::SocketImpl::read(std::shared_ptr<asio::basic_streambuf<std::allocator<char> > > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<void (std::error_code const&, unsigned long)> const&)::{lambda(std::error_code const&, unsigned long)#1}, asio::detail::is_continuation_if_running> > >::do_complete(void*, asio::detail::scheduler_operation*, std::error_code const&, unsigned long) ()
#44 0x00005557e8afe5f0 in restbed::Service::start(std::shared_ptr<restbed::Settings const> const&) ()
#45 0x00005557e86d2c20 in <lambda()>::operator()(void) const (__closure=0x5557e93fee78)
    at /home/ubuntu/openmonero/main.cpp:339
#46 0x00005557e86d6c8b in std::__invoke_impl<void, main(int, char const**)::<lambda()> >(std::__invoke_other, <lambda()> &&)
    (__f=...) at /usr/include/c++/7/bits/invoke.h:60
#47 0x00005557e86d66ac in std::__invoke<main(int, char const**)::<lambda()> >(<lambda()> &&) (__fn=...)
    at /usr/include/c++/7/bits/invoke.h:95
#48 0x00005557e86d78ac in std::thread::_Invoker<std::tuple<main(int, char const**)::<lambda()> > >::_M_invoke<0>(std::_Index_tuple<0>) (this=0x5557e93fee78) at /usr/include/c++/7/thread:234
#49 0x00005557e86d77d8 in std::thread::_Invoker<std::tuple<main(int, char const**)::<lambda()> > >::operator()(void) (
    this=0x5557e93fee78) at /usr/include/c++/7/thread:243
---Type <return> to continue, or q <return> to quit---
#50 0x00005557e86d7768 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<main(int, char const**)::<lambda()> > > >::_M_run(void) (this=0x5557e93fee70) at /usr/include/c++/7/thread:186
#51 0x00007f46ea76566f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#52 0x00007f46ec4a16db in start_thread (arg=0x7f32d2663700) at pthread_create.c:463
#53 0x00007f46e9e2288f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) c
Continuing.
Couldn't get registers: No such process.
Couldn't get registers: No such process.
(gdb) [Thread 0x7f2804440700 (LWP 13828) exited]
[Thread 0x7f2804c41700 (LWP 13827) exited]
moneroexamples commented 5 years ago

I can experience the issues when there is large number of threads. All of them seem to be related to mysql connections, either dropping, disconnecting or running out of tcp/ip sockets.

I'm looking into it. But so far I haven't been able to pin point the cause. It seems as there is a combinations of issues related to mysql. Everytime I run strees test, I get different results. Sometimes it works, other times there are mysql connection errors, tcp/ip soccets problems, etc.

I keep looking into it. But if you have any more insights, please let me now. Any bit of info can help.

skubakdj commented 5 years ago

Thanks for taking a look! I'll add more info if I come across any.