Closed shermand100 closed 2 years ago
Thanks. I will look into it.
I'm having difficulties replicating the issue, since I don't have at present RPi available for tests. Does it occur on a pc as well or only on RPi?
Hi again sorry for the delay, I've been trying to have a bit less screen time recently.
I haven't tested and don't have a PC to try this on unfortunately. The error appears on Pi 3 and 4's so perhaps the 32-bit aspect of it is a factor. I have a 64-bit Odroid device I can try and replicate this on running Armbian Debian Buster and will be able to give some feedback in about a week.
No problem.
Hi again. It's been quite some time since I've visited this issue but unfortunately the issue is still there on a Raspberry Pi, now updated the the model/version 4. On looking up any block post randomx implementation at block 1978433 and up the block explorer crashes. I manage it with systemd which detects the failure but is unable to restart.
However following on from previous messages above the block explorer works fine on another single board device I now have that is also 32bit (Odroid XU4) so that rules out the issue being exclusive to or caused by 32bit.
That then leaves the next difference being the OS (Raspberry Pi OS vs Armbian Buster 20.08.01). So my next question is do you have a list of dependencies that are critical to the running of the explorer post randomx. I'll check through them to see if perhaps they're not included in Raspberry Pi OS?
( I install the default dependencies you link to on your project page sudo apt install git build-essential cmake libboost-all-dev miniupnpc libunbound-dev graphviz doxygen libunwind8-dev pkg-config libssl-dev libcurl4-openssl-dev libgtest-dev libreadline-dev libzmq3-dev libsodium-dev libhidapi-dev libhidapi-libusb0 )
Thanks for followup. Sadly, I'm not able to run the explorer on the RPi as currently don't have one.
@moneroexamples Hi, after all this time I have had this issue in mind and it is still present on Raspberry Pi 3/4 and the HardKernel Rock64 boards I have. All of those affected devices have the ARM Cortex A53 64-Bit Processor.
However the RockPro64 board I have works absolutely fine with exactly the same build (different processor).
Then, about a month ago I came across this:
https://github.com/monero-project/monero/commit/cf10e05cc6a0ed495dbdd44ec3a76b964b14edba
Where it has some Monero specific workarounds for the A53 processor with reference to an Arm Developer errata notice where:
835769: AArch64 multiply-accumulate instruction might produce incorrect result (page 20 of that document) and 843419: A load or store might access an incorrect address (page 22)
Unfortunately I've never dabbled in C much or made my own executable so am limited in seeing in how this is applied. Do you see that this may apply and if there is a simple tweak to the functions in this onion-monero-blockchain-explorer to mitigate it's effect?
@shermand100 I could add some of these flags to CMakeLists.txt
, but I can't test if they would resolve the issue. Do you get this error with master
or devel
branch of monero? Or both?
This is monero built with:
git clone --recursive https://github.com/monero-project/monero cd monero && git submodule init && git submodule update git checkout release-v0.17 git submodule sync && git submodule update USE_SINGLE_BUILDDIR=1 make
Explorer built with: git clone https://github.com/moneroexamples/onion-monero-blockchain-explorer.git cd onion-monero-blockchain-explorer mkdir build && cd build cmake .. make
1) Would it help if I built Monero with set(CMAKE_BUILD_TYPE RelWithDebInfo)
. I may get more useful GDB readout.
Can I do similar (if so how) for the explorer build to get more useful GDB on that too.
2) Longer term I can send you a RPi3B that I wouldn't need back if you like?
@shermand100 Thanks for the offer with RPI3B, but it won't be necessary. I was planning to get it anyway, just never actually got doing it. So maybe now its good time to do that finally.
I will add the flags later as now dealing with other issue related to the upcoming HF.
By the way, did you try running the explorer in docker on RPI3B? https://github.com/moneroexamples/onion-monero-blockchain-explorer#compiling-and-running-with-docker Not sure if this can help with architecture issues though?
I added the arm flags in arm_cmake
branch https://github.com/moneroexamples/onion-monero-blockchain-explorer/tree/arm_cmake. You can clone the branch:
git clone -b arm_cmake https://github.com/moneroexamples/onion-monero-blockchain-explorer.git
Thanks for doing that. I tried it out straight away and could see the extra flags were being triggered as part of the build which was great, everything compiled successfully. But the same issue exists :( I'll do some more debugging myself as I need to teach myself this, having a live example to work through should help. I'll also see if @radfish who I think first proposed those flags could shed any light.
I'll see if I can find a solution for after the hardfork.
This issue may still exist on Debian Buster/Bullseye but I have since switched to Ubuntu Server 20.04 LTS and 22.04 Server OS's which has solved the issue for me.
Thanks for your patience.
For others finding this I would recommend on: Raspberry Pi 3 using Ubuntu 22.04 server LTS 32bit Raspberry Pi 4 using Ubuntu 22.04 server LTS 64bit
Setup
explorer version (api): master-2020-05-31-f12b46c (1.1) | monero version: 0.15.0.5-17ec003c0
Raspberry Pi 4 with HDD on USB3, latest Raspberry OS.
Explorer started with:
taskset 1 ./xmrblocks --port 8081 --enable-pusher --enable-emission-monitor --deamon-url=HTTP://192.168.1.134:18081 --daemon-login user:pasword --mempool-info-timeout 60000 --mempool-refresh-time 30 --concurrency 1
Block explorer works fine when looking up blocks via http://192.168.1.134:8081/block/1978432 and lower.
Block explorer can lookup transactions of any block height.
Issue
Viewing http://192.168.1.134:8081/block/1978433 and higher causes a standard "page cannot be displayed" at front end.
Systemd shows the processes crashes.
Some further debugging/log using "gdb"
Successful request:
Error output:
(gdb) bt full
0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
1 0xb6520230 in __GI_abort () at abort.c:79
2 0xb677a8fc in __gnu_cxx::__verbose_terminate_handler() ()
from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6 No symbol table info available.
3 0xb67785b0 in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
No symbol table info available.
4 0xb6778624 in std::terminate() ()
from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6 No symbol table info available.
5 0xb67789dc in __cxa_rethrow ()
from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6 No symbol table info available.
6 0xb677a8b4 in __gnu_cxx::__verbose_terminate_handler() ()
from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6 No symbol table info available.
7 0xb67785b0 in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
No symbol table info available.
8 0xb6778624 in std::terminate() ()
from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6 No symbol table info available.
9 0xb6778990 in __cxa_throw ()
from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6 No symbol table info available.
10 0x0073766c in __cxa_throw ()
No symbol table info available.
11 0x00c71a6c in allocLargePagesMemory(unsigned int) ()
No symbol table info available.
12 0x00c65654 in randomx_alloc_cache ()
No symbol table info available.
13 0x00b899a8 in rx_slow_hash ()
No symbol table info available.
14 0x00b2a888 in cryptonote::get_block_longhash(cryptonote::Blockchain const*, cryptonote::block const&, crypto::hash&, unsigned long long, int) ()
No symbol table info available.
15 0x00b2aaa4 in cryptonote::get_block_longhash(cryptonote::Blockchain const*, cryptonote::block const&, unsigned long long, int) ()
No symbol table info available.
16 0x0078cd0c in xmreg::page::show_block[abi:cxx11](unsigned long long) ()
No symbol table info available.
17 0x0073f33c in main::{lambda(unsigned int)#4}::operator()(unsigned int) const ()
No symbol table info available.
18 0x00748bf4 in crow::TaggedRule::operator()<main::{lambda(unsigned int)#4}>(crow::black_magic::CallHelper&&)::{lambda(crow::request const&, crow::response&, unsigned long long)#1}::operator()(crow::request const, crow::response, unsigned long long) const ()
No symbol table info available.
19 0x0074e0f4 in std::_Function_handler<void (crow::request const&, crow::response&, unsigned long long), crow::TaggedRule::operator()<main::{lambda(unsigned int)#4}>(crow::black_magic::CallHelper&&)::{lambda(crow::request const&, crow::response&, unsigned long long)#1}>::_M_invoke(std::_Any_data c
No symbol table info available.
20 0x009cdc60 in std::function<void (crow::request const&, crow::response&, unsigned long long)>::operator()(crow::request const&, crow::response&, unsigned long long) const ()
No symbol table info available.
21 0x009cba18 in crow::detail::routing_handler_call_helper::call<crow::detail::routing_handler_call_helper::call_params<std::function<void (crow::request const&, crow::response&, unsigned long long)> >, 0, 1, 0, 0, crow::black_magic::S<>, crow::black_magic::S<crow::detail::routing_handler_call_helper::call_pair<unsigned long long, 0> > >::operator()(crow::detail::routing_handler_call_helper::call_params<std::function<void (crow::request const&, crow::response&, unsigned long long)> >) ()
No symbol table info available.
22 0x009c9ef8 in crow::detail::routing_handler_call_helper::call<crow::detail::routing_handler_call_helper::call_params<std::function<void (crow::request const&, crow::response&, unsigned long long)> >, 0, 0, 0, 0, crow::black_magic::S, crow::black_magic::S<> >::operator()(crow::detail::routing_handler_call_helper::call_params<std::function<void (crow::request const&, crow::response&, unsigned long long)> >) ()
No symbol table info available.
23 0x009bedc4 in crow::TaggedRule::handle(crow::request const&, crow::response&, crow::routing_params const&) ()
No symbol table info available.
24 0x00782284 in crow::Router::handle(crow::request const&, crow::response&) ()
No symbol table info available.
25 0x008e3d68 in crow::Crow<>::handle(crow::request const&, crow::response&) ()
No symbol table info available.
26 0x008cec1c in crow::Connection<crow::SocketAdaptor, crow::Crow<>>::handle() ()
No symbol table info available.
27 0x008b71c0 in crow::HTTPParser<crow::Connection<crow::SocketAdaptor, crow::Crow<>> >::process_message() ()
No symbol table info available.
28 0x0089e480 in crow::HTTPParser<crow::Connection<crow::SocketAdaptor, crow::Crow<>> >::on_message_complete(http_parser*) ()
No symbol table info available.
29 0x0077bfd8 in http_parser_execute ()
No symbol table info available.
30 0x00883d8c in crow::HTTPParser<crow::Connection<crow::SocketAdaptor, crow::Crow<>> >::feed(char const*, int) ()
No symbol table info available.
31 0x00869498 in crow::Connection<crow::SocketAdaptor, crow::Crow<>>::do_read()::{lambda(boost::system::error_code const&, unsigned int)#1}::operator()(boost::system::error_code const&, unsigned int) const ()
No symbol table info available.
32 0x00918f28 in boost::asio::detail::binder2<crow::Connection<crow::SocketAdaptor, crow::Crow<>>::do_read()::{lambda(boost::system::error_code const&, unsigned int)#1}, boost::system::error_code, unsigned int>::operator()() ()
No symbol table info available.
33 0x0090a4b0 in void boost::asio::asio_handler_invoke<boost::asio::detail::binder2<crow::Connection<crow::SocketAdaptor, crow::Crow<>>::do_read()::{lambda(boost::system::error_code const&, unsigned int)#1}, boost::system::error_code, unsigned int> >(boost::asio::detail::binder2<crow::Connection<crow::SocketAdaptor, crow::Crow<>>::do_read()::{lambda(boost::system::error_code const&, unsigned int)#1}, boost::system::error_code, unsigned int>&, ...) ()
No symbol table info available.
34 0x008f9dac in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder2<crow::Connection<crow::SocketAdaptor, crow::Crow<>>::do_read()::{lambda(boost::system::error_code const&, unsigned int)#1}, boost::system::error_code, unsigned int>, {lambda(boost::system::error_code const&, unsigned int)#1}>(boost::asio::detail::binder2<crow::Connection<crow::SocketAdaptor, crow::Crow<>>::do_read()::{lambda(boost::system::error_code const&, unsigned int)#1}, boost::system::error_code, unsigned int>&, {lambda(boost::system::error_code const&, unsigned int)#1}&) ()
No symbol table info available.
35 0x008e6998 in void boost::asio::detail::handler_work<crow::Connection<crow::SocketAdaptor, crow::Crow<>>::do_read()::{lambda(boost::system::error_code const&, unsigned int)#1}, boost::asio::system_executor>::complete<boost::asio::detail::binder2<{lambda(boost::system::error_code const&, unsigned int)#1}, boost::system::error_code, unsigned int> >(boost::asio::detail::binder2<{lambda(boost::system::error_code const&, unsigned int)#1}, boost::system::error_code, unsigned int>&, {lambda(boost::system::error_code const&, unsigned int)#1}&) ()
No symbol table info available.
36 0x008cf1a0 in boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, crow::Connection<crow::SocketAdaptor, crow::Crow<>>::do_read()::{lambda(boost::system::error_code const&, unsigned int)#1}>::do_complete(void, boost::asio::detail::scheduler_operation, boost::system::error_code const&, unsigned int) ()
No symbol table info available.
37 0x0075e200 in boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned int) ()
No symbol table info available.
38 0x00761b74 in boost::asio::detail::epoll_reactor::descriptor_state::do_complete(void, boost::asio::detail::scheduler_operation, boost::system::error_code const&, unsigned int) ()
No symbol table info available.
39 0x0075e200 in boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned int) ()
No symbol table info available.
40 0x007629ac in boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) ()
No symbol table info available.
41 0x00762234 in boost::asio::detail::scheduler::run(boost::system::error_code&) ()
No symbol table info available.
42 0x007630e0 in boost::asio::io_context::run() ()
No symbol table info available.
43 0x008051ec in crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}::operator()() const ()
No symbol table info available.
44 0x0084ae74 in void std::__invoke_impl<void, crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}>(std::__invoke_other, crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}&&) ()
No symbol table info available.
45 0x0082c338 in std::__invoke_result<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}>::type std::invoke<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}>(std::invoke_result&&, (crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}&&)...) ()
No symbol table info available.
46 0x00938d70 in decltype (__invoke((_S_declval<0u>)())) std::thread::_Invoker<std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}> >::_M_invoke<0u>(std::_Index_tuple<0u>) ()
No symbol table info available.
47 0x00934d3c in std::thread::_Invoker<std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}> >::operator()() ()
No symbol table info available.
48 0x0092ec8c in std::future_base::_Task_setter<std::unique_ptr<std::future_base::_Result, std::__future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}> >, void>::operator()() const ()
No symbol table info available.
49 0x0092919c in std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::future_base::_Result_base::_Deleter> (), std::future_base::_Task_setter<std::unique_ptr<std::future_base::_Result, std:: future_base::_Result_base::_Deleter>, std::thread::_Invoker<std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}> >, void> >::_M_invoke(std::_Any_data const&) ()
No symbol table info available.
50 0x007cd09c in std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const ()
No symbol table info available.
51 0x00771a60 in std::future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::future_base::_Result_base::_Deleter> ()>, bool) ()
No symbol table info available.
52 0x008136c0 in void std::invoke_impl<void, void (std::future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::future_base::_Result_base, std::future_base::_Result_base::_Deleter> ()>, bool), std::future_base::_State_baseV2*, std::function<std::unique_ptr<std::future_base::_Result_base, std::future_base::_Result_base::_Deleter> ()>, bool>(std::invoke_memfun_deref, void (std::future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::future_base::_Result_base, std::future_base::_Result_base::_Deleter> ()>, bool), std::future_base::_State_baseV2&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&&, bool*&&) ()
No symbol table info available.
53 0x007edd8c in std::invoke_result<void (std::future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::future_base::_Result_base, std::future_base::_Result_base::_Deleter> ()>, bool), std::future_base::_State_baseV2*, std::function<std::unique_ptr<std::future_base::_Result_base, std::future_base::_Result_base::_Deleter> ()>, bool>::type std::invoke<void (std::future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::future_base::_Result_base, std::future_base::_Result_base::_Deleter> ()>, bool), std::future_base::_State_baseV2, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool>(void (std::__future_base::_State_baseV2::&&)(std::function<std::unique_ptr<std::future_base::_Result_base, std::future_base::_Result_base::_Deleter> ()>, bool), std::future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::future_base::_Result_base::_Deleter> ()>&&, bool&&) ()
No symbol table info available.
54 0x007ccdbc in std::call_once<void (std::future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::future_base::_Result_base, std::future_base::_Result_base::_Deleter> ()>, bool), std::future_base::_State_baseV2, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool>(std::once_flag&, void (std::__future_base::_State_baseV2::&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::future_base::_Result_base::_Deleter> ()>, bool), std::future_base::_State_baseV2*&&, std::func--Type for more, q to quit, c to continue without paging--
tion<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&&, bool&&)::{lambda()#1}::operator()() const () No symbol table info available.
55 0x007ccdf4 in std::call_once<void (std::future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::future_base::_Result_base, std::future_base::_Result_base::_Deleter> ()>, bool), std::future_base::_State_baseV2, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool>(std::once_flag&, void (std::__future_base::_State_baseV2::&&)(std::function<std::unique_ptr<std::future_base::_Result_base, std::future_base::_Result_base::_Deleter> ()>, bool), std::future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::future_base::_Result_base::_Deleter> ()>&&, bool&&)::{lambda()#2}::operator()() const ()
No symbol table info available.
56 0x007cce14 in std::call_once<void (std::future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::future_base::_Result_base, std::future_base::_Result_base::_Deleter> ()>, bool), std::future_base::_State_baseV2, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool>(std::once_flag&, void (std::__future_base::_State_baseV2::&&)(std::function<std::unique_ptr<std::future_base::_Result_base, std::future_base::_Result_base::_Deleter> ()>, bool), std::future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::future_base::_Result_base::_Deleter> ()>&&, bool&&)::{lambda()#2}::_FUN() ()
No symbol table info available.
57 0xb6ed1158 in pthread_once_slow (once_control=0xe634b4, init_routine=0x734e40 <once_proxy@plt>) at pthread_once.c:116
58 0x0073ea54 in __gthread_once(int, void ()()) ()
No symbol table info available.
59 0x007cced4 in void std::call_once<void (std::future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::future_base::_Result_base, std::future_base::_Result_base::_Deleter> ()>, bool), std::future_base::_State_baseV2, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool>(std::once_flag&, void (std::__future_base::_State_baseV2::&&)(std::function<std::unique_ptr<std::future_base::_Result_base, std::future_base::_Result_base::_Deleter> ()>, bool), std::future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::future_base::_Result_base::_Deleter> ()>&&, bool&&) ()
No symbol table info available.
60 0x00771784 in std::future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::future_base::_Result_base::_Deleter> ()>, bool) ()
No symbol table info available.
61 0x00918664 in std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}> >, void>::_Async_state_impl(std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}>&&)::{lambda()#1}::operator()() const ()
No symbol table info available.
62 0x009293ec in void std::invoke_impl<void, std::future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}> >, void>::_Async_state_impl(std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}>&&)::{lambda()#1}>(std::invoke_other, std::future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}> >, void>::_Async_state_impl(std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}>&&)::{lambda()#1}&&) ()
No symbol table info available.
63 0x00922c88 in std::__invoke_result<std::future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}> >, void>::_Async_state_impl(std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}>&&)::{lambda()#1}>::type std::invoke<std::future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}> >, void>::_Async_state_impl(std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}>&&)::{lambda()#1}>(std::future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}> >, void>::_Async_state_impl(std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}>&&)::{lambda()#1}&&, (std::__invoke_result&&)...) ()
No symbol table info available.
64 0x009ca554 in decltype (__invoke((_S_declval<0u>)())) std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}> >, void>::_Async_state_impl(std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}>&&)::{lambda()#1}> >::_M_invoke<0u>(std::_Index_tuple<0u>) ()
No symbol table info available.
65 0x009c4c20 in std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}> >, void>::_Async_state_impl(std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}>&&)::{lambda()#1}> >::operator()() ()
No symbol table info available.
66 0x009b9be4 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::__future_base::_Async_state_impl<std::thread::_Invoker<std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}> >, void>::_Async_state_impl(std::tuple<crow::Server<crow::Crow<>, crow::SocketAdaptor>::run()::{lambda()#1}>&&)::{lambda()#1}> > >::_M_run() ()
No symbol table info available.
67 0xb67a39b0 in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
No symbol table info available.
68 0xb6ec8494 in start_thread (arg=0x9ccfedd0) at pthread_create.c:486
69 0xb65e0578 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /lib/arm-linux-gnueabihf/libc.so.6
No locals. Backtrace stopped: previous frame identical to this frame (corrupt stack?) (gdb)
That log may be overkill but copy/pasted as I had it to hand.