Memory retention discussion - not a bug (yet?)

[ghost]

So, looking through valgrind and dr_memory reports, I'm seeing no explicit memory leaks (all allocated memory is still reachable). However, I am seeing what looks like a memory retention/bloat issue with easylogging++. It seems that every time the logger is used to write, it instantiates a new el::Logger object, which calls el::Logger::configure(), eventually resulting in an el::Configurations::unsafeSet() allocation that gets pushed onto a list. Culprit seems to be here:

void Configurations::unsafeSet(Level level, ConfigurationType configurationType, const std::string& value) {
  Configuration* conf = RegistryWithPred<Configuration, Configuration::Predicate>::get(level, configurationType);
  if (conf == nullptr) {
    registerNew(new Configuration(level, configurationType, value)); 
  } else {
    conf->setValue(value);
  }

which does the push:

  virtual inline void registerNew(T_Ptr* ptr) ELPP_FINAL {
    this->list().push_back(ptr);
  }

I'm not sure if/when it ever gets popped back off of this stack. And this seems to happen repetitively. My guess is that there is some sort of cleanup that needs to be happening in the xcash/monero core code after each write (or somehow convincing easylogging++ not to instantiate a Logger object every single time).

For example, this MTRACE() macro from cryptonote_protocol_handler.inl seems to be one of the instigators:

  template<class t_core>
  bool t_cryptonote_protocol_handler<t_core>::kick_idle_peers()
  {
    MTRACE("Checking for idle peers...");
    std::vector<boost::uuids::uuid> kick_connections;

Still researching it, though...

[ghost]

An example valgrind finding from running xcashd for around 10 minutes. 63 times it lost 48 bytes (3,024 / 63) so it will be a fairly slow memory growth

==313226== 3,024 bytes in 63 blocks are still reachable in loss record 757 of 792
==313226==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==313226==    by 0x5A0FB55: el::Configurations::unsafeSet(el::Level, el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (easylogging++.cc:496)
==313226==    by 0x5A0FD6F: el::Configurations::unsafeSetGlobally(el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)::{lambda()#1}::operator()() const (easylogging++.cc:524)
==313226==    by 0x5A210DC: std::_Function_handler<bool (), el::Configurations::unsafeSetGlobally(el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (std_function.h:285)
==313226==    by 0x5A26869: std::function<bool ()>::operator()() const (std_function.h:688)
==313226==    by 0x5A0CBE4: el::LevelHelper::forEachLevel(unsigned int*, std::function<bool ()> const&) (easylogging++.cc:117)
==313226==    by 0x5A0FE2B: el::Configurations::unsafeSetGlobally(el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) (easylogging++.cc:523)
==313226==    by 0x5A0FBAE: el::Configurations::unsafeSet(el::Level, el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (easylogging++.cc:501)
==313226==    by 0x5A0D671: el::Configurations::set(el::Level, el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (easylogging++.cc:291)
==313226==    by 0x5A0D731: el::Configurations::set(el::Configuration*) (easylogging++.cc:301)
==313226==    by 0x5A0D3E9: el::Configurations::setFromBase(el::Configurations*) (easylogging++.cc:262)
==313226==    by 0x5A10925: el::Logger::configure(el::Configurations const&) (easylogging++.cc:606)
==313226==    by 0x5A10451: el::Logger::Logger(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, el::Configurations const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<std::basic_fstream<char, std::char_traits<char> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<std::basic_fstream<char, std::char_traits<char> > > > > >*) (easylogging++.cc:565)
==313226==    by 0x5A17A66: el::base::RegisteredLoggers::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) (easylogging++.cc:1847)
==313226==    by 0x5A1CC10: el::base::Writer::initializeLogger(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool) (easylogging++.cc:2573)
==313226==    by 0x5A1CA59: el::base::Writer::construct(int, char const*, ...) (easylogging++.cc:2565)
==313226==    by 0x4E22AC3: cryptonote::t_cryptonote_protocol_handler<cryptonote::core>::kick_idle_peers() (cryptonote_protocol_handler.inl:1304)
==313226==    by 0x4E21BD4: operator() (mem_fn_template.hpp:49)
==313226==    by 0x4E21BD4: operator()<bool, boost::_mfi::mf0<bool, cryptonote::t_cryptonote_protocol_handler<cryptonote::core> >, boost::_bi::list0> (bind.hpp:249)
==313226==    by 0x4E21BD4: operator() (bind.hpp:1294)
==313226==    by 0x4E21BD4: do_call<boost::_bi::bind_t<bool, boost::_mfi::mf0<bool, cryptonote::t_cryptonote_protocol_handler<cryptonote::core> >, boost::_bi::list1<boost::_bi::value<cryptonote::t_cryptonote_protocol_handler<cryptonote::core>*> > > > (math_helper.h:258)
==313226==    by 0x4E21BD4: cryptonote::t_cryptonote_protocol_handler<cryptonote::core>::on_idle() (cryptonote_protocol_handler.inl:1297)
==313226==    by 0x4E14AE5: operator() (mem_fn_template.hpp:49)
==313226==    by 0x4E14AE5: operator()<bool, boost::_mfi::mf0<bool, cryptonote::t_cryptonote_protocol_handler<cryptonote::core> >, boost::_bi::list0> (bind.hpp:249)
==313226==    by 0x4E14AE5: operator() (bind.hpp:1294)
==313226==    by 0x4E14AE5: epee::net_utils::boosted_tcp_server<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >::idle_callback_conext<boost::_bi::bind_t<bool, boost::_mfi::mf0<bool, cryptonote::t_cryptonote_protocol_handler<cryptonote::core> >, boost::_bi::list1<boost::_bi::value<cryptonote::t_cryptonote_protocol_handler<cryptonote::core>*> > > >::call_handler() (abstract_tcp_server2.h:263)
==313226==    by 0x4E596D1: epee::net_utils::boosted_tcp_server<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >::global_timer_handler(boost::shared_ptr<epee::net_utils::boosted_tcp_server<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >::idle_callback_conext_base>) (abstract_tcp_server2.h:280)

And another, this time from within tools::log_stack_trace() :

==313226== 3,024 bytes in 63 blocks are still reachable in loss record 765 of 792
==313226==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==313226==    by 0x5A0FB55: el::Configurations::unsafeSet(el::Level, el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (easylogging++.cc:496)
==313226==    by 0x5A0FD6F: el::Configurations::unsafeSetGlobally(el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)::{lambda()#1}::operator()() const (easylogging++.cc:524)
==313226==    by 0x5A210DC: std::_Function_handler<bool (), el::Configurations::unsafeSetGlobally(el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (std_function.h:285)
==313226==    by 0x5A26869: std::function<bool ()>::operator()() const (std_function.h:688)
==313226==    by 0x5A0CBE4: el::LevelHelper::forEachLevel(unsigned int*, std::function<bool ()> const&) (easylogging++.cc:117)
==313226==    by 0x5A0FE2B: el::Configurations::unsafeSetGlobally(el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) (easylogging++.cc:523)
==313226==    by 0x5A0FBAE: el::Configurations::unsafeSet(el::Level, el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (easylogging++.cc:501)
==313226==    by 0x5A0D671: el::Configurations::set(el::Level, el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (easylogging++.cc:291)
==313226==    by 0x5A0D731: el::Configurations::set(el::Configuration*) (easylogging++.cc:301)
==313226==    by 0x5A0D3E9: el::Configurations::setFromBase(el::Configurations*) (easylogging++.cc:262)
==313226==    by 0x5A10925: el::Logger::configure(el::Configurations const&) (easylogging++.cc:606)
==313226==    by 0x5A10451: el::Logger::Logger(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, el::Configurations const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<std::basic_fstream<char, std::char_traits<char> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<std::basic_fstream<char, std::char_traits<char> > > > > >*) (easylogging++.cc:565)
==313226==    by 0x5A17A66: el::base::RegisteredLoggers::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) (easylogging++.cc:1847)
==313226==    by 0x5A1CC10: el::base::Writer::initializeLogger(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool) (easylogging++.cc:2573)
==313226==    by 0x5A1CA59: el::base::Writer::construct(int, char const*, ...) (easylogging++.cc:2565)
==313226==    by 0x55117A9: tools::log_stack_trace(char const*) (stack_trace.cpp:133)
==313226==    by 0x5512C06: __cxa_throw (stack_trace.cpp:95)
==313226==    by 0x59261DC: ??? (in /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.71.0)
==313226==    by 0x489FC73: cryptonote::rpc::ZmqServer::serve() (zmq_server.cpp:87)

[zachhildreth]

Hi @CygnusMiner Great work on this!

If you can continue testing and find a code fix for the slow mem leak, we can do a bug bounty on this for you. Its because this is an important problem, and yes its a slow mem leak as it takes weeks for it to stall

[ghost]

Ok, I've been squirreled by a separate finding from valgrind (below) - an actual memory leak this time :) - 2MB per allocate.

Tracing it down, it seems to be occurring here:

    // this isn't supposed to happen, but guard against it for now.
    if(hp_state == NULL)
        slow_hash_allocate_state();

The commenter seems to think that the hp_state thread pointer should already have been allocated, but the only deliberate allocation I see for the pointer is in miner.cpp and in blockchain.cpp, in both cases they specifically free the allocation after they're done with it (miner.cpp and blockchain.cpp).

This allocation seems to occur per thread so it should be a limited memory leak (unless threads are destroyed and recreated often in here - I don't know the Monero/Xcash code that well).

However, limited or not it looks like the best approach would be to remove the conditional:

    // this isn't supposed to happen, but guard against it for now.
    if(hp_state == NULL)

explicitly allocating it, and then call the slow_hash_free_state() after you're done with hp_state pointer, somewhere around here

Also, as an aside and just to apease the Paranoid OCD in me, wouldn't it be prudent to memset the block prior to freeing after each use in encrypting? Just to reduce the chance of any accidental loss of sensitive data from a forced core dump?

---

For reference, one of the findings from Valgrind:

==401653== 4,194,304 bytes in 2 blocks are definitely lost in loss record 60 of 61
==401653==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==401653==    by 0x558CAB3: slow_hash_allocate_state (slow-hash.c:689)
==401653==    by 0x558CFB6: cn_slow_hash (slow-hash.c:771)
==401653==    by 0x52DEC02: cn_slow_hash (hash.h:75)
==401653==    by 0x52DEC02: cryptonote::get_block_longhash(cryptonote::block const&, crypto::hash&, unsigned long) (cryptonote_format_utils.cpp:1035)
==401653==    by 0x52DEC8D: cryptonote::get_block_longhash(cryptonote::block const&, unsigned long) (cryptonote_format_utils.cpp:1062)
==401653==    by 0x50D6EC8: cryptonote::Blockchain::handle_block_to_main_chain(cryptonote::block const&, crypto::hash const&, cryptonote::block_verification_context&) (blockchain.cpp:3460)
==401653==    by 0x50E08C9: cryptonote::Blockchain::add_new_block(cryptonote::block const&, cryptonote::block_verification_context&) (blockchain.cpp:4332)
==401653==    by 0x5115055: cryptonote::core::add_new_block(cryptonote::block const&, cryptonote::block_verification_context&) (cryptonote_core.cpp:1333)
==401653==    by 0x5125D9E: cryptonote::core::handle_incoming_block(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cryptonote::block_verification_context&, bool) (cryptonote_core.cpp:1380)
==401653==    by 0x4E70BC4: cryptonote::t_cryptonote_protocol_handler<cryptonote::core>::try_add_next_blocks(cryptonote::cryptonote_connection_context&) (cryptonote_protocol_handler.inl:1192)
==401653==    by 0x4E74C53: cryptonote::t_cryptonote_protocol_handler<cryptonote::core>::handle_response_get_objects(int, cryptonote::NOTIFY_RESPONSE_GET_OBJECTS::request&, cryptonote::cryptonote_connection_context&) (cryptonote_protocol_handler.inl:1057)
==401653==    by 0x2BF7E2: operator() (mem_fn_template.hpp:393)
==401653==    by 0x2BF7E2: operator()<int, boost::_mfi::mf3<int, cryptonote::t_cryptonote_protocol_handler<cryptonote::core>, int, cryptonote::NOTIFY_RESPONSE_GET_OBJECTS::request&, cryptonote::cryptonote_connection_context&>, boost::_bi::rrlist3<int&, boost::value_initialized<cryptonote::NOTIFY_RESPONSE_GET_OBJECTS::request>&, cryptonote::cryptonote_connection_context&> > (bind.hpp:453)
==401653==    by 0x2BF7E2: operator()<int&, boost::value_initialized<cryptonote::NOTIFY_RESPONSE_GET_OBJECTS::request>&, cryptonote::cryptonote_connection_context&> (bind.hpp:1330)
==401653==    by 0x2BF7E2: int epee::net_utils::buff_to_t_adapter<cryptonote::t_cryptonote_protocol_handler<cryptonote::core>, cryptonote::NOTIFY_RESPONSE_GET_OBJECTS::request, cryptonote::cryptonote_connection_context, boost::_bi::bind_t<int, boost::_mfi::mf3<int, cryptonote::t_cryptonote_protocol_handler<cryptonote::core>, int, cryptonote::NOTIFY_RESPONSE_GET_OBJECTS::request&, cryptonote::cryptonote_connection_context&>, boost::_bi::list4<boost::_bi::value<cryptonote::t_cryptonote_protocol_handler<cryptonote::core>*>, boost::arg<1>, boost::arg<2>, boost::arg<3> > > >(cryptonote::t_cryptonote_protocol_handler<cryptonote::core>*, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::_bi::bind_t<int, boost::_mfi::mf3<int, cryptonote::t_cryptonote_protocol_handler<cryptonote::core>, int, cryptonote::NOTIFY_RESPONSE_GET_OBJECTS::request&, cryptonote::cryptonote_connection_context&>, boost::_bi::list4<boost::_bi::value<cryptonote::t_cryptonote_protocol_handler<cryptonote::core>*>, boost::arg<1>, boost::arg<2>, boost::arg<3> > >, cryptonote::cryptonote_connection_context&) (levin_abstract_invoke2.h:216)
==401653==    by 0x2BFCE0: int cryptonote::t_cryptonote_protocol_handler<cryptonote::core>::handle_invoke_map<cryptonote::cryptonote_connection_context>(bool, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, cryptonote::cryptonote_connection_context&, bool&) (cryptonote_protocol_handler.h:89)
==401653==    by 0x2C0262: int nodetool::node_server<cryptonote::t_cryptonote_protocol_handler<cryptonote::core> >::handle_invoke_map<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> >(bool, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context>&, bool&) (net_node.h:148)
==401653==    by 0x2C0434: nodetool::node_server<cryptonote::t_cryptonote_protocol_handler<cryptonote::core> >::notify(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context>&) (net_node.h:136)
==401653==    by 0x4E494E7: epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> >::handle_recv(void const*, unsigned long) (levin_protocol_handler_async.h:483)
==401653==    by 0x4E9549A: epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >::handle_read(boost::system::error_code const&, unsigned long) (abstract_tcp_server2.inl:340)
==401653==    by 0x4E812B8: call<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > >, const boost::system::error_code, long unsigned int> (mem_fn_template.hpp:271)
==401653==    by 0x4E812B8: operator()<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > > (mem_fn_template.hpp:286)
==401653==    by 0x4E812B8: operator()<boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::rrlist2<const boost::system::error_code&, long unsigned int const&> > (bind.hpp:398)
==401653==    by 0x4E812B8: operator()<const boost::system::error_code&, long unsigned int const&> (bind.hpp:1318)
==401653==    by 0x4E812B8: operator() (bind_handler.hpp:164)
==401653==    by 0x4E812B8: asio_handler_invoke<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, long unsigned int> > (handler_invoke_hook.hpp:69)
==401653==    by 0x4E812B8: invoke<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > (handler_invoke_helpers.hpp:37)
==401653==    by 0x4E812B8: asio_handler_invoke<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, long unsigned int> (bind_handler.hpp:207)
==401653==    by 0x4E812B8: invoke<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, long unsigned int>, boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, long unsigned int> > (handler_invoke_helpers.hpp:37)
==401653==    by 0x4E812B8: void boost::asio::detail::strand_service::dispatch<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, unsigned long> >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, unsigned long>&) (strand_service.hpp:62)
==401653==    by 0x4E821F0: operator()<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, long unsigned int> > (io_context_strand.hpp:343)
==401653==    by 0x4E821F0: initiate<boost::asio::io_context::strand::initiate_dispatch, boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, long unsigned int>, boost::asio::io_context::strand*> (async_result.hpp:82)
==401653==    by 0x4E821F0: async_initiate<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, long unsigned int>, void(), boost::asio::io_context::strand::initiate_dispatch, boost::asio::io_context::strand*> (async_result.hpp:257)
==401653==    by 0x4E821F0: dispatch<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, long unsigned int> > (io_context_strand.hpp:189)
==401653==    by 0x4E821F0: operator()<boost::system::error_code, long unsigned int> (wrapped_handler.hpp:99)
==401653==    by 0x4E821F0: operator() (bind_handler.hpp:164)
==401653==    by 0x4E821F0: operator() (wrapped_handler.hpp:191)
==401653==    by 0x4E821F0: asio_handler_invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > > (handler_invoke_hook.hpp:69)
==401653==    by 0x4E821F0: invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > >, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > (handler_invoke_helpers.hpp:37)
==401653==    by 0x4E821F0: asio_handler_invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > >, boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > (wrapped_handler.hpp:275)
==401653==    by 0x4E821F0: invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > >, boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > > (handler_invoke_helpers.hpp:37)
==401653==    by 0x4E821F0: complete<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > > (handler_work.hpp:100)
==401653==    by 0x4E821F0: do_complete (completion_handler.hpp:70)
==401653==    by 0x4E821F0: void boost::asio::detail::strand_service::dispatch<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > >&) (strand_service.hpp:88)
==401653==    by 0x4E82C38: operator()<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > > (io_context_strand.hpp:343)
==401653==    by 0x4E82C38: initiate<boost::asio::io_context::strand::initiate_dispatch, boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > >, boost::asio::io_context::strand*> (async_result.hpp:82)
==401653==    by 0x4E82C38: async_initiate<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > >, void(), boost::asio::io_context::strand::initiate_dispatch, boost::asio::io_context::strand*> (async_result.hpp:257)
==401653==    by 0x4E82C38: dispatch<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > > (io_context_strand.hpp:189)
==401653==    by 0x4E82C38: asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running> (wrapped_handler.hpp:232)
==401653==    by 0x4E82C38: invoke<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running> > (handler_invoke_helpers.hpp:37)
==401653==    by 0x4E82C38: asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int> (bind_handler.hpp:207)
==401653==    by 0x4E82C38: invoke<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int> > (handler_invoke_helpers.hpp:37)
==401653==    by 0x4E82C38: dispatch<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, std::allocator<void> > (io_object_executor.hpp:119)
==401653==    by 0x4E82C38: complete<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int> > (handler_work.hpp:72)
==401653==    by 0x4E82C38: boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::asio::detail::io_object_executor<boost::asio::executor> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) (reactive_socket_recv_op.hpp:123)

[ghost]

Also Valgrind is upset that you're testing an uninitialized array in get_network_block_database_hash()

[zachhildreth]

Thanks for more work on this! I can go ahead and add the new find to the testnet and we can test it out in a few days. I will look more into that area as I am not so familiar with that particular section of that file.

For the memset I would think so, thats how dpops was built because of that risk, so I will need to look into that as well.

For the "xcash original" code part yes that makes sense looking at it now, so that would be good to fix as well.

Let keep this going with your findings on this page and I can reference them once we add them to the testnet

[ghost]

There's also an issue on exit which I experienced during memory testing. It seems that a mutex variable is getting destroyed at daemon shutdown, followed by an attempt to unlock using that mutex. The result's a thrown boost::lock_error exception from mlocker::unlock() call.

We're exiting the daemon at this point, so it's not a big deal, but it's causing an abort prior to clean-up which is messing around with the memory checkers.

I hacked my way around it by throwing away the exception with a try/catch wrapper (below) but that's hardly a valid fix.

  void mlocker::unlock(void *ptr, size_t len)
  {
    try {
      size_t page_size = get_page_size();
      if (page_size == 0)
        return;
      CRITICAL_REGION_LOCAL(mutex());
      const size_t first = ((uintptr_t)ptr) / page_size;
      const size_t last = (((uintptr_t)ptr) + len - 1) / page_size;
      for (size_t page = first; page <= last; ++page)
        unlock_page(page);
      --num_locked_objects;
    } catch (const boost::lock_error& e) { // Something's gone wrong - return without activity
      std::cerr << "\nFailed Unlock!\n";
      return;
    }
  }

Mostly just making note of it here as an item to track down how to properly address the use of the mlocker after it's been destroyed.

Valgrind output follows:

==331274== 240 bytes in 1 blocks are possibly lost in loss record 458 of 852
==331274==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==331274==    by 0x5B131F3: __cxa_allocate_exception (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==331274==    by 0x1F36D6: void boost::throw_exception<boost::lock_error>(boost::lock_error const&) (throw_exception.hpp:70)
==331274==    by 0x48AA321: lock (mutex.hpp:64)
==331274==    by 0x48AA321: critical_region_t (syncobj.h:134)
==331274==    by 0x48AA321: epee::mlocker::get_page_size() (mlocker.cpp:98)
==331274==    by 0x48AA3F0: epee::mlocker::unlock(void*, unsigned long) (mlocker.cpp:130)
==331274==    by 0x557ED33: epee::mlocked<tools::scrubbed<crypto::ec_scalar> >::~mlocked() (mlocker.h:76)
==331274==    by 0x5E2200D: __cxa_finalize (cxa_finalize.c:83)
==331274==    by 0x556FB56: ??? (in /home/me/workspace/xcash/testnet/xcash-core/build/debug/src/crypto/libcncrypto.so)
==331274==    by 0x4011F5A: _dl_fini (dl-fini.c:138)
==331274==    by 0x5E218D6: __run_exit_handlers (exit.c:108)
==331274==    by 0x5E21A8F: exit (exit.c:139)
==331274==    by 0x5DFF0B9: (below main) (libc-start.c:342)

Oh, and also there's another uninitialized variable in wallet2.cpp height

        uint64_t height;
        if (!string_tools::get_xtype_from_string(height, fields[1]))
          continue;

        MINFO("Found segregation height via DNS: " << fields[0] << " fork height at " << height);
        uint64_t diff = height > current_height ? height - current_height : current_height - height;