search

marmarachain / marmara

This is the main repository for Marmara Credit Loops Smartchain containing the open source CC Files.
https://marmara.io/
Other
18 stars 8 forks source link

Crash in coin cache code #31

Closed dimxy closed 1 year ago

dimxy commented 2 years ago

A crash in marmara komodod on a NN dev node (on 12 Aug 2022). commit a92d8344d3c545b2641a1e04479c90d19588abe7 (HEAD, tag: v1.1.6, origin/master, origin/HEAD, master)

Core was generated by `/home/ubuntu/marmara/src/komodod -ac_name=MCL -pubkey=03a7edd6d0ba188960e39eced'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000559137aef4d5 in boost::unordered::detail::table<boost::unordered::detail::map<std::allocator<std::pair<uint256 const, CCoinsCacheEntry> >, uint256, CCoinsCacheEntry, CCoinsKeyHasher, std::equal_to<uint256> > >::erase_nodes_unique (j=0x7ff230a6e150, i=0x7ff29269b0d0, this=0x55913b19f1c8)
    at /home/ubuntu/marmara/depends/x86_64-unknown-linux-gnu/share/../include/boost/unordered/detail/implementation.hpp:4058
4058              while (prev->next_ != i)

Backtrace:

(gdb) bt
#0  0x0000559137aef4d5 in boost::unordered::detail::table<boost::unordered::detail::map<std::allocator<std::pair<uint256 const, CCoinsCacheEntry> >, uint256, CCoinsCacheEntry, CCoinsKeyHasher, std::equal_to<uint256> > >::erase_nodes_unique (j=0x7ff230a6e150, i=0x7ff29269b0d0, this=0x55913b19f1c8)
    at /home/ubuntu/marmara/depends/x86_64-unknown-linux-gnu/share/../include/boost/unordered/detail/implementation.hpp:4058
#1  boost::unordered::unordered_map<uint256, CCoinsCacheEntry, CCoinsKeyHasher, std::equal_to<uint256>, std::allocator<std::pair<uint256 const, CCoinsCacheEntry> > >::erase (this=this@entry=0x55913b19f1c8, 
    position=position@entry=...)
    at /home/ubuntu/marmara/depends/x86_64-unknown-linux-gnu/share/../include/boost/unordered/unordered_map.hpp:1703
#2  0x0000559137aec368 in CCoinsViewDB::BatchWrite (this=<optimized out>, mapCoins=..., hashBlock=..., 
    hashSproutAnchor=..., hashSaplingAnchor=..., mapSproutAnchors=..., mapSaplingAnchors=..., 
    mapSproutNullifiers=..., mapSaplingNullifiers=...) at txdb.cpp:197
#3  0x0000559137d8b986 in CCoinsViewBacked::BatchWrite (this=<optimized out>, mapCoins=..., hashBlock=..., 
    hashSproutAnchor=..., hashSaplingAnchor=..., mapSproutAnchors=..., mapSaplingAnchors=..., 
    mapSproutNullifiers=..., mapSaplingNullifiers=...) at coins.cpp:97
#4  0x0000559137d8c157 in CCoinsViewCache::Flush (this=0x55913b19f190) at coins.cpp:547
#5  0x000055913793ef5f in FlushStateToDisk (state=..., mode=mode@entry=FLUSH_STATE_PERIODIC) at main.cpp:4028
#6  0x0000559137970b39 in ActivateBestChain (fSkipdpow=fSkipdpow@entry=false, state=..., 
    pblock=pblock@entry=0x7ff29d7f8d30) at main.cpp:4798
#7  0x0000559137971e04 in ProcessNewBlock (from_miner=from_miner@entry=false, height=height@entry=0, 
    state=..., pfrom=pfrom@entry=0x7ff28c001d00, pblock=pblock@entry=0x7ff29d7f8d30, 
    fForceProcessing=<optimized out>, dbp=0x0) at main.cpp:5971
#8  0x0000559137977f0b in ProcessMessage (pfrom=pfrom@entry=0x7ff28c001d00, strCommand="block", vRecv=..., 
    nTimeReceived=nTimeReceived@entry=1660162518744488) at main.cpp:8129
#9  0x0000559137979e62 in ProcessMessages (pfrom=0x7ff28c001d00) at main.cpp:8376
#10 0x000055913797b9c2 in boost::detail::function::function_invoker1<bool (*)(CNode*), bool, CNode*>::invoke (
    function_ptr=..., a0=<optimized out>)
    at /home/ubuntu/marmara/depends/x86_64-unknown-linux-gnu/share/../include/boost/function/function_template.hpp:100
#11 0x00005591379f0776 in boost::function1<bool, CNode*>::operator() (a0=<optimized out>, 
    this=<optimized out>)
    at /home/ubuntu/marmara/depends/x86_64-unknown-linux-gnu/share/../include/boost/function/function_template.hpp:764
#12 boost::signals2::detail::call_with_tuple_args<bool>::m_invoke<boost::function<bool (CNode*)>, 0u, CNode*&>(boost::function<bool (CNode*)>&, boost::signals2::detail::unsigned_meta_array<0u>, std::tuple<CNode*&> const&, boost::disable_if<boost::is_void<boost::function<bool (CNode*)>::result_type>, void>::type*) const (
    args=std::tuple containing = {...}, func=..., this=<optimized out>)
    at /home/ubuntu/marmara/depends/x86_64-unknown-linux-gnu/share/../include/boost/signals2/detail/variadic_slot_invoker.hpp:98
---Type <return> to continue, or q <return> to quit---
#13 boost::signals2::detail::call_with_tuple_args<bool>::operator()<boost::function<bool (CNode*)>, CNode*&, 1ul>(boost::function<bool (CNode*)>&, std::tuple<CNode*&> const&, mpl_::size_t<1ul>) const (
    args=std::tuple containing = {...}, func=..., this=<optimized out>)
    at /home/ubuntu/marmara/depends/x86_64-unknown-linux-gnu/share/../include/boost/signals2/detail/variadic_slot_invoker.hpp:90
#14 boost::signals2::detail::variadic_slot_invoker<bool, CNode*>::operator()<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> > >(boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> > const&) const (connectionBody=..., this=0x7ff29d7f9af0)
    at /home/ubuntu/marmara/depends/x86_64-unknown-linux-gnu/share/../include/boost/signals2/detail/variadic_slot_invoker.hpp:134
#15 boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, CNode*>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> >::dereference() const (this=0x7ff29d7f96b0)
    at /home/ubuntu/marmara/depends/x86_64-unknown-linux-gnu/share/../include/boost/signals2/detail/slot_call_iterator.hpp:110
#16 boost::iterators::iterator_core_access::dereference<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, CNode*>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, CNode*>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> > const&)
    (f=...)
    at /home/ubuntu/marmara/depends/x86_64-unknown-linux-gnu/share/../include/boost/iterator/iterator_facade.hpp:550
#17 boost::iterators::detail::iterator_facade_base<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, CNode*>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bo---Type <return> to continue, or q <return> to quit---
ol (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> >, bool, boost::iterators::single_pass_traversal_tag, bool const&, long, false, false>::operator*() const (this=0x7ff29d7f96b0)
    at /home/ubuntu/marmara/depends/x86_64-unknown-linux-gnu/share/../include/boost/iterator/iterator_facade.hpp:656
#18 CombinerAll::operator()<boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, CNode*>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> > >(boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, CNode*>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, CNode*>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> >) const (
    last=..., first=..., this=<optimized out>) at net.h:108
#19 boost::signals2::detail::combiner_invoker<bool>::operator()<CombinerAll, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, CNode*>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> > >(CombinerAll&, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, CNode*>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> >, boost::signals2::detail::slot_call_iterator_t<boost::signals2::detail::variadic_slot_invoker<bool, CNode*>, std::_List_iterator<boost::shared_ptr<boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> > >, boost::signals2::detail::connection_body<std::pair<boost::signals2::detail::slot_meta_group, boost::optional<int> >, boost::signals2::slot<bool (CNode*), boost::function<bool (CNode*)> >, boost::signals2::mutex> >) const (last=..., first=..., combiner=..., 
    this=<optimized out>)
    at /home/ubuntu/marmara/depends/x86_64-unknown-linux-gnu/share/../include/boost/signals2/detail/result_type---Type <return> to continue, or q <return> to quit---
_wrapper.hpp:53
#20 boost::signals2::detail::signal_impl<bool (CNode*), CombinerAll, int, std::less<int>, boost::function<bool (CNode*)>, boost::function<bool (boost::signals2::connection const&, CNode*)>, boost::signals2::mutex>::operator()(CNode*) (args#0=<optimized out>, this=0x55913b0268c0)
    at /home/ubuntu/marmara/depends/x86_64-unknown-linux-gnu/share/../include/boost/signals2/detail/signal_template.hpp:247
#21 boost::signals2::signal<bool (CNode*), CombinerAll, int, std::less<int>, boost::function<bool (CNode*)>, boost::function<bool (boost::signals2::connection const&, CNode*)>, boost::signals2::mutex>::operator()(CNode*)
    (args#0=<optimized out>, this=0x559139069ef8 <g_signals+24>)
    at /home/ubuntu/marmara/depends/x86_64-unknown-linux-gnu/share/../include/boost/signals2/detail/signal_template.hpp:722
#22 ThreadMessageHandler () at net.cpp:1606
#23 0x00005591378b485d in TraceThread<void (*)()> (name=<optimized out>, 
    func=0x5591379f00c2 <ThreadMessageHandler()>) at ./util.h:272
#24 0x00005591378acf0f in boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<void (*)()> >::operator()<void (*)(char const*, void (*)()), boost::_bi::list0> (a=<synthetic pointer>..., f=<optimized out>, 
    this=<optimized out>)
    at /home/ubuntu/marmara/depends/x86_64-unknown-linux-gnu/share/../include/boost/bind/bind.hpp:319
#25 boost::_bi::bind_t<void, void (*)(char const*, void (*)()), boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<void (*)()> > >::operator() (this=<optimized out>)
    at /home/ubuntu/marmara/depends/x86_64-unknown-linux-gnu/share/../include/boost/bind/bind.hpp:1294
#26 boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(char const*, void (*)()), boost::_bi::list2<boost::_bi::value<char const*>, boost::_bi::value<void (*)()> > > >::run (this=<optimized out>)
    at /home/ubuntu/marmara/depends/x86_64-unknown-linux-gnu/share/../include/boost/thread/detail/thread.hpp:120
#27 0x000055913809b55c in thread_proxy ()
#28 0x00007ff34d7626db in start_thread (arg=0x7ff29d7fa700) at pthread_create.c:463
#29 0x00007ff34c69661f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
dimxy commented 2 years ago

My guess: in another thread some code is accessing pcoinsTip->cacheCoins object without locking, destroying it.

At the moment of crash in another thread there is MarmaraRunAutoSettlement function running:

#12 0x0000559137947043 in myGetTransaction (hash=..., txOut=..., hashBlock=...) at main.cpp:2289
#13 0x0000559137c60698 in enum_credit_loops<MarmaraRunAutoSettlement(int32_t, std::vector<CTransaction>&)::<lambda(const CTransaction&, const CTransaction&, const CTransaction&, const SMarmaraCreditLoopOpret&)> > (
    callback=..., refcurrency="MARMARA", refpk=..., maxamount=1152921504606846976, minamount=0, 
    lastheight=1073741824, firstheight=0, cp=<optimized out>, nVoutMarker=3) at cc/marmara.cpp:4136
#14 MarmaraRunAutoSettlement (height=<optimized out>, 
    settlementTransactions=std::vector of length 0, capacity 0) at cc/marmara.cpp:4184
#15 0x000055913793f596 in komodo_createnodetransactions () at komodo_gateway.h:2729
#16 0x000055913788d2ae in ThreadUpdateKomodoInternals () at init.cpp:806

In MarmaraRunAutoSettlement there is a call to CCgettxout function which accesses pcoinsTip without locking cs_main. This may be a reason to crash. Now testing a fix with added locking cs_main critical section