search

pocketnetteam / pocketnet.core

Decentralized social network based on the blockchain
https://pocketnet.app
Apache License 2.0
115 stars 28 forks source link

Reindexer segmentation fault #80

Closed tawmaz closed 2 years ago

tawmaz commented 3 years ago

Logging this issue here for investigation purposes. We are moving away from Reindexer in 0.20.00, however we need to make sure this segmentation fault is not caused by something further up the stack. This segmentation fault was captured using a debug build of 0.19.17 with gdb attached.

gdb trap:

Thread 10 "pocketcoin-http" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f0a4affd700 (LWP 302460)]
reindexer::Item::operator[] (this=this@entry=0x7f0a4affbb80, name=name@entry=0x55f40258a899 "txid") at reindexer/core/item.cc:103
103 reindexer/core/item.cc: No such file or directory.

stack:

(gdb) bt
#0  reindexer::Item::operator[] (this=this@entry=0x7f0a4affbb80, name=name@entry=0x55f40258a899 "txid") at reindexer/core/item.cc:103
#1  0x000055f401fbe27b in getPostData (itm=..., address="") at rpc/pocketrpc.cpp:265
#2  0x000055f401ff7d6c in gethierarchicalstrip (request=...) at /usr/include/c++/9/bits/basic_string.h:263
#3  0x000055f401f88161 in CRPCTable::execute (this=0x55f4027c1500 <tableRPC>, request=...) at rpc/server.cpp:510
#4  0x000055f4021f202f in HTTPReq (req=0x7f09e0007240, rpcAuthenticate=rpcAuthenticate@entry=false) at httprpc.cpp:220
#5  0x000055f4021f2da2 in HTTPReq_JSONRPC_Anonymous (req=<optimized out>) at httprpc.cpp:250
#6  0x000055f401ef7086 in std::_Function_handler<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&), bool (*)(HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::_M_invoke(std::_Any_data const&, HTTPRequest*&&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (
    __functor=..., __args#0=<optimized out>, __args#1=...) at /usr/include/c++/9/bits/std_function.h:88
#7  0x000055f4021fa108 in std::function<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::operator()(HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const (
    this=<optimized out>, __args#0=<optimized out>, __args#1=...) at /usr/include/c++/9/bits/std_function.h:263
#8  0x000055f4021fa152 in HTTPWorkItem::operator() (this=<optimized out>) at /usr/include/c++/9/bits/unique_ptr.h:360
#9  0x000055f4021f3c81 in WorkQueue<HTTPClosure>::Run (this=0x55f403ff0dc0) at /usr/include/c++/9/bits/unique_lock.h:103
#10 HTTPWorkQueueRun (queue=0x55f403ff0dc0) at httpserver.cpp:386
#11 0x000055f4021f922b in std::__invoke_impl<void, void (*)(WorkQueue<HTTPClosure>*), WorkQueue<HTTPClosure>*> (__f=<optimized out>)
    at /usr/include/c++/9/bits/invoke.h:89
#12 std::__invoke<void (*)(WorkQueue<HTTPClosure>*), WorkQueue<HTTPClosure>*> (__fn=<optimized out>) at /usr/include/c++/9/bits/invoke.h:95
#13 std::thread::_Invoker<std::tuple<void (*)(WorkQueue<HTTPClosure>*), WorkQueue<HTTPClosure>*> >::_M_invoke<0ul, 1ul> (
    this=<optimized out>) at /usr/include/c++/9/thread:244
#14 std::thread::_Invoker<std::tuple<void (*)(WorkQueue<HTTPClosure>*), WorkQueue<HTTPClosure>*> >::operator() (this=<optimized out>)
    at /usr/include/c++/9/thread:251
#15 std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (*)(WorkQueue<HTTPClosure>*), WorkQueue<HTTPClosure>*> > >::_M_run (
    this=<optimized out>) at /usr/include/c++/9/thread:195
#16 0x00007f0a55b0fde4 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#17 0x00007f0a558b5609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#18 0x00007f0a557da293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) frame 1
#1  0x000055f401fbe27b in getPostData (itm=..., address="") at rpc/pocketrpc.cpp:265

line 265 of rpc/pocketrpc.cpp is this:

    entry.pushKV("txid", itm["txid"].As<string>());

The itm argument at this point in the stack does not have the "txid" key, or anything else useful:

(gdb) info args
itm = @0x7f0a4affbb80: {impl_ = 0x0, status_ = {code_ = 11, what_ = ""}, id_ = -1}
address = ""

We could avoid the segmentation fault at this point by performing a validation check in getPostData that the required keys are available in the argument itm, and then return an empty UniValue if any keys are missing. Then everywhere where getPostData is called, if it returns and empty UniValue, simply continue to the next request, gracefully error, or simply propagate the empty UniValue.

americanpatriotdave commented 3 years ago

19.19 Node OOM:

2021-11-04T20:03:19Z RIDB rollback to block height 1425340 success! 2021-11-04T20:04:45Z +++ Block connected to chain: 1425341 BH: 32d6d4817e4c9d36b4d1fcd601b1b362908fa104b81216d4ebb3deb87906217e CC: 1 (25015) 2021-11-04T20:04:49Z +++ Block connected to chain: 1425342 BH: 9308b00858507717e9017aa0193d429761a48420dead5777f79503a99fc62055 CC: 0 (0) 2021-11-04T20:04:49Z BlockUntilSyncedToCurrentChain: txindex is catching up on block notifications 2021-11-04T20:04:49Z Error: Out of memory. Terminating.

The OS has 24GB of RAM

walkjivefly commented 3 years ago

My 19.17 debug build ran fine for several days, had no reindexer message in the debug log after about 08:09 yesterday, crashed in the early hours of this morning with a SIGSEGV similar to the OP. Debug log ended with

2021-11-05T02:38:56Z Received a POST request for /public/ from xxx:51196
...
2021-11-05T02:39:17Z Received a POST request for /public/ from xxx:52045
2021-11-05T02:39:19Z connection to xxx:37070 timeout
2021-11-05T02:39:21Z Received a POST request for /public/ from xxx:1221

gdb shows

Core was generated by `pocketcoind'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count (__r=..., 
    this=<optimized out>) at /usr/include/c++/7/bits/shared_ptr_base.h:688
688       : _M_pi(__r._M_pi)
[Current thread is 1 (Thread 0x7fe5e4ff9700 (LWP 13177))]
(gdb) bt
#0  std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count (__r=..., 
    this=<optimized out>) at /usr/include/c++/7/bits/shared_ptr_base.h:688
#1  std::__shared_ptr<reindexer::PayloadTypeImpl, (__gnu_cxx::_Lock_policy)2>::__shared_ptr (this=<optimized out>) at /usr/include/c++/7/bits/shared_ptr_base.h:1121
#2  std::shared_ptr<reindexer::PayloadTypeImpl>::shared_ptr (this=<optimized out>)
    at /usr/include/c++/7/bits/shared_ptr.h:119
#3  reindexer::shared_cow_ptr<reindexer::PayloadTypeImpl>::shared_cow_ptr (
    this=<optimized out>) at reindexer/estl/cow.h:9
#4  reindexer::PayloadType::PayloadType (this=<optimized out>)
    at reindexer/core/payload/payloadtype.h:54
#5  reindexer::ItemImpl::Type (this=0x0) at reindexer/core/itemimpl.h:48
#6  reindexer::Item::operator[] (this=this@entry=0x7fe5e4ff7910, 
    name=name@entry=0x5628ad96054f "txid") at reindexer/core/item.cc:103
#7  0x00005628ad2deba2 in getPostData (itm=..., address="") at rpc/pocketrpc.cpp:265
#8  0x00005628ad302b23 in gethierarchicalstrip (request=...)
    at rpc/pocketrpc.cpp:4009
#9  0x00005628ad2b7ee5 in CRPCTable::execute (this=0x5628ade2e860 <tableRPC>, 
    request=...) at rpc/server.cpp:510
#10 0x00005628ad482dbe in HTTPReq (req=0x7fe56c027d00, 
    rpcAuthenticate=<optimized out>) at httprpc.cpp:220
#11 0x00005628ad4858e8 in std::function<bool (HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)>::operator()(HTTPRequest*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const (__args#1="", __args#0=<optimized out>, this=0x7fe56c012780)
    at /usr/include/c++/7/bits/std_function.h:706
#12 HTTPWorkItem::operator() (this=0x7fe56c012750) at httpserver.cpp:58
#13 WorkQueue<HTTPClosure>::Run (this=0x5628aeb2e410) at httpserver.cpp:137
#14 HTTPWorkQueueRun (queue=0x5628aeb2e410) at httpserver.cpp:386
#15 0x00007fe5f57786df in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#16 0x00007fe5f5a4b6db in start_thread (arg=0x7fe5e4ff9700) at pthread_create.c:463
#17 0x00007fe5f4e3571f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)
(gdb) frame 7
#7  0x00005628ad2deba2 in getPostData (itm=..., address="") at rpc/pocketrpc.cpp:265
265     entry.pushKV("txid", itm["txid"].As<string>());
(gdb) frame 8
#8  0x00005628ad302b23 in gethierarchicalstrip (request=...) at rpc/pocketrpc.cpp:4009
4009                entry = getPostData(postItm, "");
(gdb) frame 9
#9  0x00005628ad2b7ee5 in CRPCTable::execute (this=0x5628ade2e860 <tableRPC>, request=...) at rpc/server.cpp:510
510                 ret = pcmd->actor(request);
(gdb) print request
$1 = (const JSONRPCRequest &) @0x7fe5e4ff8b00: {id = {typ = UniValue::VNUM, val = "852088932", 
    keys = std::vector of length 0, capacity 0, values = std::vector of length 0, capacity 0}, 
  strMethod = "gethierarchicalstrip", params = {typ = UniValue::VARR, val = "", keys = std::vector of length 0, capacity 0, 
    values = std::vector of length 6, capacity 6 = {{typ = UniValue::VNUM, val = "0", 
        keys = std::vector of length 0, capacity 0, values = std::vector of length 0, capacity 0}, {typ = UniValue::VSTR, 
        val = "", keys = std::vector of length 0, capacity 0, values = std::vector of length 0, capacity 0}, {
        typ = UniValue::VNUM, val = "10", keys = std::vector of length 0, capacity 0, 
        values = std::vector of length 0, capacity 0}, {typ = UniValue::VSTR, val = "en", 
        keys = std::vector of length 0, capacity 0, values = std::vector of length 0, capacity 0}, {typ = UniValue::VARR, 
        val = "", keys = std::vector of length 0, capacity 0, values = std::vector of length 0, capacity 0}, {
        typ = UniValue::VSTR, val = "", keys = std::vector of length 0, capacity 0, 
        values = std::vector of length 0, capacity 0}}}, fHelp = false, URI = "/public/", authUser = "", 
  peerAddr = "xxx:51196"}
(gdb) frame 10
#10 0x00005628ad482dbe in HTTPReq (req=0x7fe56c027d00, rpcAuthenticate=<optimized out>) at httprpc.cpp:220
220             UniValue result = tableRPC.execute(jreq);
(gdb) print jreq
$2 = {id = {typ = UniValue::VNUM, val = "852088932", keys = std::vector of length 0, capacity 0, 
    values = std::vector of length 0, capacity 0}, strMethod = "gethierarchicalstrip", params = {typ = UniValue::VARR, 
    val = "", keys = std::vector of length 0, capacity 0, values = std::vector of length 6, capacity 6 = {{
        typ = UniValue::VNUM, val = "0", keys = std::vector of length 0, capacity 0, 
        values = std::vector of length 0, capacity 0}, {typ = UniValue::VSTR, val = "", 
        keys = std::vector of length 0, capacity 0, values = std::vector of length 0, capacity 0}, {typ = UniValue::VNUM, 
        val = "10", keys = std::vector of length 0, capacity 0, values = std::vector of length 0, capacity 0}, {
        typ = UniValue::VSTR, val = "en", keys = std::vector of length 0, capacity 0, 
        values = std::vector of length 0, capacity 0}, {typ = UniValue::VARR, val = "", 
        keys = std::vector of length 0, capacity 0, values = std::vector of length 0, capacity 0}, {typ = UniValue::VSTR, 
        val = "", keys = std::vector of length 0, capacity 0, values = std::vector of length 0, capacity 0}}}, fHelp = false, 
  URI = "/public/", authUser = "", peerAddr = "xxx:51196"}

The core is nearly 20GB. I don't have enough disk space for a second. I'll hang onto it for a couple of hours, then bin it and restart with 19.19 non-debug. Shout if you want anything else from it, @andyoknen @tawmaz

americanpatriotdave commented 3 years ago

Another 19.19 dead node

2021-11-05T10:58:14Z RIDB rollback to block height 1426231 success! 2021-11-05T10:58:22Z +++ Block connected to chain: 1426232 BH: b3448553eacd77b88fa6839d887d27bce52ecc34d68eff7863d0c37ec08b4cfd CC: 0 (0) 2021-11-05T10:58:42Z +++ Block connected to chain: 1426233 BH: e3c99268afd3c73f5b13c41e030bbe16797b3fd99fcf2c16308215917954b3ac CC: 0 (0) 2021-11-05T10:58:42Z Available coins count 449 BestHeader: 1426233 e3c99268afd3c73f5b13c41e030bbe16797b3fd99fcf2c16308215917954b3ac 2021-11-05T10:58:55Z RIDB rollback to block height 1426232 success! 2021-11-05T10:58:55Z RIDB rollback to block height 1426231 success! 2021-11-05T10:58:55Z 700004: Transaction RI data Not Found (88f0ff07ded13482b75680e5e178ac2602250f0f2efa78f8f6afc28586658a2d) 2021-11-05T10:58:55Z ERROR: Failed find reindexer transaction part (88f0ff07ded13482b75680e5e178ac2602250f0f2efa78f8f6afc28586658a2d) 2021-11-05T10:58:55Z ERROR: ConnectTip: ConnectBlock 5d59085500b20c117a487f1748a7966b2ef94a348c529c4d8cf2423e3c4faf6a failed, failed-find-rtransaction (code 96) 2021-11-05T10:58:55Z 700004: Transaction RI data Not Found (88f0ff07ded13482b75680e5e178ac2602250f0f2efa78f8f6afc28586658a2d) 2021-11-05T10:58:55Z ERROR: Failed find reindexer transaction part (88f0ff07ded13482b75680e5e178ac2602250f0f2efa78f8f6afc28586658a2d) 2021-11-05T10:58:55Z ERROR: ConnectTip: ConnectBlock b3448553eacd77b88fa6839d887d27bce52ecc34d68eff7863d0c37ec08b4cfd failed, failed-find-rtransaction (code 96) 2021-11-05T10:59:17Z +++ Block connected to chain: 1426232 BH: b3448553eacd77b88fa6839d887d27bce52ecc34d68eff7863d0c37ec08b4cfd CC: 0 (0) 2021-11-05T10:59:17Z 700004: Transaction RI data Not Found (84ec97046f5f77c424c94cded30a790944e5b70b65591f1f2f43e0e6780e333d) 2021-11-05T10:59:17Z ERROR: Failed find reindexer transaction part (84ec97046f5f77c424c94cded30a790944e5b70b65591f1f2f43e0e6780e333d) 2021-11-05T10:59:17Z ERROR: ConnectTip: ConnectBlock e3c99268afd3c73f5b13c41e030bbe16797b3fd99fcf2c16308215917954b3ac failed, failed-find-rtransaction (code 96) 2021-11-05T10:59:17Z Available coins count 449 BestHeader: 1426232 b3448553eacd77b88fa6839d887d27bce52ecc34d68eff7863d0c37ec08b4cfd 2021-11-05T10:59:19Z Available coins count 449 BestHeader: 1426232 b3448553eacd77b88fa6839d887d27bce52ecc34d68eff7863d0c37ec08b4cfd 2021-11-05T10:59:19Z Available coins count 449 BestHeader: 1426232 b3448553eacd77b88fa6839d887d27bce52ecc34d68eff7863d0c37ec08b4cfd 2021-11-05T10:59:20Z ERROR: UndoReadFromDisk: Deserialize or I/O error 2 (26) - CAutoFile::read: end of file: iostream error 2021-11-05T10:59:20Z ERROR: DisconnectBlock(): failure reading undo data 2021-11-05T10:59:20Z ERROR: DisconnectTip(): DisconnectBlock b3448553eacd77b88fa6839d887d27bce52ecc34d68eff7863d0c37ec08b4cfd failed 2021-11-05T10:59:20Z Failed to disconnect block; see debug.log for details 2021-11-05T10:59:20Z Error: Error: A fatal internal error occurred, see debug.log for details 2021-11-05T10:59:20Z ERROR: ProcessNewBlock: ActivateBestChain failed (Failed to disconnect block; see debug.log for details (code 0)) 2021-11-05T10:59:20Z ERROR: UndoReadFromDisk: Deserialize or I/O error 2 (26) - CAutoFile::read: end of file: iostream error 2021-11-05T10:59:20Z ERROR: DisconnectBlock(): failure reading undo data 2021-11-05T10:59:20Z ERROR: DisconnectTip(): DisconnectBlock b3448553eacd77b88fa6839d887d27bce52ecc34d68eff7863d0c37ec08b4cfd failed 2021-11-05T10:59:20Z Failed to disconnect block; see debug.log for details 2021-11-05T10:59:20Z Error: Error: A fatal internal error occurred, see debug.log for details 2021-11-05T10:59:20Z ERROR: ProcessNewBlock: ActivateBestChain failed (Failed to disconnect block; see debug.log for details (code 0)) 2021-11-05T10:59:20Z tor: Thread interrupt 2021-11-05T10:59:20Z Shutdown: In progress... 2021-11-05T10:59:20Z torcontrol thread exit 2021-11-05T10:59:20Z addcon thread exit 2021-11-05T10:59:20Z net thread exit 2021-11-05T10:59:20Z ERROR: UndoReadFromDisk: Deserialize or I/O error 2 (26) - CAutoFile::read: end of file: iostream error 2021-11-05T10:59:20Z ERROR: DisconnectBlock(): failure reading undo data 2021-11-05T10:59:20Z ERROR: DisconnectTip(): DisconnectBlock b3448553eacd77b88fa6839d887d27bce52ecc34d68eff7863d0c37ec08b4cfd failed 2021-11-05T10:59:20Z *** Failed to disconnect block; see debug.log for details 2021-11-05T10:59:20Z Error: Error: A fatal internal error occurred, see debug.log for details 2021-11-05T10:59:20Z ERROR: ProcessNewBlock: ActivateBestChain failed (Failed to disconnect block; see debug.log for details (code 0)) 2021-11-05T10:59:20Z msghand thread exit 2021-11-05T10:59:20Z Warning! Block generate (CheckInputs): 3accf4897ae40284c967a6c6dad04fc7ab5277800b96c432de650001a5981a1c 2021-11-05T10:59:20Z opencon thread exit 2021-11-05T10:59:20Z Warning! Block generate (CheckInputs): 3accf4897ae40284c967a6c6dad04fc7ab5277800b96c432de650001a5981a1c 2021-11-05T10:59:22Z Warning! Block generate (CheckInputs): 3accf4897ae40284c967a6c6dad04fc7ab5277800b96c432de650001a5981a1c 2021-11-05T10:59:22Z ~HTTPRequest: Unhandled request ... ... 2021-11-05T10:59:22Z ~HTTPRequest: Unhandled request 2021-11-05T10:59:22Z Pocketcoin Staker terminated 2021-11-05T10:59:22Z scheduler thread interrupt 2021-11-05T10:59:22Z [default wallet] keypool return 1 2021-11-05T11:00:22Z Close reindexer DB 2021-11-05T11:00:22Z [default wallet] Releasing wallet 2021-11-05T11:00:22Z Shutdown: done

dmaltsiniotis commented 3 years ago

Same issue here on the latest 0.19 version

Pocketnet Core Daemon version v0.19.19.0-a293bf1

2021-11-05T10:57:53Z Available coins count 0 BestHeader: 1426232 5d59085500b20c117a487f1748a7966b2ef94a348c529c4d8cf2423e3c4faf6a
2021-11-05T10:58:09Z Available coins count 0 BestHeader: 1426232 5d59085500b20c117a487f1748a7966b2ef94a348c529c4d8cf2423e3c4faf6a
2021-11-05T10:58:15Z RIDB rollback to block height 1426231 success!
2021-11-05T10:58:17Z +++ Block connected to chain: 1426232 BH: b3448553eacd77b88fa6839d887d27bce52ecc34d68eff7863d0c37ec08b4cfd CC: 0 (0)
2021-11-05T10:58:26Z +++ Block connected to chain: 1426233 BH: e3c99268afd3c73f5b13c41e030bbe16797b3fd99fcf2c16308215917954b3ac CC: 0 (0)
2021-11-05T10:58:26Z Available coins count 0 BestHeader: 1426233 e3c99268afd3c73f5b13c41e030bbe16797b3fd99fcf2c16308215917954b3ac
2021-11-05T10:58:33Z RIDB rollback to block height 1426232 success!
2021-11-05T10:58:33Z RIDB rollback to block height 1426231 success!
2021-11-05T10:58:33Z 700004: Transaction RI data Not Found (88f0ff07ded13482b75680e5e178ac2602250f0f2efa78f8f6afc28586658a2d)
2021-11-05T10:58:33Z ERROR: Failed find reindexer transaction part (88f0ff07ded13482b75680e5e178ac2602250f0f2efa78f8f6afc28586658a2d)
2021-11-05T10:58:33Z ERROR: ConnectTip: ConnectBlock 5d59085500b20c117a487f1748a7966b2ef94a348c529c4d8cf2423e3c4faf6a failed, failed-find-rtransaction (code 96)
2021-11-05T10:58:33Z 700004: Transaction RI data Not Found (88f0ff07ded13482b75680e5e178ac2602250f0f2efa78f8f6afc28586658a2d)
2021-11-05T10:58:33Z ERROR: Failed find reindexer transaction part (88f0ff07ded13482b75680e5e178ac2602250f0f2efa78f8f6afc28586658a2d)
2021-11-05T10:58:33Z ERROR: ConnectTip: ConnectBlock b3448553eacd77b88fa6839d887d27bce52ecc34d68eff7863d0c37ec08b4cfd failed, failed-find-rtransaction (code 96)
2021-11-05T10:58:36Z +++ Block connected to chain: 1426232 BH: b3448553eacd77b88fa6839d887d27bce52ecc34d68eff7863d0c37ec08b4cfd CC: 0 (0)
2021-11-05T10:58:36Z 700004: Transaction RI data Not Found (84ec97046f5f77c424c94cded30a790944e5b70b65591f1f2f43e0e6780e333d)
2021-11-05T10:58:36Z ERROR: Failed find reindexer transaction part (84ec97046f5f77c424c94cded30a790944e5b70b65591f1f2f43e0e6780e333d)
2021-11-05T10:58:36Z ERROR: ConnectTip: ConnectBlock e3c99268afd3c73f5b13c41e030bbe16797b3fd99fcf2c16308215917954b3ac failed, failed-find-rtransaction (code 96)
2021-11-05T10:58:45Z +++ Block connected to chain: 1426233 BH: e3c99268afd3c73f5b13c41e030bbe16797b3fd99fcf2c16308215917954b3ac CC: 0 (0)
2021-11-05T10:58:45Z Available coins count 0 BestHeader: 1426233 e3c99268afd3c73f5b13c41e030bbe16797b3fd99fcf2c16308215917954b3ac
2021-11-05T10:58:45Z ERROR: UndoReadFromDisk: Deserialize or I/O error 2 (25) - CAutoFile::read: end of file: iostream error
2021-11-05T10:58:45Z ERROR: DisconnectBlock(): failure reading undo data
2021-11-05T10:58:45Z ERROR: DisconnectTip(): DisconnectBlock e3c99268afd3c73f5b13c41e030bbe16797b3fd99fcf2c16308215917954b3ac failed
2021-11-05T10:58:45Z *** Failed to disconnect block; see debug.log for details
2021-11-05T10:58:45Z Error: Error: A fatal internal error occurred, see debug.log for details
2021-11-05T10:58:45Z ERROR: ProcessNewBlock: ActivateBestChain failed (Failed to disconnect block; see debug.log for details (code 0))
2021-11-05T10:58:45Z tor: Thread interrupt
2021-11-05T10:58:45Z Shutdown: In progress...
2021-11-05T10:58:45Z torcontrol thread exit
2021-11-05T10:58:45Z opencon thread exit
2021-11-05T10:58:45Z addcon thread exit
2021-11-05T10:58:45Z ERROR: UndoReadFromDisk: Deserialize or I/O error 2 (25) - CAutoFile::read: end of file: iostream error
2021-11-05T10:58:45Z ERROR: DisconnectBlock(): failure reading undo data
2021-11-05T10:58:45Z ERROR: DisconnectTip(): DisconnectBlock e3c99268afd3c73f5b13c41e030bbe16797b3fd99fcf2c16308215917954b3ac failed
2021-11-05T10:58:45Z *** Failed to disconnect block; see debug.log for details
2021-11-05T10:58:45Z Error: Error: A fatal internal error occurred, see debug.log for details
2021-11-05T10:58:45Z ERROR: ProcessNewBlock: ActivateBestChain failed (Failed to disconnect block; see debug.log for details (code 0))
2021-11-05T10:58:45Z msghand thread exit
2021-11-05T10:58:45Z net thread exit
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z ~HTTPRequest: Unhandled request
2021-11-05T10:58:47Z scheduler thread interrupt
2021-11-05T10:58:47Z Pocketcoin Staker terminated
2021-11-05T10:58:47Z [default wallet] keypool return 1
2021-11-05T10:58:47Z Dumped mempool: 1.7e-05s to copy, 0.004181s to dump
2021-11-05T10:59:15Z Close reindexer DB
2021-11-05T10:59:15Z [default wallet] Releasing wallet
2021-11-05T10:59:15Z Shutdown: done
dmaltsiniotis commented 3 years ago

What is extremely interesting is that this happens at the same time on multiple seemingly unrelated nodes. The timestamps for my crash and @americanpatriotdave's line up exactly,

SnowflakeCrusher commented 3 years ago

I also had a similar crash this morning:

2021-11-05T16:48:12Z +++ Block connected to chain: 1426582 BH: adf44b51b951133738dfe1aaf197a02a1d9247cbca2958563adf542b69829751 CC: 0 (0)
2021-11-05T16:48:12Z ERROR: UndoReadFromDisk: Deserialize or I/O error 2 (26) - CAutoFile::read: end of file: iostream error
2021-11-05T16:48:12Z ERROR: DisconnectBlock(): failure reading undo data
2021-11-05T16:48:12Z ERROR: DisconnectTip(): DisconnectBlock adf44b51b951133738dfe1aaf197a02a1d9247cbca2958563adf542b69829751 failed
2021-11-05T16:48:12Z *** Failed to disconnect block; see debug.log for details
2021-11-05T16:48:12Z Error: Error: A fatal internal error occurred, see debug.log for details
2021-11-05T16:48:12Z ERROR: ProcessNewBlock: ActivateBestChain failed (Failed to disconnect block; see debug.log for details (code 0))
2021-11-05T16:48:12Z tor: Thread interrupt
2021-11-05T16:48:12Z Shutdown: In progress...
2021-11-05T16:48:12Z torcontrol thread exit
2021-11-05T16:48:12Z addcon thread exit
2021-11-05T16:48:12Z opencon thread exit
2021-11-05T16:48:12Z net thread exit
2021-11-05T16:48:12Z msghand thread exit
2021-11-05T16:48:12Z scheduler thread interrupt
2021-11-05T16:48:12Z Pocketcoin Staker terminated
2021-11-05T16:48:12Z [default wallet] keypool return 1
2021-11-05T16:48:41Z Close reindexer DB
2021-11-05T16:48:41Z [default wallet] Releasing wallet
2021-11-05T16:48:41Z Shutdown: done
americanpatriotdave commented 3 years ago

0.19.19 Debug from another dead node:

2021-11-06T09:19:25Z +++ Block connected to chain: 1427558 BH: 53ca5580a58bc9880d359c523e60ed673d575038d25475deff5bf906b725b1d4 CC: 0 (0) 2021-11-06T09:19:25Z RIDB rollback to block height 1427557 success! 2021-11-06T09:19:25Z RIDB rollback to block height 1427556 success! 2021-11-06T09:19:25Z 700004: Transaction RI data Not Found (e14c2f643e554fc900f143a37a09e1be6dd8bd371353a2b051a4caa1c4845e88) 2021-11-06T09:19:25Z ERROR: Failed find reindexer transaction part (e14c2f643e554fc900f143a37a09e1be6dd8bd371353a2b051a4caa1c4845e88) 2021-11-06T09:19:25Z ERROR: ConnectTip: ConnectBlock 291e9cf98b9979504c9bbb29a36a752807fe7bf8009fdff3944d53790b656eae failed, failed-find-rtransaction (code 96) 2021-11-06T09:19:25Z 700004: Transaction RI data Not Found (3fb9a9a9acf98dc913f4633725eba62f3755432e8623269a36fe7f388c68af0c) 2021-11-06T09:19:25Z ERROR: Failed find reindexer transaction part (3fb9a9a9acf98dc913f4633725eba62f3755432e8623269a36fe7f388c68af0c) 2021-11-06T09:19:25Z ERROR: ConnectTip: ConnectBlock c589bb894480461a75faff091aa65411cf412ed4f6970cccc4b0cb78828cbfae failed, failed-find-rtransaction (code 96) 2021-11-06T09:19:26Z +++ Block connected to chain: 1427557 BH: c589bb894480461a75faff091aa65411cf412ed4f6970cccc4b0cb78828cbfae CC: 0 (0) 2021-11-06T09:19:26Z 700004: Transaction RI data Not Found (8e7f3c35680288a64f9dba815fa3a6bf75584a01226b232a3507a57af72d0eac) 2021-11-06T09:19:26Z ERROR: Failed find reindexer transaction part (8e7f3c35680288a64f9dba815fa3a6bf75584a01226b232a3507a57af72d0eac) 2021-11-06T09:19:26Z ERROR: ConnectTip: ConnectBlock 53ca5580a58bc9880d359c523e60ed673d575038d25475deff5bf906b725b1d4 failed, failed-find-rtransaction (code 96) 2021-11-06T09:19:26Z Available coins count 33 BestHeader: 1427557 c589bb894480461a75faff091aa65411cf412ed4f6970cccc4b0cb78828cbfae 2021-11-06T09:19:42Z +++ Block connected to chain: 1427558 BH: 53ca5580a58bc9880d359c523e60ed673d575038d25475deff5bf906b725b1d4 CC: 0 (0) 2021-11-06T09:19:42Z Available coins count 33 BestHeader: 1427558 53ca5580a58bc9880d359c523e60ed673d575038d25475deff5bf906b725b1d4 2021-11-06T09:19:42Z Available coins count 33 BestHeader: 1427558 53ca5580a58bc9880d359c523e60ed673d575038d25475deff5bf906b725b1d4 2021-11-06T09:19:43Z ERROR: UndoReadFromDisk: Deserialize or I/O error 2 (26) - CAutoFile::read: end of file: iostream error 2021-11-06T09:19:43Z ERROR: DisconnectBlock(): failure reading undo data 2021-11-06T09:19:43Z ERROR: DisconnectTip(): DisconnectBlock 53ca5580a58bc9880d359c523e60ed673d575038d25475deff5bf906b725b1d4 failed 2021-11-06T09:19:43Z Failed to disconnect block; see debug.log for details 2021-11-06T09:19:43Z Error: Error: A fatal internal error occurred, see debug.log for details 2021-11-06T09:19:43Z ERROR: ProcessNewBlock: ActivateBestChain failed (Failed to disconnect block; see debug.log for details (code 0)) 2021-11-06T09:19:43Z ERROR: UndoReadFromDisk: Deserialize or I/O error 2 (26) - CAutoFile::read: end of file: iostream error 2021-11-06T09:19:43Z ERROR: DisconnectBlock(): failure reading undo data 2021-11-06T09:19:43Z ERROR: DisconnectTip(): DisconnectBlock 53ca5580a58bc9880d359c523e60ed673d575038d25475deff5bf906b725b1d4 failed 2021-11-06T09:19:43Z Failed to disconnect block; see debug.log for details 2021-11-06T09:19:43Z Error: Error: A fatal internal error occurred, see debug.log for details 2021-11-06T09:19:43Z ERROR: ProcessNewBlock: ActivateBestChain failed (Failed to disconnect block; see debug.log for details (code 0)) 2021-11-06T09:19:43Z tor: Thread interrupt 2021-11-06T09:19:43Z Shutdown: In progress... 2021-11-06T09:19:43Z torcontrol thread exit 2021-11-06T09:19:43Z opencon thread exit 2021-11-06T09:19:43Z addcon thread exit 2021-11-06T09:19:43Z net thread exit 2021-11-06T09:19:43Z ERROR: UndoReadFromDisk: Deserialize or I/O error 2 (26) - CAutoFile::read: end of file: iostream error 2021-11-06T09:19:43Z ERROR: DisconnectBlock(): failure reading undo data 2021-11-06T09:19:43Z ERROR: DisconnectTip(): DisconnectBlock 53ca5580a58bc9880d359c523e60ed673d575038d25475deff5bf906b725b1d4 failed 2021-11-06T09:19:43Z *** Failed to disconnect block; see debug.log for details 2021-11-06T09:19:43Z Error: Error: A fatal internal error occurred, see debug.log for details 2021-11-06T09:19:43Z ERROR: ProcessNewBlock: ActivateBestChain failed (Failed to disconnect block; see debug.log for details (code 0)) 2021-11-06T09:19:43Z msghand thread exit 2021-11-06T09:19:43Z Pocketcoin Staker terminated 2021-11-06T09:19:43Z [default wallet] keypool return 1 2021-11-06T09:19:43Z scheduler thread interrupt 2021-11-06T09:20:31Z Close reindexer DB 2021-11-06T09:20:31Z [default wallet] Releasing wallet 2021-11-06T09:20:31Z Shutdown: done

pocketnetteam commented 3 years ago

See #76 for the serialization error