Some causes of rewind_single_block were fixed before 5.0.beta2 but the issue remains.
Issue: The node enters a loop of "rewind_single_block" and becomes unresponsive to API and fails to sync new blocks.
Testcase:
Start with an archive node
Call the nodes "get_block" API to fetch each block from 0 to tip.
Observe the node logs for message like 20201213 16:47:41.127 WARN grin_chain::txhashset::txhashset - rewind_single_block: 3 output_pos entries missing for: 000251721c7d at 781244
Expected Result:
no api timeouts
node should not fall behind in sync
Actual Results:
Observe that the node gives api timeouts while that rewind event is happening
Observe that immediately after the rewind event concludes the node reports being out of sync:
Current DISK READ: 23.92 M/s | Current DISK WRITE: 42.91 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
288633 be/4 ubuntu 22.27 M/s 0.00 B/s 0.00 % 97.18 % grin [peer_read]
2) it reports dozens or hundreds of "no peers" messages
It appears that this is the thread and stack trace for the event:
(gdb) thread 69
[Switching to thread 69 (Thread 0x7e9d675e6700 (LWP 288633))]
#0 0x00005648d2915664 in mdb_page_search_root ()
(gdb) where
#0 0x00005648d2915664 in mdb_page_search_root ()
#1 0x00005648d2915975 in mdb_page_search ()
#2 0x00005648d29160b2 in mdb_cursor_set ()
#3 0x00005648d29166e4 in mdb_get ()
#4 0x00005648d28d970b in grin_store::lmdb::Batch::get_ser ()
#5 0x00005648d28e9f0f in grin_chain::store::Batch::get_block_header ()
#6 0x00005648d28f89b5 in grin_chain::pipe::rewind_and_apply_fork ()
#7 0x00005648d28678c5 in grin_chain::txhashset::txhashset::extending_readonly ()
#8 0x00005648d28bd498 in grin_chain::chain::Chain::convert_block_v2 ()
#9 0x00005648d2462817 in <grin_servers::common::adapters::NetToChainAdapter<B,P,V> as grin_p2p::types::ChainAdapter>::get_block ()
#10 0x00005648d28595ab in <grin_p2p::peers::Peers as grin_p2p::types::ChainAdapter>::get_block ()
#11 0x00005648d280986e in <grin_p2p::peer::TrackingAdapter as grin_p2p::types::ChainAdapter>::get_block ()
#12 0x00005648d2837a15 in <grin_p2p::protocol::Protocol as grin_p2p::conn::MessageHandler>::consume ()
#13 0x00005648d282a726 in grin_p2p::conn::poll::{{closure}} ()
#14 0x00005648d2825026 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#15 0x00005648d280a446 in core::ops::function::FnOnce::call_once{{vtable-shim}} ()
#16 0x00005648d2ac03aa in <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once () at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/alloc/src/boxed.rs:1042
#17 <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once () at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/alloc/src/boxed.rs:1042
#18 std::sys::unix::thread::Thread::new::thread_start () at library/std/src/sys/unix/thread.rs:87
#19 0x00007f1d6d679609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#20 0x00007f1d6d911293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)
Some causes of rewind_single_block were fixed before 5.0.beta2 but the issue remains.
Issue: The node enters a loop of "rewind_single_block" and becomes unresponsive to API and fails to sync new blocks.
Testcase: Start with an archive node Call the nodes "get_block" API to fetch each block from 0 to tip. Observe the node logs for message like
20201213 16:47:41.127 WARN grin_chain::txhashset::txhashset - rewind_single_block: 3 output_pos entries missing for: 000251721c7d at 781244
Expected Result: no api timeouts node should not fall behind in sync
Actual Results: Observe that the node gives api timeouts while that rewind event is happening Observe that immediately after the rewind event concludes the node reports being out of sync:
Some additional information
While the node is unresponsive:
1) its doing a lot of disk IO. From iotop:
2) it reports dozens or hundreds of "no peers" messages
3) The API accepts connections but does not reply after that:
4) After 5 or 10 minutes there is a flood of "rewind_single_block" messages
It appears that this is the thread and stack trace for the event: