Node stuck during Body Sync

quentinlesceller commented 5 years ago

Node (on latest master) is stuck at downloading block with the same message repeating:

20181106 12:06:44.466 DEBUG grin_servers::grin::sync::body_sync - body_sync: body_head - 0367b255, 22765, header_head - 04064432, 27662, sync_head - 04064432, 27662
20181106 12:06:44.466 ERROR grin_servers::grin::sync::body_sync - body_sync: header_head not found in chain db: 04064432 at 27662
20181106 12:06:44.466 DEBUG grin_servers::grin::sync::body_sync - body_sync: cannot sync full blocks earlier than horizon. oldest_height: 0
20181106 12:06:44.477 DEBUG grin_servers::grin::sync::body_sync - body_sync: no pending block request, asking more

Previously:

20181106 12:01:43.254 DEBUG grin_servers::common::types - sync_state: sync_status: HeaderSync { current_height: 27662, highest_height: 27663 } -> BodySync { current_height: 22765, highest_height: 27663 }
20181106 12:01:43.266 DEBUG grin_servers::grin::sync::body_sync - body_sync: expecting 1 more blocks and none received for a while
20181106 12:01:43.266 DEBUG grin_servers::grin::sync::body_sync - body_sync: body_head - 0367b255, 22765, header_head - 04064432, 27662, sync_head - 04064432, 27662
20181106 12:01:43.266 ERROR grin_servers::grin::sync::body_sync - body_sync: header_head not found in chain db: 04064432 at 27662
20181106 12:01:43.266 DEBUG grin_servers::grin::sync::body_sync - body_sync: cannot sync full blocks earlier than horizon. oldest_height: 0
20181106 12:01:43.267 INFO grin_servers::common::adapters - Received block headers [01367626] from 35.197.194.192:13414
20181106 12:01:43.267 DEBUG grin_chain::pipe - pipe: sync_block_headers: 1 headers from 01367626 at 27663
20181106 12:01:43.268 DEBUG grin_servers::common::adapters - Block headers refused by chain: Error { inner: NotFoundErr("BLOCK HEADER: 04064432")

stack backtrace:
   0:        0x10a9c0f3e - backtrace::backtrace::trace::h3de9173b23f2043b
   1:        0x10a9c145c - backtrace::capture::Backtrace::new_unresolved::h1b42e013254f0225
   2:        0x10a9c074d - failure::backtrace::internal::InternalBacktrace::new::hae5d13e0fd2a4952
   3:        0x10a9c09f4 - <failure::backtrace::Backtrace as core::default::Default>::default::h1637e5713182d5b9
   4:        0x10a9c0a6d - failure::backtrace::Backtrace::new::h3f49bb9ce98446ae
   5:        0x10a82e814 - <failure::error::error_impl::ErrorImpl as core::convert::From<F>>::from::h02b1d4b37be58500
   6:        0x10a81f6f6 - <grin_chain::error::Error as core::convert::From<grin_store::lmdb::Error>>::from::h572f7f513589cded
   7:        0x10a824411 - grin_chain::pipe::sync_block_headers::h0c4a2574fb0d240d
   8:        0x10a84fa38 - grin_chain::chain::Chain::sync_block_headers::he58827fafbf736c3
   9:        0x10a436bde - <grin_servers::common::adapters::NetToChainAdapter as grin_p2p::types::ChainAdapter>::headers_received::h8205d73d9c59222d
  10:        0x10a7f1826 - <grin_p2p::peers::Peers as grin_p2p::types::ChainAdapter>::headers_received::hb4cebeb27782f0ce
  11:        0x10a7d7d21 - <grin_p2p::peer::TrackingAdapter as grin_p2p::types::ChainAdapter>::headers_received::h5b528a661516440b
  12:        0x10a7da869 - <grin_p2p::protocol::Protocol as grin_p2p::conn::MessageHandler>::consume::hfbd85afc7dd32a85
  13:        0x10a7f9c03 - std::sys_common::backtrace::__rust_begin_short_backtrace::hef51030611529e1a
  14:        0x10a7fb164 - std::panicking::try::do_call::hb3d3759a7f8b81f8
  15:        0x10aaf2e6e - ___rust_maybe_catch_panic
  16:        0x10a7fc462 - <F as alloc::boxed::FnBox<A>>::call_box::hdf3d8a8822bf43e9
  17:        0x10aae4437 - std::sys_common::thread::start_thread::ha68f94e048f96d59
  18:        0x10aacd298 - std::sys::unix::thread::Thread::new::thread_start::h57baa5b47ae41e12
  19:     0x7fff6262233c - __pthread_body
  20:     0x7fff626252a6 - __pthread_start

Store Error: NotFoundErr("BLOCK HEADER: 04064432") }

Attaching log. grin-server0.log

hashmap commented 5 years ago

Oh, nice, perhaps it explains why it happens, we miss a header somehow. It's duplicate of https://github.com/mimblewimble/grin/issues/1845

quentinlesceller commented 5 years ago

Oh okay closing then.

mimblewimble / grin

Node stuck during Body Sync #1935