Event-emitter node panic on `entered unreachable code` `event_dispatcher.rs:208`

[zone117x]

Mainnet nodes running as event-emitters crashed with error:

Process abort due to thread panic: panicked at 'internal error: entered unreachable code', testnet/stacks-node/src/event_dispatcher.rs:208:18

The offending line: https://github.com/stacks-network/stacks-blockchain/blob/378fc1b80e161ce1b82cc98b3802565222100816/testnet/stacks-node/src/event_dispatcher.rs#L208

It looks like this happened at block height 80743. The log line from before this panic was:

Reached chain tip 741011472dd5dc64a593781799eba2c2b2db3c68/6b83b44571365e6e530d679536578c71d6c376b07666f3671786b6fd8fac049c from 02d4567079a50bd211183b10a9aafb0aa353c175/8b9b67b39393e18e6def2c98a2a253a094227c8a644bfb5c9af4549d1e98717b

Full panic backtrace:

Panic backtrace:    0: stacks_node::main::{{closure}}
             at src/testnet/stacks-node/src/main.rs:59:18
   1: std::panicking::rust_panic_with_hook
             at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:702:17
   2: std::panicking::begin_panic_handler::{{closure}}
             at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:586:13
   3: std::sys_common::backtrace::__rust_end_short_backtrace
             at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/sys_common/backtrace.rs:138:18
   4: rust_begin_unwind
             at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:584:5
   5: core::panicking::panic_fmt
             at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/panicking.rs:142:14
   6: core::panicking::panic
             at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/panicking.rs:48:5
   7: stacks_node::event_dispatcher::EventObserver::generate_payload_info_for_receipt
             at src/testnet/stacks-node/src/event_dispatcher.rs:208:18
      stacks_node::event_dispatcher::EventObserver::make_new_block_txs_payload
             at src/testnet/stacks-node/src/event_dispatcher.rs:244:36
   8: stacks_node::event_dispatcher::EventObserver::send
             at src/testnet/stacks-node/src/event_dispatcher.rs:357:27
      stacks_node::event_dispatcher::EventDispatcher::process_chain_tip
             at src/testnet/stacks-node/src/event_dispatcher.rs:717:17
      <stacks_node::event_dispatcher::EventDispatcher as blockstack_lib::chainstate::coordinator::BlockEventDispatcher>::announce_block
             at src/testnet/stacks-node/src/event_dispatcher.rs:463:9
   9: blockstack_lib::chainstate::stacks::db::blocks::<impl blockstack_lib::chainstate::stacks::db::StacksChainState>::process_next_staging_block
             at src/src/chainstate/stacks/db/blocks.rs:5909:13
      blockstack_lib::chainstate::stacks::db::blocks::<impl blockstack_lib::chainstate::stacks::db::StacksChainState>::process_blocks
             at src/src/chainstate/stacks/db/blocks.rs:5983:19
  10: blockstack_lib::chainstate::coordinator::ChainsCoordinator<T,N,U,CE,FE>::process_ready_blocks
             at src/src/chainstate/coordinator/mod.rs:708:13
  11: blockstack_lib::chainstate::coordinator::ChainsCoordinator<T,N,U,CE,FE>::handle_new_stacks_block
             at src/src/chainstate/coordinator/mod.rs:550:35
      blockstack_lib::chainstate::coordinator::ChainsCoordinator<T,blockstack_lib::chainstate::coordinator::comm::ArcCounterCoordinatorNotices,blockstack_lib::chainstate::coordinator::OnChainRewardSetProvider,CE,FE>::run
             at src/src/chainstate/coordinator/mod.rs:315:37
  12: stacks_node::run_loop::neon::RunLoop::spawn_chains_coordinator::{{closure}}
             at src/testnet/stacks-node/src/run_loop/neon.rs:459:17
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/sys_common/backtrace.rs:122:18
  13: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}
             at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/thread/mod.rs:505:17
      <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/panic/unwind_safe.rs:271:9
      std::panicking::try::do_call
             at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:492:40
      std::panicking::try
             at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:456:19
      std::panic::catch_unwind
             at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panic.rs:137:14
      std::thread::Builder::spawn_unchecked_::{{closure}}
             at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/thread/mod.rs:504:30
      core::ops::function::FnOnce::call_once{{vtable.shim}}
             at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/ops/function.rs:248:5
  14: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/alloc/src/boxed.rs:1951:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/alloc/src/boxed.rs:1951:9
      std::sys::unix::thread::Thread::new::thread_start
             at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/sys/unix/thread.rs:108:17

[jcnelson]

Woah, looks like someone mined a PoisonMicroblock transaction.

This panic is fixed in 2.05.0.5.0-rc1 and develop. I urge affected users to upgrade.

On Mon, Oct 24, 2022, 8:34 AM Matthew Little @.***> wrote:

Mainnet nodes running as event-emitters crashed with error:

Process abort due to thread panic: panicked at 'internal error: entered unreachable code', testnet/stacks-node/src/event_dispatcher.rs:208:18

The offending line: https://github.com/stacks-network/stacks-blockchain/blob/378fc1b80e161ce1b82cc98b3802565222100816/testnet/stacks-node/src/event_dispatcher.rs#L208

It looks like this happened at block height 80743. The log line from before this panic was:

Reached chain tip 741011472dd5dc64a593781799eba2c2b2db3c68/6b83b44571365e6e530d679536578c71d6c376b07666f3671786b6fd8fac049c from 02d4567079a50bd211183b10a9aafb0aa353c175/8b9b67b39393e18e6def2c98a2a253a094227c8a644bfb5c9af4549d1e98717b

Full panic backtrace:

Panic backtrace: 0: stacks_node::main::{{closure}} at src/testnet/stacks-node/src/main.rs:59:18 1: std::panicking::rust_panic_with_hook at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:702:17 2: std::panicking::begin_panic_handler::{{closure}} at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:586:13 3: std::sys_common::backtrace::__rust_end_short_backtrace at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/sys_common/backtrace.rs:138:18 4: rust_begin_unwind at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:584:5 5: core::panicking::panic_fmt at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/panicking.rs:142:14 6: core::panicking::panic at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/panicking.rs:48:5 7: stacks_node::event_dispatcher::EventObserver::generate_payload_info_for_receipt at src/testnet/stacks-node/src/event_dispatcher.rs:208:18 stacks_node::event_dispatcher::EventObserver::make_new_block_txs_payload at src/testnet/stacks-node/src/event_dispatcher.rs:244:36 8: stacks_node::event_dispatcher::EventObserver::send at src/testnet/stacks-node/src/event_dispatcher.rs:357:27 stacks_node::event_dispatcher::EventDispatcher::process_chain_tip at src/testnet/stacks-node/src/event_dispatcher.rs:717:17

::announce_block at src/testnet/stacks-node/src/event_dispatcher.rs:463:9 9: blockstack_lib::chainstate::stacks::db::blocks::::process_next_staging_block at src/src/chainstate/stacks/db/blocks.rs:5909:13 blockstack_lib::chainstate::stacks::db::blocks::::process_blocks at src/src/chainstate/stacks/db/blocks.rs:5983:19 10: blockstack_lib::chainstate::coordinator::ChainsCoordinator::process_ready_blocks at src/src/chainstate/coordinator/mod.rs:708:13 11: blockstack_lib::chainstate::coordinator::ChainsCoordinator::handle_new_stacks_block at src/src/chainstate/coordinator/mod.rs:550:35 blockstack_lib::chainstate::coordinator::ChainsCoordinator::run at src/src/chainstate/coordinator/mod.rs:315:37 12: stacks_node::run_loop::neon::RunLoop::spawn_chains_coordinator::{{closure}} at src/testnet/stacks-node/src/run_loop/neon.rs:459:17 std::sys_common::backtrace::__rust_begin_short_backtrace at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/sys_common/backtrace.rs:122:18 13: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}} at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/thread/mod.rs:505:17 as core::ops::function::FnOnce<()>>::call_once at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/panic/unwind_safe.rs:271:9 std::panicking::try::do_call at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:492:40 std::panicking::try at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panicking.rs:456:19 std::panic::catch_unwind at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/panic.rs:137:14 std::thread::Builder::spawn_unchecked_::{{closure}} at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/thread/mod.rs:504:30 core::ops::function::FnOnce::call_once{{vtable.shim}} at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/core/src/ops/function.rs:248:5 14: as core::ops::function::FnOnce>::call_once at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/alloc/src/boxed.rs:1951:9 as core::ops::function::FnOnce>::call_once at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/alloc/src/boxed.rs:1951:9 std::sys::unix::thread::Thread::new::thread_start at rustc/4b91a6ea7258a947e59c6522cd5898e7c0a6a88f/library/std/src/sys/unix/thread.rs:108:17 — Reply to this email directly, view it on GitHub , or unsubscribe . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[saralab]

This has been fixed in 2.05.0.5.0-rc1, closing