search

sigp / lighthouse

Ethereum consensus client in Rust
https://lighthouse.sigmaprime.io/
Apache License 2.0
2.88k stars 724 forks source link

Task Panic. This is a bug! #5498

Open jimmyisthis opened 5 months ago

jimmyisthis commented 5 months ago

Description

I run several Ethereum nodes using Raspberry Pi devices (Raspberry Pi 5 and 4).

Raspberry Pi 5 hosts Geth and Lighthouse, while Raspberry Pi 4-based setup is a dual-device configuration (one device for Geth and one for Lighthouse).

Something (I don't know what it was) happened to the network in my location, resulting in all Lighthouse clients (both on RPi5 and RPi4) failing with the message CRIT Task panic. This is a bug! It happened when I was not around, and I found out about the problem only a few hours later. Restarting the router fixed the networking issues.

Corresponding Geth clients stayed operational, although they lost the Internet connection.

Version

OS & Version: Linux (Ubuntu 23.10) on Raspberry Pi 5 OS & Version: Linux (Ubuntu 20.04.6 LTS) on Raspberry Pi 4

Geth version: 1.13.14-stable-2bd6bd01 Lighthouse version: Lighthouse/v5.1.0-10a38a8, official binary release

Present Behaviour

This happened only once after the networking issue mentioned in the previous paragraph. The problem/issue caused Lighthouse clients to fail abruptly across all my devices.

Client logs:

Raspberry Pi 5 - device 0:

Mar 27 15:55:41.001 WARN Syncing deposit contract block cache    est_blocks_remaining: 1280, service: slot_notifier
Mar 27 15:55:53.000 WARN Low peer count                          peer_count: 0, service: slot_notifier
Mar 27 15:55:53.000 INFO Synced                                  slot: 8727577, block:    …  empty, epoch: 272736, finalized_epoch: 272630, finalized_root: 0x7512…d3b8, exec_hash: 0x9932…93d1 (verified), peers: 0, service: slot_notifier
Mar 27 15:55:53.001 WARN Syncing deposit contract block cache    est_blocks_remaining: 1280, service: slot_notifier
Mar 27 15:56:05.000 WARN Low peer count                          peer_count: 0, service: slot_notifier
Mar 27 15:56:05.000 INFO Synced                                  slot: 8727578, block:    …  empty, epoch: 272736, finalized_epoch: 272630, finalized_root: 0x7512…d3b8, exec_hash: 0x9932…93d1 (verified), peers: 0, service: slot_notifier
Mar 27 15:56:05.001 WARN Syncing deposit contract block cache    est_blocks_remaining: 1280, service: slot_notifier
Mar 27 15:56:17.000 WARN Low peer count                          peer_count: 0, service: slot_notifier
Mar 27 15:56:17.001 INFO Synced                                  slot: 8727579, block:    …  empty, epoch: 272736, finalized_epoch: 272630, finalized_root: 0x7512…d3b8, exec_hash: 0x9932…93d1 (verified), peers: 0, service: slot_notifier
Mar 27 15:56:17.001 WARN Syncing deposit contract block cache    est_blocks_remaining: 1280, service: slot_notifier
Mar 27 15:56:29.000 WARN Low peer count                          peer_count: 0, service: slot_notifier
Mar 27 15:56:29.000 INFO Synced                                  slot: 8727580, block:    …  empty, epoch: 272736, finalized_epoch: 272630, finalized_root: 0x7512…d3b8, exec_hash: 0x9932…93d1 (verified), peers: 0, service: slot_notifier
Mar 27 15:56:29.001 WARN Syncing deposit contract block cache    est_blocks_remaining: 1280, service: slot_notifier
Mar 27 15:56:30.546 WARN Execution endpoint is not synced        last_seen_block_unix_timestamp: 1711514591, endpoint: http://localhost:8551/, auth=true, service: deposit_contract_rpc
Mar 27 15:56:30.547 ERRO Error updating deposit contract cache   error: Failed to get remote head and new block ranges: EndpointError(FarBehind), retry_millis: 60000, service: deposit_contract_rpc
Mar 27 15:56:30.551 INFO Updated local ENR                       udp6: None, tcp6: None, tcp4: Some(9000), udp4: Some(9000), ip4: Some(xxx.xxx.xxx.xxx), id: 0x2926..b774, seq: 206, enr: enr:-MW4QOWEjgPGYOyOC5m2HuMCbTTC0__QM6847kueRhyH4AgdEf2bxpfSluRX3z509gV6V1sO2jhV5NorCVcgu0atfaSBzodhdHRuZXRziAAAAAAAAAYAhGV0aDKQapWhqQQAAAD__________4JpZIJ2NIJpcITZQ91KhHF1aWOCIymJc2VjcDI1NmsxoQLeXCxyAQr4si90hX3m9BxbZxt32I_KoCas5kxiWO_KxIhzeW5jbmV0cwCDdGNwgiMog3VkcIIjKA, service: libp2p
Mar 27 15:56:30.551 INFO Updated local ENR                       udp6: None, tcp6: None, tcp4: Some(9000), udp4: Some(9000), ip4: Some(xxx.xxx.xxx.xxx), id: 0x2926..b774, seq: 207, enr: enr:-MW4QEVeJe1K1hzsKX8LrxIzi3aYzTDfryI9JqQ6kZCBe09PQVPG7AvrVgtDOrh7fRcpX9SvNkdPludeTjbSyixI3SmBz4dhdHRuZXRziAAAAAAAAAYAhGV0aDKQapWhqQQAAAD__________4JpZIJ2NIJpcITZQ91KhHF1aWOCIymJc2VjcDI1NmsxoQLeXCxyAQr4si90hX3m9BxbZxt32I_KoCas5kxiWO_KxIhzeW5jbmV0cwCDdGNwgiMog3VkcIIjKA, service: libp2p
Mar 27 15:56:34.499 CRIT Task panic. This is a bug!              advice: Please check above for a backtrace and notify the developers, backtrace:    0: lighthouse::run::{{closure}}
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2029:9
   2: std::panicking::rust_panic_with_hook
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:783:13

Mar 27 15:56:34.499 CRIT Task panic. This is a bug!              advice: Please check above for a backtrace and notify the developers, backtrace:    0: lighthouse::run::{{closure}}
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2029:9
   2: std::panicking::rust_panic_with_hook
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:783:13
   3: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:657:13
   4: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
   5: rust_begin_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
   6: core::panicking::panic_fmt
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
   7: core::panicking::panic_display
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:196:5
   8: core::panicking::panic_str
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:171:5
   9: core::option::expect_failed
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/option.rs:1980:5
  10: <libp2p_upnp::behaviour::Behaviour as libp2p_swarm::behaviour::NetworkBehaviour>::poll
  11: libp2p_swarm::Swarm<TBehaviour>::poll_next_event
  12: lighthouse_network::service::Network<AppReqId,TSpec>::poll_network
  13: network::service::NetworkService<T>::spawn_service::{{closure}}
  14: task_executor::TaskExecutor::spawn_handle::{{closure}}
  15: tokio::runtime::task::raw::poll
  16: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  17: tokio::runtime::task::raw::poll
  18: std::sys_common::backtrace::__rust_begin_short_backtrace
  19: core::ops::function::FnOnce::call_once{{vtable.shim}}
  20: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  21: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  22: std::sys::unix::thread::Thread::new::thread_start
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys/unix/thread.rs:108:17
  23: <unknown>
  24: <unknown>
, message: mapping should exist, location: /cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-upnp-0.2.1/src/behaviour.rs:450:38
Mar 27 15:56:34.503 INFO Saved DHT state                         service: network
Mar 27 15:56:34.503 INFO Network service shutdown                service: network
Mar 27 15:56:34.562 INFO Internal shutdown received              reason: Panic (fatal error)
Mar 27 15:56:34.562 INFO Shutting down..                         reason: Failure("Panic (fatal error)")
Mar 27 15:56:34.563 CRIT Task panic. This is a bug!              advice: Please check above for a backtrace and notify the developers, backtrace:    0: lighthouse::run::{{closure}}
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2029:9
   2: std::panicking::rust_panic_with_hook
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:783:13
   3: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:657:13
   4: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
   5: rust_begin_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
   6: core::panicking::panic_fmt
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
   7: core::result::unwrap_failed
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/result.rs:1649:5
   8: libp2p_upnp::tokio::search_gateway::{{closure}}
   9: tokio::runtime::task::raw::poll
  10: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  11: tokio::runtime::task::raw::poll
  12: std::sys_common::backtrace::__rust_begin_short_backtrace
  13: core::ops::function::FnOnce::call_once{{vtable.shim}}
  14: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  15: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  16: std::sys::unix::thread::Thread::new::thread_start
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys/unix/thread.rs:108:17
  17: <unknown>
  18: <unknown>
, message: receiver should be available: SendError { kind: Disconnected }, location: /cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-upnp-0.2.1/src/tokio.rs:164:18
Mar 27 15:56:39.393 INFO Saved beacon chain to disk              service: beacon
Panic (fatal error)

Raspberry Pi 5 - device 1:

Mar 27 15:58:29.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
Mar 27 15:58:29.001 INFO Searching for peers                     current_slot: 8727590, head_slot: 8724214, finalized_epoch: 272630, finalized_root: 0x7512…d3b8, peers: 0, service: slot_notifier
Mar 27 15:58:41.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
Mar 27 15:58:41.001 INFO Searching for peers                     current_slot: 8727591, head_slot: 8724214, finalized_epoch: 272630, finalized_root: 0x7512…d3b8, peers: 0, service: slot_notifier
Mar 27 15:58:53.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
Mar 27 15:58:53.001 INFO Searching for peers                     current_slot: 8727592, head_slot: 8724214, finalized_epoch: 272630, finalized_root: 0x7512…d3b8, peers: 0, service: slot_notifier
Mar 27 15:59:05.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
Mar 27 15:59:05.001 INFO Searching for peers                     current_slot: 8727593, head_slot: 8724214, finalized_epoch: 272630, finalized_root: 0x7512…d3b8, peers: 0, service: slot_notifier
Mar 27 15:59:17.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
Mar 27 15:59:17.001 INFO Searching for peers                     current_slot: 8727594, head_slot: 8724214, finalized_epoch: 272630, finalized_root: 0x7512…d3b8, peers: 0, service: slot_notifier
Mar 27 15:59:29.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
Mar 27 15:59:29.001 INFO Searching for peers                     current_slot: 8727595, head_slot: 8724214, finalized_epoch: 272630, finalized_root: 0x7512…d3b8, peers: 0, service: slot_notifier
Mar 27 15:59:31.143 INFO Updated local ENR                       udp6: None, tcp6: None, tcp4: Some(9300), udp4: Some(9300), ip4: Some(xxx.xxx.xxx.xxx), id: 0xbbd2..1590, seq: 82, enr: enr:-MS4QJgbtzx7mOS-FQzSfZoFqmWY8VmVzcuzTUmgmE1zXDfMXEOpwPgp8TrruAOwz0Wd5BFDpYly4UbQVrmRqaC351ZSh2F0dG5ldHOIAAAAAAAAwACEZXRoMpBqlaGpBAAAAP__________gmlkgnY0gmlwhNlD3UqEcXVpY4IkVYlzZWNwMjU2azGhAuK36sraWjBf0_sFQSPm6FjLyrITK0HEJO78sUWn7C9diHN5bmNuZXRzAIN0Y3CCJFSDdWRwgiRU, service: libp2p
Mar 27 15:59:31.144 INFO Updated local ENR                       udp6: None, tcp6: None, tcp4: Some(9300), udp4: Some(9300), ip4: Some(xxx.xxx.xxx.xxx), id: 0xbbd2..1590, seq: 83, enr: enr:-MS4QBJDeLJ7b9xmfNRiV7iH7MszpMeZTzWPw5AyZUEnV_txJZHd4_QxTXIIt_5bA_7FN_SyBVEV8t_uJCSUMaTXs-dTh2F0dG5ldHOIAAAAAAAAwACEZXRoMpBqlaGpBAAAAP__________gmlkgnY0gmlwhNlD3UqEcXVpY4IkVYlzZWNwMjU2azGhAuK36sraWjBf0_sFQSPm6FjLyrITK0HEJO78sUWn7C9diHN5bmNuZXRzAIN0Y3CCJFSDdWRwgiRU, service: libp2p
Mar 27 15:59:41.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
Mar 27 15:59:41.001 INFO Searching for peers                     current_slot: 8727596, head_slot: 8724214, finalized_epoch: 272630, finalized_root: 0x7512…d3b8, peers: 0, service: slot_notifier
Mar 27 15:59:41.504 CRIT Task panic. This is a bug!              advice: Please check above for a backtrace and notify the developers, backtrace:    0: lighthouse::run::{{closure}}
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2029:9
   2: std::panicking::rust_panic_with_hook
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:783:13
   3: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:657:13
   4: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
   5: rust_begin_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
   6: core::panicking::panic_fmt
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
   7: core::panicking::panic_display
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:196:5
   8: core::panicking::panic_str
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:171:5
   9: core::option::expect_failed
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/option.rs:1980:5
  10: <libp2p_upnp::behaviour::Behaviour as libp2p_swarm::behaviour::NetworkBehaviour>::poll
  11: libp2p_swarm::Swarm<TBehaviour>::poll_next_event
  12: lighthouse_network::service::Network<AppReqId,TSpec>::poll_network
  13: network::service::NetworkService<T>::spawn_service::{{closure}}
  14: task_executor::TaskExecutor::spawn_handle::{{closure}}
  15: tokio::runtime::task::raw::poll
  16: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  17: tokio::runtime::task::raw::poll
  18: std::sys_common::backtrace::__rust_begin_short_backtrace
  19: core::ops::function::FnOnce::call_once{{vtable.shim}}
  20: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  21: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  22: std::sys::unix::thread::Thread::new::thread_start
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys/unix/thread.rs:108:17
  23: <unknown>
  24: <unknown>
, message: mapping should exist, location: /cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-upnp-0.2.1/src/behaviour.rs:450:38
Mar 27 15:59:41.523 INFO Saved DHT state                         service: network
Mar 27 15:59:41.523 INFO Network service shutdown                service: network
Mar 27 15:59:41.716 INFO Internal shutdown received              reason: Panic (fatal error)
Mar 27 15:59:41.716 INFO Shutting down..                         reason: Failure("Panic (fatal error)")
Mar 27 15:59:41.718 CRIT Task panic. This is a bug!              advice: Please check above for a backtrace and notify the developers, backtrace:    0: lighthouse::run::{{closure}}
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2029:9
   2: std::panicking::rust_panic_with_hook
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:783:13
   3: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:657:13
   4: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
   5: rust_begin_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
   6: core::panicking::panic_fmt
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
   7: core::result::unwrap_failed
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/result.rs:1649:5
   8: libp2p_upnp::tokio::search_gateway::{{closure}}
   9: tokio::runtime::task::raw::poll
  10: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  11: tokio::runtime::task::raw::poll
  12: std::sys_common::backtrace::__rust_begin_short_backtrace
  13: core::ops::function::FnOnce::call_once{{vtable.shim}}
  14: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  15: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  16: std::sys::unix::thread::Thread::new::thread_start
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys/unix/thread.rs:108:17
  17: <unknown>
  18: <unknown>
, message: receiver should be available: SendError { kind: Disconnected }, location: /cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-upnp-0.2.1/src/tokio.rs:164:18
Mar 27 15:59:48.438 INFO Saved beacon chain to disk              service: beacon
Panic (fatal error)

Raspberry Pi 4 - device 2:

Mar 27 15:56:05.001 INFO Synced                                  slot: 8727578, block:    …  empty, epoch: 272736, finalized_epoch: 272630, finalized_root: 0x7512…d3b8, exec_hash: 0x9932…93d1 (verified), peers: 0, service: slot_notifier
Mar 27 15:56:05.004 WARN Syncing deposit contract block cache    est_blocks_remaining: 1280, service: slot_notifier
Mar 27 15:56:05.019 ERRO Error during execution engine upcheck   error: HttpClient(url: http://ethtest1.local:8551/, kind: timeout, detail: operation timed out), service: exec
Mar 27 15:56:17.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
Mar 27 15:56:17.002 INFO Synced                                  slot: 8727579, block:    …  empty, epoch: 272736, finalized_epoch: 272630, finalized_root: 0x7512…d3b8, exec_hash: 0x9932…93d1 (verified), peers: 0, service: slot_notifier
Mar 27 15:56:17.004 WARN Syncing deposit contract block cache    est_blocks_remaining: 1280, service: slot_notifier
Mar 27 15:56:17.020 ERRO Error during execution engine upcheck   error: HttpClient(url: http://ethtest1.local:8551/, kind: timeout, detail: operation timed out), service: exec
Mar 27 15:56:29.001 WARN Low peer count                          peer_count: 0, service: slot_notifier
Mar 27 15:56:29.002 INFO Synced                                  slot: 8727580, block:    …  empty, epoch: 272736, finalized_epoch: 272630, finalized_root: 0x7512…d3b8, exec_hash: 0x9932…93d1 (verified), peers: 0, service: slot_notifier
Mar 27 15:56:29.005 WARN Syncing deposit contract block cache    est_blocks_remaining: 1280, service: slot_notifier
Mar 27 15:56:29.021 ERRO Error during execution engine upcheck   error: HttpClient(url: http://ethtest1.local:8551/, kind: timeout, detail: operation timed out), service: exec
Mar 27 15:56:41.000 WARN Low peer count                          peer_count: 0, service: slot_notifier
Mar 27 15:56:41.001 INFO Synced                                  slot: 8727581, block:    …  empty, epoch: 272736, finalized_epoch: 272630, finalized_root: 0x7512…d3b8, exec_hash: 0x9932…93d1 (verified), peers: 0, service: slot_notifier
Mar 27 15:56:41.001 WARN Syncing deposit contract block cache    est_blocks_remaining: 1280, service: slot_notifier
Mar 27 15:56:41.021 ERRO Error during execution engine upcheck   error: HttpClient(url: http://ethtest1.local:8551/, kind: timeout, detail: operation timed out), service: exec
Mar 27 15:56:44.833 WARN Error connecting to eth1 node endpoint  endpoint: http://ethtest1.local:8551/, auth=true, service: deposit_contract_rpc
Mar 27 15:56:44.833 ERRO Error updating deposit contract cache   error: Invalid endpoint state: RequestFailed("eth_chainId call failed HttpClient(url: http://ethtest1.local:8551/, kind: request, detail: error trying to connect: dns error: failed to lookup address information: Name or service not known)"), retry_millis: 60000, service: deposit_contract_rpc
Mar 27 15:56:53.000 WARN Low peer count                          peer_count: 0, service: slot_notifier
Mar 27 15:56:53.001 INFO Synced                                  slot: 8727582, block:    …  empty, epoch: 272736, finalized_epoch: 272630, finalized_root: 0x7512…d3b8, exec_hash: 0x9932…93d1 (verified), peers: 0, service: slot_notifier
Mar 27 15:56:53.004 WARN Syncing deposit contract block cache    est_blocks_remaining: 1280, service: slot_notifier
Mar 27 15:56:53.023 ERRO Error during execution engine upcheck   error: HttpClient(url: http://ethtest1.local:8551/, kind: timeout, detail: operation timed out), service: exec
Mar 27 15:57:01.198 INFO Updated local ENR                       udp6: None, tcp6: None, tcp4: Some(9100), udp4: Some(9100), ip4: Some(xxx.xxx.xxx.xxx), id: 0xb8a7..97f1, seq: 137, enr: enr:-MW4QAhKWGl1wq_YjiwbmwHBv8bxsq_rFPBU3GNG3Y8hUB0sOJIszmkkRWV3fSrA9RqG4SS-jUkoIPNnrnUPcFWujSOBiYdhdHRuZXRziAAAAAAAAMAAhGV0aDKQapWhqQQAAAD__________4JpZIJ2NIJpcITZQ91KhHF1aWOCI42Jc2VjcDI1NmsxoQPYBQw0brzJjCFZ4Xv_kHEhbEiHX7GEyLLhjpXCoobDkohzeW5jbmV0cwCDdGNwgiOMg3VkcIIjjA, service: libp2p
Mar 27 15:57:01.199 INFO Updated local ENR                       udp6: None, tcp6: None, tcp4: Some(9100), udp4: Some(9100), ip4: Some(xxx.xxx.xxx.xxx), id: 0xb8a7..97f1, seq: 138, enr: enr:-MW4QLOO0XXhqiBdGxE1A07j7rFEEIfWZrz2iF5uY8moaqw8fiJRG1HgxJ88VMnEUOanZ56fqz4kp_Yn3_woq58a7S2BiodhdHRuZXRziAAAAAAAAMAAhGV0aDKQapWhqQQAAAD__________4JpZIJ2NIJpcITZQ91KhHF1aWOCI42Jc2VjcDI1NmsxoQPYBQw0brzJjCFZ4Xv_kHEhbEiHX7GEyLLhjpXCoobDkohzeW5jbmV0cwCDdGNwgiOMg3VkcIIjjA, service: libp2p
Mar 27 15:57:02.139 INFO UPnP route established                  addr: /ip4/xxx.xxx.xxx.xxx/udp/9101/quic-v1, service: libp2p
Mar 27 15:57:02.241 CRIT Task panic. This is a bug!              advice: Please check above for a backtrace and notify the developers, backtrace:    0: lighthouse::run::{{closure}}
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2029:9
   2: std::panicking::rust_panic_with_hook
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:783:13
   3: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:657:13
   4: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
   5: rust_begin_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
   6: core::panicking::panic_fmt
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
   7: core::panicking::panic_display
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:196:5
   8: core::panicking::panic_str
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:171:5
   9: core::option::expect_failed
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/option.rs:1980:5
  10: <libp2p_upnp::behaviour::Behaviour as libp2p_swarm::behaviour::NetworkBehaviour>::poll
  11: libp2p_swarm::Swarm<TBehaviour>::poll_next_event
  12: lighthouse_network::service::Network<AppReqId,TSpec>::poll_network
  13: network::service::NetworkService<T>::spawn_service::{{closure}}
  14: task_executor::TaskExecutor::spawn_handle::{{closure}}
  15: tokio::runtime::task::raw::poll
  16: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  17: tokio::runtime::task::raw::poll
  18: std::sys_common::backtrace::__rust_begin_short_backtrace
  19: core::ops::function::FnOnce::call_once{{vtable.shim}}
  20: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  21: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  22: std::sys::unix::thread::Thread::new::thread_start
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys/unix/thread.rs:108:17
  23: start_thread
  24: <unknown>
, message: mapping should exist, location: /cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-upnp-0.2.1/src/behaviour.rs:419:38
Mar 27 15:57:02.248 INFO Saved DHT state                         service: network
Mar 27 15:57:02.248 INFO Network service shutdown                service: network
Mar 27 15:57:02.295 INFO Internal shutdown received              reason: Panic (fatal error)
Mar 27 15:57:02.296 INFO Shutting down..                         reason: Failure("Panic (fatal error)")
Mar 27 15:57:02.299 CRIT Task panic. This is a bug!              advice: Please check above for a backtrace and notify the developers, backtrace:    0: lighthouse::run::{{closure}}
   1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2029:9
   2: std::panicking::rust_panic_with_hook
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:783:13
   3: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:657:13
   4: std::sys_common::backtrace::__rust_end_short_backtrace
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys_common/backtrace.rs:171:18
   5: rust_begin_unwind
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/panicking.rs:645:5
   6: core::panicking::panic_fmt
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/panicking.rs:72:14
   7: core::result::unwrap_failed
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/core/src/result.rs:1649:5
   8: libp2p_upnp::tokio::search_gateway::{{closure}}
   9: tokio::runtime::task::raw::poll
  10: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  11: tokio::runtime::task::raw::poll
  12: std::sys_common::backtrace::__rust_begin_short_backtrace
  13: core::ops::function::FnOnce::call_once{{vtable.shim}}
  14: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  15: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/alloc/src/boxed.rs:2015:9
  16: std::sys::unix::thread::Thread::new::thread_start
             at /rustc/07dca489ac2d933c78d3c5158e3f43beefeb02ce/library/std/src/sys/unix/thread.rs:108:17
  17: start_thread
  18: <unknown>
, message: receiver should be available: SendError { kind: Disconnected }, location: /cargo/registry/src/index.crates.io-6f17d22bba15001f/libp2p-upnp-0.2.1/src/tokio.rs:164:18
Mar 27 15:57:10.128 INFO Saved beacon chain to disk              service: beacon
Panic (fatal error)

Expected Behaviour

Most probably, the clients should stay operational but without Internet access.

Steps to resolve

The database was not corrupted, so fixing the networking issue (restarting the router) and rerunning Lighthouse clients was enough for the Ethereum nodes to start syncing.

jxs commented 5 months ago

Hi, and thanks for the report. Seems similar to #5444 I am working on a fix to submit upstream