[Node Issue]: Downloading blocks stuck at 0.00%

Contact Details

edgardo@xlabs.xyz

What happened?

I am running a testnet node with the latest release version: v2.1.1. After downloading the snapshot as the docs say, the node shows the following logs and is stuck downloading a specific block WARN chain: Failed to validate chunk extra ERROR client: Banning chunk producer for producing invalid chunk

INFO stats: #172275195 Downloading blocks 0.00% (36593 left; at 172275195) 28 peers ⬇ 1.64 MB/s ⬆ 226 kB/s 13.50 bps 249 Tgas/s CPU: 110%, Mem: 2.25 GB

Version

neard (release 2.1.1) (build 2.1.1) (rustc 1.80.0) (protocol 70) (db 40)
neard (release 2.1.0-rc.3) (build 2.1.0-rc.3) (rustc 1.79.0) (protocol 70) (db 40)

Node type

RPC (Default)

Are you a validator?

[ ] I am a validator.

Relevant log output

WARN chain: Failed to validate chunk extra err=InvalidStateRoot prev_block_hash=CaXaVoM3P7iDQFFBmGqkJdDbrWQRvCc4GsMQmTAckm3n block_hash=GhuakoYDPcQTpzbdpXiTPZzyPwxSDk4aXmUFAMhWrcic shard_id=0 prev_chunk_height_included=172275195 prev_chunk_extra=V3(ChunkExtraV3 { state_root: HDEVWwuiddwJvq356kP2NDgU39QxsH8cXaTxBASRLJ44, outcome_root: 3KFFtSMFVCzqpvaqLGQ9mnWqACWensRjTj7w91YgLURW, validator_proposals: [], gas_used: 22151623424567, gas_limit: 1000000000000000, balance_burnt: 1675794682312800000000, congestion_info: V1(CongestionInfoV1 { delayed_receipts_gas: 0, buffered_receipts_gas: 0, receipt_bytes: 0, allowed_shard: 3 }) }) chunk_header=V3(ShardChunkHeaderV3 { inner: V3(ShardChunkHeaderInnerV3 { prev_block_hash: CaXaVoM3P7iDQFFBmGqkJdDbrWQRvCc4GsMQmTAckm3n, prev_state_root: 8ZEp6ubajyE3vg63EwQ5GD2jEWs7hMesp1LeDxCb31Zs, prev_outcome_root: DXGvTeYMQ2G8HCPPdPw4G8pKbNc3tJtG7epKtDm4XoH1, encoded_merkle_root: F2zqiPhSauvYuB5zU6spLjeHhzi8f87zD93s2zeLQmF1, encoded_length: 1086, height_created: 172275196, shard_id: 0, prev_gas_used: 22179466972664, gas_limit: 1000000000000000, prev_balance_burnt: 1677743730679600000000, prev_outgoing_receipts_root: Dm14o3pGYfVEco48T2YS3EBtk6B5mbvRTjp3c7BQZia3, tx_root: 11111111111111111111111111111111, prev_validator_proposals: [], congestion_info: V1(CongestionInfoV1 { delayed_receipts_gas: 0, buffered_receipts_gas: 0, receipt_bytes: 0, allowed_shard: 3 }) }), height_included: 172275196, signature: ed25519:4NJdcd1pNuWjpDEFovJ37r3eU2AV5WjcSijFpDZD8fNRhvxHNHDeR2gba7Q7mAgq4twGHKnNapHc3f3HJNEd6qVE, hash: ChunkHash(4S1PMjeJLHLU15efZXc5NhoHXQ5DprnELDsgzSz8GCVs) })

ERROR client: Banning chunk producer for producing invalid chunk chunk_producer=AccountId("legends.pool.f863973.m0") epoch_id=EpochId(9LAj8rDVaetfb6yntp17G1Ba9CH4kijv5akxbSt1rnEi) chunk_hash=ChunkHash(4S1PMjeJLHLU15efZXc5NhoHXQ5DprnELDsgzSz8GCVs)

INFO stats: #172275195 Downloading blocks 0.00% (36593 left; at 172275195) 28 peers ⬇ 1.64 MB/s ⬆ 226 kB/s 13.50 bps 249 Tgas/s CPU: 110%, Mem: 2.25 GB


### Node head info

```shell
2024-08-23T13:26:12.755625Z  WARN genesis: Skipped genesis validation
2024-08-23T13:26:12.755662Z  WARN genesis: Skipped genesis validation
thread 'main' panicked at tools/state-viewer/src/cli.rs:144:55:
called `Result::unwrap()` on an `Err` value: DbDoesNotExist
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::result::unwrap_failed
   3: state_viewer::cli::StateViewerSubCommand::run
   4: neard::cli::NeardCmd::parse_and_run
   5: neard::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Aborted (core dumped)


### Node upgrade history

```shell
- Upgrade to latest release version v2.1.1: August 20th 13:00 (UTC)

DB reset history

- The node get stucked trying to download block: `172101382`. Yesterday I restarted the DB a re-synced it from snapshot and the same happened at block `172275195`

near / nearcore-support

[Node Issue]: Downloading blocks stuck at 0.00% #8