search

anza-xyz / agave

Web-Scale Blockchain for fast, secure, scalable, decentralized apps and marketplaces.
https://www.anza.xyz/
Apache License 2.0
212 stars 81 forks source link

Validator fails to restart #1491

Open jacklevin74 opened 1 month ago

jacklevin74 commented 1 month ago

After running for few days with high load (CU 42M +) validator fails to restart with an error:

[2024-05-24T00:34:03.140014735Z WARN solana_rpc::rpc] Bank with Finalized not found at slot: 0 [2024-05-24T00:34:03.140074965Z WARN solana_rpc::rpc_health] health check: behind by 415 slots: me=1002936, latest cluster=1003351 0: rust_begin_unwind at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:645:5 1: core::panicking::panic_fmt at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:72:14 2: solana_accounts_db::epoch_accounts_hash::manager::Manager::wait_get_epoch_accounts_hash 3: solana_runtime::bank::Bank::hash_internal_state 4: solana_runtime::bank::Bank::freeze 5: solana_ledger::blockstore_processor::process_blockstore_from_root 6: solana_core::validator::ProcessBlockStore::process 7: solana_core::validator::ProcessBlockStore::process_to_create_tower 8: solana_core::validator::Validator::new 9: solana_validator::main note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace. [2024-05-24T00:34:03.152454200Z ERROR solana_metrics::metrics] datapoint: panic program="validator" thread="main" one=1i message="panicked at accounts-db/src/epoch_accounts_hash/manager.rs:72:35: The epoch accounts hash cannot be awaited when Invalid!" location="accounts-db/src/epoch_accounts_hash/manager.rs:72:35" version="1.18.12 (src:b9c13825; feat:4215500110, client:SolanaLabs)"

brooksprumo commented 2 weeks ago

Can you share the whole log beginning from the restart?

jacklevin74 commented 2 weeks ago

I don't have it anymore, but I'll setup a test cluster and reproduce it

On Wed, Jun 12, 2024 at 3:18 PM Brooks @.***> wrote:

Can you share the whole log beginning from the restart?

— Reply to this email directly, view it on GitHub https://github.com/anza-xyz/agave/issues/1491#issuecomment-2163992863, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEEDZD74RC3YCE5AWWOZDLZHDCMHAVCNFSM6AAAAABIIPOQUGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRTHE4TEOBWGM . You are receiving this because you authored the thread.Message ID: @.***>