search

NethermindEth / nethermind

A robust execution client for Ethereum node operators.
https://nethermind.io/nethermind-client
GNU General Public License v3.0
1.26k stars 432 forks source link

Processing loop threw an exception #4514

Closed dasudiy closed 4 months ago

dasudiy commented 2 years ago

Describe the bug Sudden encounter error during mainnet sync, and stuck here, using default mainnet.cfg

2022-09-01 12:40:23.7525|Processing part of a long blocks branch 0/193. Block: 15452578 (0x85fa78...59239c)
2022-09-01 12:40:23.8040|Processing loop threw an exception. Block: Nethermind.Consensus.Processing.BlockRef, Exception: Nethermind.Trie.TrieException: Failed to load key 8a35acfbc15ff81a39ae7d344fd709f28e8600b4aa8c65c6b64bfe7fe36bd19b from root hash 0x1bc592869c66ccc91eb8f90291dbb92ad707ab7acf18785a9eea9dec3fdc2277.
 ---> Nethermind.Trie.TrieException: Node 0x1bc592869c66ccc91eb8f90291dbb92ad707ab7acf18785a9eea9dec3fdc2277 is missing from the DB
   at Nethermind.Trie.Pruning.TrieStore.LoadRlp(Keccak keccak, IKeyValueStore keyValueStore)
   at Nethermind.Trie.TrieNode.ResolveNode(ITrieNodeResolver tree)
   at Nethermind.Trie.PatriciaTree.Run(Span`1 updatePath, Int32 nibblesCount, Byte[] updateValue, Boolean isUpdate, Boolean ignoreMissingDelete, Keccak startRootHash)
   at Nethermind.Trie.PatriciaTree.Get(Span`1 rawKey, Keccak rootHash)
   --- End of inner exception stack trace ---
   at Nethermind.Trie.PatriciaTree.Get(Span`1 rawKey, Keccak rootHash)
   at Nethermind.State.PersistentStorageProvider.LoadFromTree(StorageCell storageCell)
   at Nethermind.State.PersistentStorageProvider.GetCurrentValue(StorageCell storageCell)
   at Nethermind.Evm.VirtualMachine.ExecuteCall(EvmState vmState, Byte[] previousCallResult, ZeroPaddedSpan previousCallOutput, UInt256& previousCallOutputDestination, IReleaseSpec spec)
   at Nethermind.Evm.VirtualMachine.Run(EvmState state, IWorldState worldState, ITxTracer txTracer)
   at Nethermind.Evm.TransactionProcessing.TransactionProcessor.Execute(Transaction transaction, BlockHeader block, ITxTracer txTracer, ExecutionOptions executionOptions)
   at Nethermind.Consensus.Processing.TransactionProcessorAdapterExtensions.ProcessTransaction(ITransactionProcessorAdapter transactionProcessor, Block block, Transaction currentTx, BlockReceiptsTracer receiptsTracer, ProcessingOptions processingOptions, IStateProvider stateProvider)
   at Nethermind.Consensus.Processing.BlockProcessor.BlockValidationTransactionsExecutor.ProcessTransactions(Block block, ProcessingOptions processingOptions, BlockReceiptsTracer receiptsTracer, IReleaseSpec spec)
   at Nethermind.Consensus.Processing.BlockProcessor.ProcessBlock(Block block, IBlockTracer blockTracer, ProcessingOptions options)
   at Nethermind.Consensus.Processing.BlockProcessor.ProcessOne(Block suggestedBlock, ProcessingOptions options, IBlockTracer blockTracer)
   at Nethermind.Consensus.Processing.BlockProcessor.Process(Keccak newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer)
   at Nethermind.Consensus.Processing.BlockchainProcessor.ProcessBranch(ProcessingBranch processingBranch, ProcessingOptions options, IBlockTracer tracer)
   at Nethermind.Consensus.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer)
   at Nethermind.Consensus.Processing.BlockchainProcessor.RunProcessingLoop()
2022-09-01 12:40:27.3861|Processing part of a long blocks branch 0/194. Block: 15452578 (0x85fa78...59239c)
2022-09-01 12:40:27.4605|Processing loop threw an exception. Block: Nethermind.Consensus.Processing.BlockRef, Exception: Nethermind.Trie.TrieException: Failed to load key 8a35acfbc15ff81a39ae7d344fd709f28e8600b4aa8c65c6b64bfe7fe36bd19b from root hash 0x1bc592869c66ccc91eb8f90291dbb92ad707ab7acf18785a9eea9dec3fdc2277.
 ---> Nethermind.Trie.TrieException: Node 0x1bc592869c66ccc91eb8f90291dbb92ad707ab7acf18785a9eea9dec3fdc2277 is missing from the DB
   at Nethermind.Trie.Pruning.TrieStore.LoadRlp(Keccak keccak, IKeyValueStore keyValueStore)
   at Nethermind.Trie.TrieNode.ResolveNode(ITrieNodeResolver tree)
   at Nethermind.Trie.PatriciaTree.Run(Span`1 updatePath, Int32 nibblesCount, Byte[] updateValue, Boolean isUpdate, Boolean ignoreMissingDelete, Keccak startRootHash)
   at Nethermind.Trie.PatriciaTree.Get(Span`1 rawKey, Keccak rootHash)
   --- End of inner exception stack trace ---
   at Nethermind.Trie.PatriciaTree.Get(Span`1 rawKey, Keccak rootHash)
   at Nethermind.State.PersistentStorageProvider.LoadFromTree(StorageCell storageCell)
   at Nethermind.State.PersistentStorageProvider.GetCurrentValue(StorageCell storageCell)
   at Nethermind.Evm.VirtualMachine.ExecuteCall(EvmState vmState, Byte[] previousCallResult, ZeroPaddedSpan previousCallOutput, UInt256& previousCallOutputDestination, IReleaseSpec spec)
   at Nethermind.Evm.VirtualMachine.Run(EvmState state, IWorldState worldState, ITxTracer txTracer)
   at Nethermind.Evm.TransactionProcessing.TransactionProcessor.Execute(Transaction transaction, BlockHeader block, ITxTracer txTracer, ExecutionOptions executionOptions)
   at Nethermind.Consensus.Processing.TransactionProcessorAdapterExtensions.ProcessTransaction(ITransactionProcessorAdapter transactionProcessor, Block block, Transaction currentTx, BlockReceiptsTracer receiptsTracer, ProcessingOptions processingOptions, IStateProvider stateProvider)
   at Nethermind.Consensus.Processing.BlockProcessor.BlockValidationTransactionsExecutor.ProcessTransactions(Block block, ProcessingOptions processingOptions, BlockReceiptsTracer receiptsTracer, IReleaseSpec spec)
   at Nethermind.Consensus.Processing.BlockProcessor.ProcessBlock(Block block, IBlockTracer blockTracer, ProcessingOptions options)
   at Nethermind.Consensus.Processing.BlockProcessor.ProcessOne(Block suggestedBlock, ProcessingOptions options, IBlockTracer blockTracer)
   at Nethermind.Consensus.Processing.BlockProcessor.Process(Keccak newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer)
   at Nethermind.Consensus.Processing.BlockchainProcessor.ProcessBranch(ProcessingBranch processingBranch, ProcessingOptions options, IBlockTracer tracer)
   at Nethermind.Consensus.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer)
   at Nethermind.Consensus.Processing.BlockchainProcessor.RunProcessingLoop()

Expected behavior continue syncing

Desktop (please complete the following information):

marcindsobczak commented 2 years ago

That's very worrisome, we had problems with TrieException in the past, looks like it's not fully fixed. Would you be able to send full logs? It will help us to find source of the problem.

dasudiy commented 2 years ago

mainnet.logs.txt.zip It's my latest logs file

LukaszRozmej commented 4 months ago

old pruning issue probably