search

NethermindEth / nethermind

A robust execution client for Ethereum node operators.
https://nethermind.io/nethermind-client
GNU General Public License v3.0
1.25k stars 431 forks source link

Prune corrupts databse every time I run it #5643

Closed wojciech-skowronek closed 1 year ago

wojciech-skowronek commented 1 year ago

Description After running full prune my database is corrupted

Steps to Reproduce My hardware:

Memory settings:

2023-05-02 21:26:50.9817|Setting up memory allowances 
2023-05-02 21:26:50.9817|  memory hint:        1024MB 
2023-05-02 21:26:50.9817|  general memory:     32MB 
2023-05-02 21:26:50.9817|  peers memory:       50MB 
2023-05-02 21:26:50.9915|  Netty memory:       134MB 
2023-05-02 21:26:50.9915|  mempool memory:     122MB 
2023-05-02 21:26:50.9915|  fast blocks memory: 68MB 
2023-05-02 21:26:50.9915|  trie memory:        123MB 
2023-05-02 21:26:50.9950|  DB memory:          493MB

In order to replicate the behavior, please provide a detailed list of steps:

  1. Start full prune

Actual behavior When prune finishes I'm getting following information that all is fine:

2023-05-02 21:26:29.7073|Peers | with known best block: 49 | all: 49 | 
2023-05-02 21:26:32.6840|Full Pruning Finished: 14:06:42.7608290 1,595.64 mln nodes mirrored. 
2023-05-02 21:26:36.9720|Received a new payload: 17175745 (0xd6e7ce96e269db381128821be896347814af26f3eb45cdf9cf56843bca92ea40) 
2023-05-02 21:26:37.2166|Processed   17175745 |      241ms of  11,118ms, mgasps  173.78 total   52.19, tps 1587.54 total  569.77, bps    4.15 total    1.72, recv queue 0, proc queue 0 
2023-05-02 21:26:37.2166|Valid. Result of a new payload: 17175745 (0xd6e7ce96e269db381128821be896347814af26f3eb45cdf9cf56843bca92ea40). 
2023-05-02 21:26:37.2928|Received: ForkchoiceState: (HeadBlockHash: 0xd6e7ce96e269db381128821be896347814af26f3eb45cdf9cf56843bca92ea40, SafeBlockHash: 0x2a309322156ad23597dc28acd5b067d4f8b22901f9b9037f8367e88ab07c3a93, FinalizedBlockHash: 0x03bc4168b2206d9e37156f5ce3c3d24752ecd
27cd58667838b7272b5b352896e)  
2023-05-02 21:26:37.2928|FCU - block 17175745 (0xd6e7ce...92ea40) was processed. 
2023-05-02 21:26:37.3030|Disposing DB State0 
2023-05-02 21:26:42.9621|Full Pruning completed successfully, shutting down as requested in the configuration. 
2023-05-02 21:26:42.9621|Block 0xd6e7ce96e269db381128821be896347814af26f3eb45cdf9cf56843bca92ea40 was set as head. 
2023-05-02 21:26:42.9621|Valid. Request: ForkchoiceState: (HeadBlockHash: 0xd6e7ce96e269db381128821be896347814af26f3eb45cdf9cf56843bca92ea40, SafeBlockHash: 0x2a309322156ad23597dc28acd5b067d4f8b22901f9b9037f8367e88ab07c3a93, FinalizedBlockHash: 0x03bc4168b2206d9e37156f5ce3c3d24
752ecd27cd58667838b7272b5b352896e) . 
2023-05-02 21:26:42.9740|Closing, please wait until all functions are stopped properly... 

...

2023-05-02 21:26:45.2468|Closing DBs... 
2023-05-02 21:26:45.2468|Disposing DB Witness 
2023-05-02 21:26:45.2468|Disposing DB CanonicalHashTrie 
2023-05-02 21:26:45.2503|Disposing DB Metadata 
2023-05-02 21:26:45.2851|Disposing DB Headers 
2023-05-02 21:26:45.3408|Disposing DB Bloom 
2023-05-02 21:26:45.3705|Disposing DB Receipts 
2023-05-02 21:26:46.7947|Disposing DB BlockInfos 
2023-05-02 21:26:46.8501|Disposing DB State1 
2023-05-02 21:26:47.0022|Disposing DB Code 
2023-05-02 21:26:47.0436|Disposing DB Blocks 
2023-05-02 21:26:48.0887|All DBs closed. 
2023-05-02 21:26:48.0887|Ethereum shutdown complete... please wait for all components to close 
2023-05-02 21:26:48.0887|All done, goodbye! 
2023-05-02 21-26-50.1950|Nethermind starting initialization.

After automatic restart I'm getting following error:

2023-05-02 21:33:36.0780|Received a new payload: 17175780 (0x93aa7dc5e1aad250aa87f620dffe0e1a5b878b43b39552060e2ccd9adf3a77fb) 
2023-05-02 21:33:36.2603|Processed   17175780 |      179ms of   9,966ms, mgasps  238.79 total   96.40, tps 2106.31 total  997.88, bps    5.59 total    2.95, recv queue 0, proc queue 0 
2023-05-02 21:33:36.2603|Valid. Result of a new payload: 17175780 (0x93aa7dc5e1aad250aa87f620dffe0e1a5b878b43b39552060e2ccd9adf3a77fb). 
2023-05-02 21:33:36.3276|Received: ForkchoiceState: (HeadBlockHash: 0x93aa7dc5e1aad250aa87f620dffe0e1a5b878b43b39552060e2ccd9adf3a77fb, SafeBlockHash: 0x20e3897863d0bceae8514948e1ad173cf6eea6a9f6414e8bde66cbf74681ffc3, FinalizedBlockHash: 0x2a309322156ad23597dc28acd5b067d4f8b22
901f9b9037f8367e88ab07c3a93)  
2023-05-02 21:33:36.3276|FCU - block 17175780 (0x93aa7d...3a77fb) was processed. 
2023-05-02 21:33:36.3276|Block 0x93aa7dc5e1aad250aa87f620dffe0e1a5b878b43b39552060e2ccd9adf3a77fb was set as head. 
2023-05-02 21:33:36.3276|Valid. Request: ForkchoiceState: (HeadBlockHash: 0x93aa7dc5e1aad250aa87f620dffe0e1a5b878b43b39552060e2ccd9adf3a77fb, SafeBlockHash: 0x20e3897863d0bceae8514948e1ad173cf6eea6a9f6414e8bde66cbf74681ffc3, FinalizedBlockHash: 0x2a309322156ad23597dc28acd5b067d
4f8b22901f9b9037f8367e88ab07c3a93) . 
2023-05-02 21:33:41.9605|Peers | with known best block: 48 | all: 48 | 
2023-05-02 21:33:48.2710|Received a new payload: 17175781 (0x9c77508333aff85ed78c82ee73ea48131b45b3aa6147f9dad846e192e8fea339) 
2023-05-02 21:33:48.4342|Processing loop threw an exception. Block: 17175781 (0x9c7750...fea339), Exception: Nethermind.Trie.TrieException: Failed to load key 5b9645eb80585256a30a1240ba235751a2a2fc30d2fa7a7bc816699652e03bfd from root hash 0x3543d6af6ac2fb00dbcaff3a661332960d8da
1c980de2fe1c3c45080e70b94c5.
 ---> Nethermind.Trie.TrieException: Node 0x4f4db6b987e53a39acfb2a36865f1542de709531f30b7e65a5fbbbb6332ed9c5 is missing from the DB
   at Nethermind.Trie.Pruning.TrieStore.LoadRlp(Keccak keccak, IKeyValueStore keyValueStore) in /src/Nethermind/Nethermind.Trie/Pruning/TrieStore.cs:line 320
   at Nethermind.Trie.TrieNode.ResolveNode(ITrieNodeResolver tree) in /src/Nethermind/Nethermind.Trie/TrieNode.cs:line 302
   at Nethermind.Trie.PatriciaTree.TraverseBranch(TrieNode node, TraverseContext traverseContext) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 647
   at Nethermind.Trie.PatriciaTree.TraverseNode(TrieNode node, TraverseContext traverseContext) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 409
   at Nethermind.Trie.PatriciaTree.TraverseBranch(TrieNode node, TraverseContext traverseContext) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 647
   at Nethermind.Trie.PatriciaTree.TraverseNode(TrieNode node, TraverseContext traverseContext) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 409
   at Nethermind.Trie.PatriciaTree.TraverseBranch(TrieNode node, TraverseContext traverseContext) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 647
   at Nethermind.Trie.PatriciaTree.TraverseNode(TrieNode node, TraverseContext traverseContext) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 409
   at Nethermind.Trie.PatriciaTree.TraverseBranch(TrieNode node, TraverseContext traverseContext) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 647
   at Nethermind.Trie.PatriciaTree.TraverseNode(TrieNode node, TraverseContext traverseContext) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 409
   at Nethermind.Trie.PatriciaTree.TraverseBranch(TrieNode node, TraverseContext traverseContext) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 647
   at Nethermind.Trie.PatriciaTree.TraverseNode(TrieNode node, TraverseContext traverseContext) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 409
   at Nethermind.Trie.PatriciaTree.Get(Span`1 rawKey, Keccak rootHash) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 317
   --- End of inner exception stack trace ---
   at Nethermind.Trie.PatriciaTree.Get(Span`1 rawKey, Keccak rootHash) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 317
   at Nethermind.State.StateProvider.GetAndAddToCache(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 657
   at Nethermind.State.StateProvider.GetAccount(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 103
   at Nethermind.Evm.TransactionProcessing.TransactionProcessor.Execute(Transaction transaction, BlockHeader block, ITxTracer txTracer, ExecutionOptions executionOptions) in /src/Nethermind/Nethermind.Evm/TransactionProcessing/TransactionProcessor.cs:line 504
   at Nethermind.Consensus.Processing.BlockProcessor.ProcessBlock(Block block, IBlockTracer blockTracer, ProcessingOptions options) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 227
   at Nethermind.Consensus.Processing.BlockProcessor.Process(Keccak newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 146
   at Nethermind.Consensus.Processing.BlockchainProcessor.ProcessBranch(ProcessingBranch processingBranch, ProcessingOptions options, IBlockTracer tracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 495
   at Nethermind.Consensus.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 327
   at Nethermind.Consensus.Processing.BlockchainProcessor.RunProcessingLoop() in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 275 
2023-05-02 21:33:48.4342|engine_newPayloadV2 failed: Nethermind.Blockchain.BlockchainException: Block processing threw exception.
 ---> Nethermind.Trie.TrieException: Failed to load key 5b9645eb80585256a30a1240ba235751a2a2fc30d2fa7a7bc816699652e03bfd from root hash 0x3543d6af6ac2fb00dbcaff3a661332960d8da1c980de2fe1c3c45080e70b94c5.
 ---> Nethermind.Trie.TrieException: Node 0x4f4db6b987e53a39acfb2a36865f1542de709531f30b7e65a5fbbbb6332ed9c5 is missing from the DB
   at Nethermind.Trie.Pruning.TrieStore.LoadRlp(Keccak keccak, IKeyValueStore keyValueStore) in /src/Nethermind/Nethermind.Trie/Pruning/TrieStore.cs:line 320
   at Nethermind.Trie.TrieNode.ResolveNode(ITrieNodeResolver tree) in /src/Nethermind/Nethermind.Trie/TrieNode.cs:line 302
   at Nethermind.Trie.PatriciaTree.TraverseBranch(TrieNode node, TraverseContext traverseContext) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 647
   at Nethermind.Trie.PatriciaTree.TraverseNode(TrieNode node, TraverseContext traverseContext) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 409
   at Nethermind.Trie.PatriciaTree.TraverseBranch(TrieNode node, TraverseContext traverseContext) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 647
   at Nethermind.Trie.PatriciaTree.TraverseNode(TrieNode node, TraverseContext traverseContext) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 409
   at Nethermind.Trie.PatriciaTree.TraverseBranch(TrieNode node, TraverseContext traverseContext) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 647
   at Nethermind.Trie.PatriciaTree.TraverseNode(TrieNode node, TraverseContext traverseContext) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 409
   at Nethermind.Trie.PatriciaTree.TraverseBranch(TrieNode node, TraverseContext traverseContext) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 647
   at Nethermind.Trie.PatriciaTree.TraverseNode(TrieNode node, TraverseContext traverseContext) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 409
   at Nethermind.Trie.PatriciaTree.TraverseBranch(TrieNode node, TraverseContext traverseContext) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 647
   at Nethermind.Trie.PatriciaTree.TraverseNode(TrieNode node, TraverseContext traverseContext) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 409
   at Nethermind.Trie.PatriciaTree.Get(Span`1 rawKey, Keccak rootHash) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 317
   --- End of inner exception stack trace ---
   at Nethermind.Trie.PatriciaTree.Get(Span`1 rawKey, Keccak rootHash) in /src/Nethermind/Nethermind.Trie/PatriciaTree.cs:line 317
   at Nethermind.State.StateProvider.GetAndAddToCache(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 657
   at Nethermind.State.StateProvider.GetAccount(Address address) in /src/Nethermind/Nethermind.State/StateProvider.cs:line 103
   at Nethermind.Evm.TransactionProcessing.TransactionProcessor.Execute(Transaction transaction, BlockHeader block, ITxTracer txTracer, ExecutionOptions executionOptions) in /src/Nethermind/Nethermind.Evm/TransactionProcessing/TransactionProcessor.cs:line 504
   at Nethermind.Consensus.Processing.BlockProcessor.ProcessBlock(Block block, IBlockTracer blockTracer, ProcessingOptions options) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 227
   at Nethermind.Consensus.Processing.BlockProcessor.Process(Keccak newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockProcessor.cs:line 146
   at Nethermind.Consensus.Processing.BlockchainProcessor.ProcessBranch(ProcessingBranch processingBranch, ProcessingOptions options, IBlockTracer tracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 495
   at Nethermind.Consensus.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer) in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 327
   at Nethermind.Consensus.Processing.BlockchainProcessor.RunProcessingLoop() in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 275
   --- End of inner exception stack trace ---
   at Nethermind.Merge.Plugin.TimeoutUtils.TimeoutOn[T](Task`1 task, Task timeoutTask) in /src/Nethermind/Nethermind.Merge.Plugin/TimeoutUtils.cs:line 20
   at Nethermind.Merge.Plugin.Handlers.NewPayloadHandler.ValidateBlockAndProcess(Block block, BlockHeader parent, ProcessingOptions processingOptions) in /src/Nethermind/Nethermind.Merge.Plugin/Handlers/NewPayloadHandler.cs:line 379
   at Nethermind.Merge.Plugin.Handlers.NewPayloadHandler.HandleAsync(ExecutionPayload request) in /src/Nethermind/Nethermind.Merge.Plugin/Handlers/NewPayloadHandler.cs:line 208
   at Nethermind.Merge.Plugin.EngineRpcModule.NewPayload(ExecutionPayload executionPayload, Int32 version) in /src/Nethermind/Nethermind.Merge.Plugin/EngineRpcModule.Paris.cs:line 97

Manual restarts are not helping and I'm getting same behavior (few correct responses and then DB errors)

Expected behavior All works fine

Desktop (please complete the following information): Please provide the following information regarding your setup:

Additional context I'm running my node through rocketpool

asdacap commented 1 year ago

Hello, can you try again with 1.18?

wojciech-skowronek commented 1 year ago

Works with 1.18. Thanks!