search

NethermindEth / nethermind

A robust execution client for Ethereum node operators.
https://nethermind.io/nethermind-client
GNU General Public License v3.0
1.25k stars 430 forks source link

DB corruption #5509

Open noskill opened 1 year ago

noskill commented 1 year ago

I run nethermind in docker from this guide https://hackmd.io/@philknows/HkROkZW55#Option-13-Nethermind-Execution-Node

Each time i create container and run nethermind it gets corrupted db at some pont:

2023-03-28 21:36:28.0917|Syncing beacon headers... Request: ForkchoiceState: (HeadBlockHash: 0 xe82447a37eee530bc9be695b63d95f33c12cec98f5d0f2e029ef21ab81c8e3ec, SafeBlockHash: 0xc6a8c81161 30b58e4a765adb0f6077189d8afec6ffacb5602832f84bf016ea50, FinalizedBlockHash: 0x794af1120494d593 1ae614601e17b9d35208c9b05c71bc33e720736712670fb8) . 2023-03-28 21:36:28.5972|Corrupted DB detected on path /data/nethermind_db/mainnet/code. Pleas e restart Nethermind to attempt repair. 2023-03-28 21:36:28.7673|Corrupted DB detected on path /data/nethermind_db/mainnet/code. Pleas e restart Nethermind to attempt repair. 2023-03-28 21:36:28.8132|Error when handling response RocksDbSharp.RocksDbException: Corruption: block checksum mismatch: stored = 2559870909, computed = 722275884, type = 1 in /data/nethermind_db/mainnet/code/001460.sst offset 6098955 size 14268 at Nethermind.Db.Rocks.DbOnTheRocks.set_Item(Byte[] key, Byte[] value) in /src/Nethermind/Nethermind.Db.Rocks/DbOnTheRocks.cs:line 360 at Nethermind.Synchronization.SnapSync.SnapProvider.AddCodes(Keccak[] requestedHashes, Byte[][] codes) in /src/Nethermind/Nethermind.Synchronization/SnapSync/SnapProvider.cs:line 252 at Nethermind.Synchronization.SnapSync.SnapSyncFeed.HandleResponse(SnapSyncBatch batch, PeerInfo peer) in /src/Nethermind/Nethermind.Synchronization/SnapSync/SnapSyncFeed.cs:line 72 at Nethermind.Synchronization.ParallelSync.SyncDispatcher`1.DoHandleResponse(T request, PeerInfo allocatedPeer) in /src/Nethermind/Nethermind.Synchronization/ParallelSync/SyncDispatcher.cs:line 173

MarekM25 commented 1 year ago

@noskill is it still the issue? Did you try the latest version of Nethermind?

kamilchodola commented 8 months ago

I think we solved plenty of things around corruptions - if no response we can close. @noskill Please check if this is still a case for You.

noskill commented 8 months ago

I can't reproduce the issue, i got some problems with beacon node:

beacon_node_1 | CodeError: Transport (@libp2p/tcp) could not listen on any available address nethermind_docker | 18 Jan 14:26:10 | Peers | with best block: 38 | all: 38 | eth66 (8 %), eth67 (5 %), eth68 (87 %) | Active: None | Sleeping: All | Geth (50 %), Erigon (21 %), Nethermind (21 %), OpenEthereum (5 %), Besu (3 %) nethermind_docker | 18 Jan 14:26:10 | No incoming messages from the consensus client that is required for sync.

kamilchodola commented 8 months ago

@noskill can You send some more CL logs for analysis? Which one is that?

noskill commented 8 months ago

sure beacon logs

nethermind logs

kamilchodola commented 8 months ago

@noskill Don't You have any other CL running on similar ports? Seems liek Lodestar can't acquire port for libp2p communication?

noskill commented 8 months ago

Indeed i had another process that use libp2p running. So far beacon and nethermind show no errors.

noskill commented 8 months ago

Sync is stack at 7.83% for 10 hours or so:

19 Jan 20:13:16 | Sync mode changed from SnapSync to FastSync, SnapSync 
19 Jan 20:13:16 | Snap         State Ranges (Phase 1): (  7.83 %) [*****                                                                  ] 
19 Jan 20:13:17 | Changing state FastSync, SnapSync to SnapSync at processed: 0 | state: 0 | block: 19043191 | header: 19043191 | target block: 19043223 | peer block: 19043223 
19 Jan 20:13:17 | Sync mode changed from FastSync, SnapSync to SnapSync 
19 Jan 20:13:17 | Snap         State Ranges (Phase 1): (  7.83 %) [*****                                                                  ] 
......................................................

0 Jan 07:14:37 | Received new block:  19046512 (0x31b8fc...763026) 
20 Jan 07:14:37 | Syncing... Inserting block 19046512 (0x31b8fc...763026). 
20 Jan 07:14:37 | Received ForkChoice: Head: 19046512 (0x31b8fc...763026), Safe: 19046469 (0x65d53b...ea9675), Finalized: 19046437 (0x6507d9...899371) 
20 Jan 07:14:37 | Syncing beacon headers, Request: ForkChoice: Head: 19046512 (0x31b8fc...763026), Safe: 19046469 (0x65d53b...ea9675), Finalized: 19046437 (0x6507d9...899371) 
20 Jan 07:14:37 | Snap         State Ranges (Phase 1): (  7.83 %) [*****                                                                  ] 
20 Jan 07:14:38 | Changing state SnapSync to FastSync, SnapSync at processed: 0 | state: 0 | block: 19046479 | header: 19046479 | target block: 19046512 | peer block: 19046512 
20 Jan 07:14:38 | Sync mode changed from SnapSync to FastSync, SnapSync 
20 Jan 07:14:39 | Changing state FastSync, SnapSync to SnapSync at processed: 0 | state: 0 | block: 19046480 | header: 19046480 | target block: 19046512 | peer block: 19046512 
20 Jan 07:14:39 | Sync mode changed from FastSync, SnapSync to SnapSync 
20 Jan 07:14:39 | Snap         State Ranges (Phase 1): (  7.83 %) [*****