search

streamingfast / firehose-ethereum

Ethereum on StreamingFast
Apache License 2.0
78 stars 24 forks source link

(mindreader-node) WARN [11-18|20:48:51.723] Failed to load old bad blocks error="leveldb: not found" (log_plugin/to_zap_log_plugin.go:131) #4

Closed matthewdarwin closed 2 years ago

matthewdarwin commented 2 years ago

geth with mindreader crashed

$ sfeth --version sfeth version -dev

matthewdarwin commented 2 years ago

syslog-firehose-crash-2021-11-18.log

matthewdarwin commented 2 years ago

Matt | StreamingFast — Today at 9:07 AM This is a message from Geth directly, not clear why it seems that some low level LevelDB files are not found anymore.

Had you done in operational stuff (backup?) recently? Had you issue any Mindreader management command? Anything else prior that that could indicate some form of maintenance operation?

matthewdarwin commented 2 years ago

Was just running fine and then wasn't anymore. It is not clear why. I

maoueh commented 2 years ago

Ok I read the whole log.

It seems the levedb: not found is more a real warning than anything else. After that there is an error without any message:

Nov 18 20:48:57 eth-sfdm25 sfeth[72]: 2021-11-18T20:48:57.424Z (mindreader-node) ERROR[11-18|20:48:57.423]  (log_plugin/to_zap_log_plugin.go:131)

And finally the real culprit is this:

Nov 18 20:48:57 eth-sfdm25 sfeth[72]: 2021-11-18T20:48:57.420Z (mindreader-node) failed storing block in archiver, shutting down. You will need to reprocess over this range to get this block (mindreader/mindreader.go:275){"error": "blocks non contiguous, expectedBlock: 13107519, got block: 13107518", "block": "#13107518 (3236a68532faf89e844eb555e323047c0fe0adc147c27d3738f061f007ac1b05)"}

It appears the non-contiguous check is a bit broken for Ethereum since it seems bad block can be re-processed breaking the assumption that we will always go forward monotonically:

Nov 18 20:48:57 eth-sfdm25 sfeth[72]: 2021-11-18T20:48:57.418Z (mindreader-node) INFO [11-18|20:48:57.418] Skip duplicated bad block                number=13,107,518 hash=3236a6..ac1b05 (log_plugin/to_zap_log_plugin.go:131)

This is a message from Geth saying it's skipping duplicated block. This probably explains the leveldb: not found warning. It tried to fetch bad block but did not find it.

Tomorrow I'll change the logic to accept re-processing the same block as the one expected. We might need to disable the check completely depending on the behavior.

maoueh commented 2 years ago

I was about to fix that but the code was refactored recently and the check is gone.

However, I feel it still a good thing to ensure we do not miss any block at all when auto-merging files in mindreader. I'm checking the status of this and will report back later.

sduchesneau commented 2 years ago

Since the issue is resolved (by removing the continuity checker), I'm closing this issue. The improvements that you "feel would be a good thing" are now here: https://github.com/streamingfast/node-manager/issues/16