search

Fantom-foundation / go-opera

Opera blockchain protocol secured by the Lachesis consensus algorithm
https://fantom.foundation/
GNU Lesser General Public License v3.0
278 stars 282 forks source link

Node did not receive new block events anymore, had to restart #543

Closed flolege closed 3 months ago

flolege commented 5 months ago

Describe the bug My read only node got stuck, meaning no new block events and therefore no pool reorgs, mempool piled up.

To Reproduce No idea how to reproduce that

Expected behavior Node does not get stuck

Screenshots Here are the last few loglines: WARN [04-13|06:59:34.865] Pre-internal transaction reverted txid=0xafbd6d03a6acfb591e90e8e5d89dfffa61f3ceb6351648e814aef6e933c17882 WARN [04-13|06:59:34.865] Internal transaction reverted txid=0xa8c423405c7cb80923925b3b1104c75326e02f43a6705dbde9d74feefbd0b157 INFO [04-13|06:59:34.889] New block index=79264528 Atropos=276163:6035:e57490 root="268b90…69cb73" gas_used=3,734,054 txs=4/0 age=3.122s t=24.466ms WARN [04-13|06:59:35.790] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" WARN [04-13|06:59:35.791] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" WARN [04-13|06:59:35.794] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" WARN [04-13|06:59:35.796] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" WARN [04-13|06:59:35.806] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" WARN [04-13|06:59:35.807] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" INFO [04-13|06:59:36.115] New DAG summary new=174 last_id=276164:1:652fdc age=340.215ms t=219.406ms WARN [04-13|06:59:36.115] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" WARN [04-13|06:59:36.655] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" WARN [04-13|06:59:37.423] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" WARN [04-13|06:59:37.643] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" WARN [04-13|06:59:37.996] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" WARN [04-13|06:59:38.056] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" WARN [04-13|06:59:38.356] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" WARN [04-13|06:59:38.398] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" WARN [04-13|06:59:38.545] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" WARN [04-13|06:59:40.462] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash"

Desktop (please complete the following information): Ubuntu

flolege commented 5 months ago

Can't start my node. Here is the log:

INFO [04-13|08:27:08.358] New LLR summary last_epoch=0 last_block=79266627 new_evs=0 new_ers=0 new_bvs=60 new_brs=0 age=none WARN [04-13|08:27:09.895] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" INFO [04-13|08:27:18.393] New LLR summary last_epoch=276173 last_block=79266627 new_evs=14 new_ers=2 new_bvs=300 new_brs=0 age=9m10.124s INFO [04-13|08:27:18.393] New DAG summary new=1 last_id=276164:1:652fdc age=1h27m42.617s t="104.608µs" INFO [04-13|08:27:42.020] New LLR summary last_epoch=276173 last_block=79267103 new_evs=0 new_ers=0 new_bvs=2712 new_brs=0 age=9m33.752s INFO [04-13|08:27:42.020] New DAG summary new=1 last_id=276164:1:652fdc age=1h28m6.245s t="79.724µs" WARN [04-13|08:27:42.020] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" INFO [04-13|08:28:34.343] New LLR summary last_epoch=276173 last_block=79267115 new_evs=0 new_ers=0 new_bvs=12 new_brs=0 age=10m26.074s WARN [04-13|08:28:34.343] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" INFO [04-13|08:28:38.095] Looking for peers peercount=17 tried=2260 static=0 INFO [04-13|08:28:56.114] New LLR summary last_epoch=276174 last_block=79267115 new_evs=7 new_ers=1 new_bvs=72 new_brs=0 age=1m13.832s INFO [04-13|08:28:56.114] New DAG summary new=1 last_id=276164:1:652fdc age=1h29m20.339s t="111.512µs" WARN [04-13|08:28:56.149] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" INFO [04-13|08:29:30.109] New LLR summary last_epoch=276174 last_block=79267115 new_evs=0 new_ers=0 new_bvs=60 new_brs=0 age=1m47.827s INFO [04-13|08:29:30.109] New DAG summary new=2 last_id=276164:1:652fdc age=1h29m54.334s t="167.741µs" WARN [04-13|08:29:30.109] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" INFO [04-13|08:30:12.380] New DAG summary new=1 last_id=276164:1:652fdc age=1h30m36.605s t="75.2µs" WARN [04-13|08:30:12.380] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" INFO [04-13|08:30:38.112] Looking for peers peercount=17 tried=2191 static=0 INFO [04-13|08:31:07.232] New DAG summary new=1 last_id=276164:1:652fdc age=1h31m31.457s t="72.918µs" WARN [04-13|08:31:07.232] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" WARN [04-13|08:31:13.128] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" INFO [04-13|08:32:38.252] New LLR summary last_epoch=276174 last_block=79267179 new_evs=0 new_ers=0 new_bvs=64 new_brs=0 age=4m55.970s INFO [04-13|08:32:38.252] New DAG summary new=1 last_id=276164:1:652fdc age=1h33m2.477s t="69.742µs" WARN [04-13|08:32:38.414] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" INFO [04-13|08:35:39.795] New LLR summary last_epoch=276174 last_block=79267243 new_evs=0 new_ers=0 new_bvs=320 new_brs=0 age=7m57.512s INFO [04-13|08:35:39.795] New DAG summary new=1 last_id=276164:1:652fdc age=1h36m4.019s t="87.968µs" INFO [04-13|08:35:51.339] New LLR summary last_epoch=276174 last_block=79267307 new_evs=0 new_ers=0 new_bvs=576 new_brs=0 age=8m9.057s INFO [04-13|08:35:51.339] New DAG summary new=1 last_id=276164:1:652fdc age=1h36m15.563s t="75.057µs" WARN [04-13|08:35:51.339] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" INFO [04-13|08:36:29.122] New DAG summary new=1 last_id=276164:1:652fdc age=1h36m53.346s t="70.798µs" WARN [04-13|08:36:29.122] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" INFO [04-13|08:36:38.262] New DAG summary new=1 last_id=276164:1:652fdc age=1h37m2.486s t="84.212µs" WARN [04-13|08:36:38.262] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" INFO [04-13|08:36:53.106] New LLR summary last_epoch=276174 last_block=79267341 new_evs=0 new_ers=0 new_bvs=34 new_brs=0 age=9m10.824s INFO [04-13|08:37:05.106] New LLR summary last_epoch=276174 last_block=79267341 new_evs=0 new_ers=0 new_bvs=340 new_brs=0 age=9m22.823s INFO [04-13|08:37:21.122] New LLR summary last_epoch=276175 last_block=79267341 new_evs=5 new_ers=1 new_bvs=306 new_brs=0 age=37.448s INFO [04-13|08:37:21.122] New DAG summary new=1 last_id=276164:1:652fdc age=1h37m45.347s t="113.779µs" WARN [04-13|08:37:21.122] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" INFO [04-13|08:37:46.095] New DAG summary new=1 last_id=276164:1:652fdc age=1h38m10.320s t="72.824µs" WARN [04-13|08:37:46.095] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" INFO [04-13|08:39:01.360] New LLR summary last_epoch=276175 last_block=79267341 new_evs=0 new_ers=0 new_bvs=34 new_brs=0 age=2m17.686s WARN [04-13|08:39:01.898] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" INFO [04-13|08:39:44.340] New LLR summary last_epoch=276175 last_block=79267405 new_evs=0 new_ers=0 new_bvs=456 new_brs=0 age=3m666.592ms INFO [04-13|08:39:44.340] New DAG summary new=2 last_id=276164:1:652fdc age=1h40m8.565s t="191.352µs" WARN [04-13|08:39:44.340] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" INFO [04-13|08:41:05.093] New LLR summary last_epoch=276175 last_block=79267469 new_evs=0 new_ers=0 new_bvs=64 new_brs=0 age=4m21.419s INFO [04-13|08:42:15.093] New LLR summary last_epoch=276175 last_block=79267533 new_evs=0 new_ers=0 new_bvs=320 new_brs=0 age=5m31.419s WARN [04-13|08:42:22.365] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" INFO [04-13|08:43:31.527] New LLR summary last_epoch=276175 last_block=79267597 new_evs=0 new_ers=0 new_bvs=320 new_brs=0 age=6m47.853s INFO [04-13|08:43:31.527] New DAG summary new=1 last_id=276164:1:652fdc age=1h43m55.752s t="74.34µs" INFO [04-13|08:44:14.114] New LLR summary last_epoch=276176 last_block=79267597 new_evs=1 new_ers=0 new_bvs=256 new_brs=0 age=7m30.440s INFO [04-13|08:44:46.001] New LLR summary last_epoch=276176 last_block=79267597 new_evs=4 new_ers=1 new_bvs=0 new_brs=0 age=39.397s INFO [04-13|08:44:46.001] New DAG summary new=1 last_id=276164:1:652fdc age=1h45m10.226s t="82.941µs" WARN [04-13|08:44:46.001] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" INFO [04-13|08:46:16.094] New LLR summary last_epoch=276176 last_block=79267711 new_evs=0 new_ers=0 new_bvs=314 new_brs=0 age=2m9.489s INFO [04-13|08:46:52.113] New LLR summary last_epoch=276176 last_block=79267775 new_evs=0 new_ers=0 new_bvs=320 new_brs=0 age=2m45.508s INFO [04-13|08:47:23.345] New LLR summary last_epoch=276176 last_block=79267775 new_evs=0 new_ers=0 new_bvs=256 new_brs=0 age=3m16.740s INFO [04-13|08:47:23.345] New DAG summary new=1 last_id=276164:1:652fdc age=1h47m47.569s t="75.897µs" WARN [04-13|08:47:23.345] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" INFO [04-13|08:48:23.170] New DAG summary new=1 last_id=276164:1:652fdc age=1h48m47.395s t="76.756µs" WARN [04-13|08:48:23.170] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" WARN [04-13|08:48:29.095] Incoming event rejected event=276164:1:652fdc creator=17 err="wrong event epoch hash" INFO [04-13|08:48:33.092] New LLR summary last_epoch=276176 last_block=79267839 new_evs=0 new_ers=0 new_bvs=64 new_brs=0 age=4m26.488s INFO [04-13|08:48:33.092] New DAG summary new=1 last_id=276164:1:652fdc age=1h48m57.317s t="70.192µs"

flolege commented 3 months ago

I had to throw away the DB.

flolege commented 3 months ago

Closing

khoau2u commented 3 months ago

@flolege Did you found out the reason why the DB is corrupted? How can we reproduce this?

flolege commented 3 months ago

It was my mistake, a coding error on my side.