search

erigontech / erigon

Ethereum implementation on the efficiency frontier https://erigon.gitbook.io
GNU Lesser General Public License v3.0
3.11k stars 1.09k forks source link

[e3] FindBlockNum(...): seems broken TxNum value #12009

Open wmitsuda opened 2 days ago

wmitsuda commented 2 days ago

Using today's main, sepolia.

Node was working fine, then suddenly it entered an unrecoverable state. Restart doesn't work, it seems DB got inconsistent.

INFO[09-16|18:33:26.485] P2P                                      app=caplin peers=0
INFO[09-16|18:33:33.695] head updated                             head=0x62f81df94163466eb6024365e79010317db27e396e1253a6f7d5ce11749cc2f2 hash
=0x62f81df94163466eb6024365e79010317db27e396e1253a6f7d5ce11749cc2f2 number=6705172 execution=58.656917ms mgas/s=217.76 average mgas/s=215.79 c
ommit=189.25µs alloc=5.3GB sys=11.3GB
INFO[09-16|18:33:44.785] head updated                             head=0x76829ae6ed935537200f8cbabb98f5c18a0032ca2e732e06653075613b86dcf1 hash
=0x76829ae6ed935537200f8cbabb98f5c18a0032ca2e732e06653075613b86dcf1 number=6705173 execution=94.075459ms mgas/s=131.49 average mgas/s=194.72 c
ommit=1.967458ms alloc=4.0GB sys=11.3GB
INFO[09-16|18:33:59.302] head updated                             head=0xf6ee4b72b7c13012d94e4d181eae84045dcda197d8538a568b59374d8b15ee69 hash
=0xf6ee4b72b7c13012d94e4d181eae84045dcda197d8538a568b59374d8b15ee69 number=6705174 execution=41.8355ms mgas/s=228.76 average mgas/s=201.53 com
mit=182.666µs alloc=4.2GB sys=11.3GB
INFO[09-16|18:34:04.566] head updated                             head=0xa1c43d17f238bc8da86e26c3704544316cb721bde0e03a99db5ccd97e89b1607 hash
=0xa1c43d17f238bc8da86e26c3704544316cb721bde0e03a99db5ccd97e89b1607 number=6705175 execution=183.084166ms mgas/s=163.61 average mgas/s=195.21
commit=254.833µs alloc=4.6GB sys=11.3GB
INFO[09-16|18:34:20.320] head updated                             head=0x65f3d7a6481e1a9cb8cbc1b9e6172f24d72b8b688952ba5c67be13a5a0bd723d hash
=0x65f3d7a6481e1a9cb8cbc1b9e6172f24d72b8b688952ba5c67be13a5a0bd723d number=6705176 execution=47.73225ms mgas/s=177.08 average mgas/s=192.62 co
mmit=111.583µs alloc=5.0GB sys=11.3GB
INFO[09-16|18:34:26.486] P2P                                      app=caplin peers=0
INFO[09-16|18:34:57.063] Adding light client update               period=720
INFO[09-16|18:34:57.488] [4/6 Execution] Done Commit every block  blk=6705179 blks=3 blk/s=18.1 txs=305 tx/s=1.83k gas/s=258.55M buf=5.3MB/512
.0MB stepsInDB=0.00 step=198.0 alloc=4.5GB sys=11.3GB
INFO[09-16|18:34:57.746] head updated                             head=0x2ef5003ec9ef3da04805f0767c87ec7f91430499c21577f4758387efaed295d0 hash
=0x2ef5003ec9ef3da04805f0767c87ec7f91430499c21577f4758387efaed295d0 commit=139.25µs alloc=4.5GB sys=11.3GB
INFO[09-16|18:34:57.747] cl reorg                                 new_head_slot=5899074 fork_slot=5899071 old_canonical=0xa28fcfec529b147bfbc2
21011b1b44b6f836998de10aa4d9728f697187f2ecaf new_canonical=0x6c2e9f4384c66360f6ef1093b84834b44cbd8ed1b1518d4f300422dbcd8c64cd
INFO[09-16|18:35:26.487] P2P                                      app=caplin peers=1
INFO[09-16|18:35:34.059] [4/6 Execution] Done Commit every block  blk=6705181 blks=2 blk/s=19.1 txs=228 tx/s=2.18k gas/s=202.66M buf=4.2MB/512
.0MB stepsInDB=0.00 step=198.0 alloc=6.8GB sys=11.3GB
INFO[09-16|18:35:34.296] head updated                             head=0x281205fdd092035bf2306e03ad71b43bcbfc2012206c4081fcf7809e2790097c hash
=0x281205fdd092035bf2306e03ad71b43bcbfc2012206c4081fcf7809e2790097c commit=125.625µs alloc=6.8GB sys=11.3GB
INFO[09-16|18:35:34.296] cl reorg                                 new_head_slot=5899077 fork_slot=5899074 old_canonical=0x6c2e9f4384c66360f6ef
1093b84834b44cbd8ed1b1518d4f300422dbcd8c64cd new_canonical=0x78dff54a3409a83e703c86c796505caf585a9f1cbd3067db408e1acd22129679
INFO[09-16|18:36:24.447] [p2p] GoodPeers                          eth68=1
INFO[09-16|18:36:24.447] [mem] memory stats                       alloc=3.2GB sys=11.3GB
INFO[09-16|18:36:24.463] [txpool] stat                            pending=974 baseFee=554 queued=906 alloc=3.2GB sys=11.3GB
INFO[09-16|18:36:26.485] P2P                                      app=caplin peers=0
INFO[09-16|18:37:24.118] [4/6 Execution] Done Commit every block  blk=6705189 blks=8 blk/s=15.1 txs=1184 tx/s=2.22k gas/s=259.13M buf=16.6MB/5
12.0MB stepsInDB=0.00 step=198.0 alloc=3.1GB sys=11.3GB
INFO[09-16|18:37:24.445] cl reorg                                 new_head_slot=5899086 fork_slot=5899077 old_canonical=0x78dff54a3409a83e703c
86c796505caf585a9f1cbd3067db408e1acd22129679 new_canonical=0x14a92944edea64ba437ad3c7dcce7e767e485e1178b50ff903b0468f00960236
INFO[09-16|18:37:26.485] P2P                                      app=caplin peers=1
INFO[09-16|18:37:27.154] head updated                             head=0xc57e56b1b702df51beb61d7e94af203c2e1dbb0347d7102c42fd89e1e47cdf02 hash
=0xc57e56b1b702df51beb61d7e94af203c2e1dbb0347d7102c42fd89e1e47cdf02 commit=81.751042ms alloc=3.1GB sys=11.3GB
INFO[09-16|18:37:38.649] head updated                             head=0x58b4dc5296093cd42bd2be48417d88b66c6b9c73e84af15c07ffd550f248ded3 hash
=0x58b4dc5296093cd42bd2be48417d88b66c6b9c73e84af15c07ffd550f248ded3 number=6705190 execution=71.543792ms mgas/s=239.96 average mgas/s=198.54 c
ommit=91.25µs alloc=3.3GB sys=11.3GB
INFO[09-16|18:38:26.485] P2P                                      app=caplin peers=1
INFO[09-16|18:38:37.284] [snapshots:merge] Stat                   blocks=6.70M indices=6.70M alloc=2.4GB sys=11.3GB
INFO[09-16|18:38:39.749] [snapshots:blocks:retire] Stat           blocks=6.60M indices=6.60M alloc=2.3GB sys=11.3GB
WARN[09-16|18:38:44.548] Cannot update chain head                 hash=0xfbef5597530ee40b275df28b359e4441a37c9e71ae37ec7ead2c255cb15b22f0 err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6eb, 0); last in db: (6705194, 3
09449660)"
WARN[09-16|18:38:44.549] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6eb, 0); last in db: (6705194, 309449660)"
WARN[09-16|18:39:01.427] Cannot update chain head                 hash=0xb9ba16189c8d505f534fec4a671c4932560f09fd637eea326914680a3f8ba306 err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ec, 0); last in db: (6705195, 3
09449979)"
WARN[09-16|18:39:01.427] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ec, 0); last in db: (6705195, 309449979)"
WARN[09-16|18:39:19.023] Cannot update chain head                 hash=0xabcf83c7f5b5975203c0da0517d6e1b67e36a29dc988e359c24b619edea3ed59 err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ed, 0); last in db: (6705196, 3
09450048)"
WARN[09-16|18:39:19.024] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ed, 0); last in db: (6705196, 309450048)"
INFO[09-16|18:39:24.448] [p2p] GoodPeers                          eth68=1
INFO[09-16|18:39:24.448] [mem] memory stats                       alloc=2.8GB sys=11.3GB
INFO[09-16|18:39:24.462] [txpool] stat                            pending=660 baseFee=1277 queued=898 alloc=2.8GB sys=11.3GB
INFO[09-16|18:39:26.486] P2P                                      app=caplin peers=2
WARN[09-16|18:39:32.763] Cannot update chain head                 hash=0x494d34f29a44c80a3dcb1b23f6a75638e6f1ec7e3dbb8604b3fc48f0ee0dad7a err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ee, 0); last in db: (6705197, 3
09450244)"
WARN[09-16|18:39:32.763] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ee, 0); last in db: (6705197, 309450244)"
WARN[09-16|18:39:37.985] Cannot update chain head                 hash=0x3eeeca948dae8c83f24cb4acb7b06c10876bce57bfac5e259b9e81167a36ebab err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ef, 0); last in db: (6705198, 3
09450304)"
WARN[09-16|18:39:37.985] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6ef, 0); last in db: (6705198, 309450304)"
WARN[09-16|18:39:56.893] Cannot update chain head                 hash=0x75a9e77912aabf4633e4df22cec110189b16f1248fa14f0dc00ad10688784b6c err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f0, 0); last in db: (6705199, 3
09450403)"
WARN[09-16|18:39:56.893] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f0, 0); last in db: (6705199, 309450403)"
WARN[09-16|18:40:04.047] Cannot update chain head                 hash=0x699b2a4be35670cd7788a652e8c3fdc2477b7eb83f551f34c362b537d2ddbc9a err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f1, 0); last in db: (6705200, 3
09450830)"
WARN[09-16|18:40:04.047] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f1, 0); last in db: (6705200, 309450830)"
WARN[09-16|18:40:16.405] Cannot update chain head                 hash=0xbecc0d44eb415a8f4413e77e31fbf1f7177410494c3e533827ac21a975eaecf6 err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f2, 0); last in db: (6705201, 3
09450960)"
WARN[09-16|18:40:16.405] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f2, 0); last in db: (6705201, 309450960)"
WARN[09-16|18:40:24.946] Cannot update chain head                 hash=0x15a2a648e99925d5a401a5a50875dce78889464e9d20e3336bc313801a2a282b err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f3, 0); last in db: (6705202, 3
09451034)"
WARN[09-16|18:40:24.946] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f3, 0); last in db: (6705202, 309451034)"
INFO[09-16|18:40:26.486] P2P                                      app=caplin peers=4
WARN[09-16|18:40:44.725] Cannot update chain head                 hash=0xde67d7dcd065dfc0d5a51efddafacfc6e39137030cf50bea19e62e302c2678d3 err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f4, 0); last in db: (6705203, 3
09451609)"
WARN[09-16|18:40:44.725] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f4, 0); last in db: (6705203, 309451609)"
WARN[09-16|18:40:48.600] Cannot update chain head                 hash=0x57923f615a5cf427ced8a96d8b08be0e4306ffcf102fd83c112e9dd4ef76f99d err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f5, 0); last in db: (6705204, 3
09451696)"
WARN[09-16|18:40:48.600] error executing clstage                  app=caplin stage=ForkChoice err="failed to compute and notify services of ne
w fork choice: failed to run forkchoice: execution Client RPC failed to retrieve ForkChoiceUpdate response, err: updateForkChoice: [4/6 Execut
ion] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f5, 0); last in db: (6705204, 309451696)"
WARN[09-16|18:41:08.657] Cannot update chain head                 hash=0xe09e674ded60d1838526d5b7e6f950eb60bcc1847c501b774911e760cce98f4a err=
"updateForkChoice: [4/6 Execution] FindBlockNum(309449193): seems broken TxNum value: 0000000000000000 -> (64b6f6, 0); last in db: (6705205, 3
09451767)"
wmitsuda commented 2 days ago

using the "delete chaindata" workaround... FYI only 1 of my sepolia nodes went crazy, the other one is fine

taratorio commented 9 hours ago

Same here, happens on Amoy too