search

Fantom-foundation / go-opera

Opera blockchain protocol secured by the Lachesis consensus algorithm
https://fantom.foundation/
GNU Lesser General Public License v3.0
278 stars 282 forks source link

Txtracing crash on block 67571797 #505

Closed zhy827827 closed 1 year ago

zhy827827 commented 1 year ago

version: go-opera/v1.1.2-txtracing-rc.6

The ftm height is 67571797 and the block is not synchronized. What is the reason?

Aug 29 11:32:16 fantom opera[3820728]: INFO [08-29|11:32:16.370] Generating state snapshot                root=09ecf3..1f9830 at=af0162..aa21a8 accounts=81,963,632 slots=243,517,090 storage=22.01GiB elapsed=2m8.020s    eta=17m37.445s
Aug 29 11:32:24 fantom opera[3820728]: INFO [08-29|11:32:24.370] Generating state snapshot                root=09ecf3..1f9830 at=afbc71..a0c10e accounts=82,306,101 slots=243,573,893 storage=22.04GiB elapsed=2m16.020s   eta=17m16.174s
Aug 29 11:32:32 fantom opera[3820728]: INFO [08-29|11:32:32.371] Generating state snapshot                root=09ecf3..1f9830 at=b0758d..616de4 accounts=82,645,872 slots=243,729,825 storage=22.07GiB elapsed=2m24.022s   eta=16m57.425s
Aug 29 11:32:40 fantom opera[3820728]: INFO [08-29|11:32:40.372] Generating state snapshot                root=09ecf3..1f9830 at=b11dd0..f14d5f accounts=82,954,171 slots=244,056,008 storage=22.12GiB elapsed=2m32.022s   eta=16m46.322s
Aug 29 11:32:48 fantom opera[3820728]: INFO [08-29|11:32:48.372] Generating state snapshot                root=09ecf3..1f9830 at=b1e52f..190b76 accounts=83,319,359 slots=244,135,244 storage=22.15GiB elapsed=2m40.022s   eta=16m24.481s
Aug 29 11:32:54 fantom opera[3820728]: INFO [08-29|11:32:54.520] New LLR summary                          last_epoch=0 last_block=67573032 new_evs=0 new_ers=0 new_bvs=64 new_brs=0 age=none
Aug 29 11:32:56 fantom opera[3820728]: INFO [08-29|11:32:56.372] Generating state snapshot                root=09ecf3..1f9830 at=b2859d..de995e accounts=83,612,842 slots=244,338,545 storage=22.18GiB elapsed=2m48.022s   eta=16m17.169s
Aug 29 11:33:04 fantom opera[3820728]: INFO [08-29|11:33:04.372] Generating state snapshot                root=09ecf3..1f9830 at=b32d31..4034df accounts=83,919,291 slots=244,449,928 storage=22.21GiB elapsed=2m56.022s   eta=16m7.506s
Aug 29 11:33:12 fantom opera[3820728]: INFO [08-29|11:33:12.372] Generating state snapshot                root=09ecf3..1f9830 at=b3d0f9..036de3 accounts=84,219,143 slots=244,611,660 storage=22.24GiB elapsed=3m4.022s    eta=15m59.151s
Aug 29 11:33:20 fantom opera[3820728]: INFO [08-29|11:33:20.374] Generating state snapshot                root=09ecf3..1f9830 in=b4723a..ea2d93 at=001bf6..c60361 accounts=84,513,725 slots=244,800,702 storage=22.27GiB elapsed=3m12.025s   eta=15m51.58s
Aug 29 11:33:28 fantom opera[3820728]: INFO [08-29|11:33:28.374] Generating state snapshot                root=09ecf3..1f9830 in=b4b4a8..1b9d80 at=290dec..f3e563 accounts=84,635,474 slots=245,839,931 storage=22.35GiB elapsed=3m20.025s   eta=16m11.289s
Aug 29 11:33:36 fantom opera[3820728]: INFO [08-29|11:33:36.374] Generating state snapshot                root=09ecf3..1f9830 at=b55f91..ff5752 accounts=84,948,153 slots=245,943,191 storage=22.38GiB elapsed=3m28.025s   eta=15m59.851s
quick-pawiromitchel commented 1 year ago

experiencing the same problem, we're also on go-opera/v1.1.2-txtracing-rc.6

0xBasically commented 1 year ago

same problem on release/txtracing/1.1.2-rc.5

jenikd commented 1 year ago

Please update the branch release/txtracing/1.1.2-rc.6 as it now includes fix for this

zhy827827 commented 1 year ago

with new version branch release/txtracing/1.1.2-rc.6
Compiled opera starts to report an error:

 Vector clock error                       err="Event A=232664:683:26e7ff not found"

I used this version before version: 1.1.3-txtracing-rc.3

@HonzaDajc

uprendis commented 1 year ago

The latest hotfixes are now available in the following branches:

For version 1.1.2, refer to the branch release/txtracing/1.1.2-rc.6.1. For version 1.1.3, check out release/txtracing/1.1.3-rc.3.1. If you previously used the release/txtracing/1.1.2-rc.5 branch, we recommend upgrading directly to release/txtracing/1.1.2-rc.6.1.

While the earlier hotfix addressed this specific crash, there remained potential for other related crashes. These have now been addressed in the recent update.

If you experienced this crash, It's likely that your node DB is in now a dirty state and won't restart. You can execute the command opera --datadir /path/to/datadir db heal --experimental for recovery. When you restart your node, it will start the EVM snapshot reconstruction. This recovery process may be time-consuming, with the duration primarily influenced by your disk speed.

zhy827827 commented 1 year ago

I use release/txtracing/1.1.3-rc.3.1,When it is synchronized to 67571792, it will not be synchronized, or has it not been repaired? @uprendis

Go-Opera Version: 1.1.3-txtracing-rc.3.1 Git Commit: 5173adf944ce11a28761dfc78679a89c75001ffa

Aug 29 11:32:16 fantom opera[3820728]: INFO [08-29|11:32:16.370] Generating state snapshot                root=09ecf3..1f9830 at=af0162..aa21a8 accounts=81,963,632 slots=243,517,090 storage=22.01GiB elapsed=2m8.020s    eta=17m37.445s
Aug 29 11:32:24 fantom opera[3820728]: INFO [08-29|11:32:24.370] Generating state snapshot                root=09ecf3..1f9830 at=afbc71..a0c10e accounts=82,306,101 slots=243,573,893 storage=22.04GiB elapsed=2m16.020s   eta=17m16.174s
Aug 29 11:32:32 fantom opera[3820728]: INFO [08-29|11:32:32.371] Generating state snapshot                root=09ecf3..1f9830 at=b0758d..616de4 accounts=82,645,872 slots=243,729,825 storage=22.07GiB elapsed=2m24.022s   eta=16m57.425s
Aug 29 11:32:40 fantom opera[3820728]: INFO [08-29|11:32:40.372] Generating state snapshot                root=09ecf3..1f9830 at=b11dd0..f14d5f accounts=82,954,171 slots=244,056,008 storage=22.12GiB elapsed=2m32.022s   eta=16m46.322s
Aug 29 11:32:48 fantom opera[3820728]: INFO [08-29|11:32:48.372] Generating state snapshot                root=09ecf3..1f9830 at=b1e52f..190b76 accounts=83,319,359 slots=244,135,244 storage=22.15GiB elapsed=2m40.022s   eta=16m24.481s
Aug 29 11:32:54 fantom opera[3820728]: INFO [08-29|11:32:54.520] New LLR summary                          last_epoch=0 last_block=67573032 new_evs=0 new_ers=0 new_bvs=64 new_brs=0 age=none
Aug 29 11:32:56 fantom opera[3820728]: INFO [08-29|11:32:56.372] Generating state snapshot                root=09ecf3..1f9830 at=b2859d..de995e accounts=83,612,842 slots=244,338,545 storage=22.18GiB elapsed=2m48.022s   eta=16m17.169s
Aug 29 11:33:04 fantom opera[3820728]: INFO [08-29|11:33:04.372] Generating state snapshot                root=09ecf3..1f9830 at=b32d31..4034df accounts=83,919,291 slots=244,449,928 storage=22.21GiB elapsed=2m56.022s   eta=16m7.506s
Aug 29 11:33:12 fantom opera[3820728]: INFO [08-29|11:33:12.372] Generating state snapshot                root=09ecf3..1f9830 at=b3d0f9..036de3 accounts=84,219,143 slots=244,611,660 storage=22.24GiB elapsed=3m4.022s    eta=15m59.151s
Aug 29 11:33:20 fantom opera[3820728]: INFO [08-29|11:33:20.374] Generating state snapshot                root=09ecf3..1f9830 in=b4723a..ea2d93 at=001bf6..c60361 accounts=84,513,725 slots=244,800,702 storage=22.27GiB elapsed=3m12.025s   eta=15m51.58s
Aug 29 11:33:28 fantom opera[3820728]: INFO [08-29|11:33:28.374] Generating state snapshot                root=09ecf3..1f9830 in=b4b4a8..1b9d80 at=290dec..f3e563 accounts=84,635,474 slots=245,839,931 storage=22.35GiB elapsed=3m20.025s   eta=16m11.289s
Aug 29 11:33:36 fantom opera[3820728]: INFO [08-29|11:33:36.374] Generating state snapshot                root=09ecf3..1f9830 at=b55f91..ff5752 accounts=84,948,153 slots=245,943,191 storage=22.38GiB elapsed=3m28.025s   eta=15m59.851s

when i use the command opera --datadir /path/to/datadir db heal --experimental for recovery .also it will not be synchronized error log:

Unhandled trie error: missing trie node bf6ef5fb438fd19bfbe56273838f118760a924a1783ffb0dc74a27fd2e3f3ae6 (path )