search

axonweb3 / axon

Axon is a Layer 2 framework of CKB with native cross-chain and interoperability.
https://axonweb3.io
MIT License
65 stars 39 forks source link

My sync node got error `InvalidStateRoot` unexpectedly. #1573

Closed yangby-cryptape closed 10 months ago

yangby-cryptape commented 11 months ago

Description

I started a sync node since 2023-11-06. Every workday, it was connected to our internal testnet when I started work, and it was stopped after I got off work.

Every two or three days, I re-compiled a new Axon binary with latest commit on main branch. Current version is 0828794.

I was trying to write a tool to connect p2p network directly today (2023-11-17), so I didn't use my sync node whole day, but it was still started in background.

Half an hour ago, I tried to start it manually, but it could not do sync anymore. It printed the following errors again and again.

[INFO core_consensus::synchronization] [synchronization]: synced block, status: number:655420
[WARN overlord::state::process] Overlord: state receive an outdated status, height 655421, self height 655421
[WARN core_consensus::message] sync: receive remote block ProtocolError { kind: Consensus, error: InvalidStateRoot { expect: 0x33dbabe0f76b1e79948e0038bbf0c2b050e099a71b4c831c35f495a2ae7849c9, actual: 0x
e1f9583e2438d2718a3875435f237c68e534c759ade4e33aa21ffa488e93addc } }                                                                                                                                                                           
[INFO core_consensus::synchronization] [synchronization]: sync start, remote block number 671754 current block number 655420
[INFO core_consensus::synchronization] [synchronization]: try syncing block, current_consented_number 655420,syncing_number 655421                                                                         
[DEBUG core_executor::system_contract] execute addr 0xffffffffffffffffffffffffffffffffffffff01
[ERROR core_consensus::synchronization] [synchronization]: commit block 655420 error
[ERROR core_consensus::synchronization] [synchronization]: err, current_number 655420 err_msg: ProtocolError { kind: Consensus, error: InvalidStateRoot { expect: 0x33dbabe0f76b1e79948e0038bbf0c2b050e099a
71b4c831c35f495a2ae7849c9, actual: 0xe1f9583e2438d2718a3875435f237c68e534c759ade4e33aa21ffa488e93addc } }

I found the first error was happened at 2023-11-17T10:44:44.267931834+08:00. From then, my sync node had not synced any blocks.

Update-1

I tried:

Update-2

Commits between ad4375b..9a0bdbd are:

Skip the changes that don't modify source codes, there are only 3 commits which could introduced the break change, as I flagged above.

Update-3

I found another bad block at number 656501.

I tried:

Result

:dart: The commit 166506f introduced the break change.

Since I don't have bad data anymore, and the data is too large that I don't want to sync them again. (ref: #1574)

:sos: So my investigation is stopped. I won't work on this issue anymore. Please assign another one to continue this task, based on my works above.

Flouse commented 11 months ago

Result

🎯 The commit 166506f introduced the break change.

cc @KaoImin

Flouse commented 10 months ago

Conclusion