Implement Rollback Mechanism for Neo Node to Sync Plugin Information

Jim8y commented 4 months ago

Summary or problem description Neo nodes support various plugins that provide a wide range of services, such as applicationlogs, stateservice, and more. However, these plugins sometimes crash or pause, causing them to miss critical block execution information. This desynchronization between the plugins and the blockchain can lead to inconsistencies and outdated information.

Currently, the only way to resynchronize these plugins is to reprocess the entire blockchain from the genesis block, which is inefficient and time-consuming. This process also increases the risk of further errors and delays in service restoration.

Do you have any solution you want to propose?

Solution:

To address this issue, we propose implementing a rollback mechanism in the Neo node. This mechanism will allow the blockchain to roll back to a specified block height, enabling plugins to resynchronize their data starting from that point. By rolling back to a known good state and reapplying blocks, plugins can catch up with the latest block execution information without the need for a full blockchain resynchronization.

Proposed Implementation:

Monitor and Record State Changes:

Implement a system to monitor and record state changes of the blockchain during block commits. Save these state changes to a persistent store indexed by block height. Rollback Command:

Introduce a new command (fallback ledger) that allows users to specify a target block height to roll back to. When executed, the command will revert the blockchain state to the specified height by applying the recorded state changes in reverse order.

Reapply Blocks:

After rolling back to the specified height, the node will reprocess blocks from that height onward, ensuring that plugins can update their state with the latest block execution information.

Benefits:

Efficiency: The rollback mechanism will significantly reduce the time and resources needed to resynchronize plugins after a crash or pause.
Consistency: Ensures that plugins have up-to-date and consistent block execution information.
Flexibility: Allows for targeted rollback to specific block heights, providing greater control over the resynchronization process.

Where in the software does this update applies to?

Plugins
Ledger
DataCache

roman-khimov commented 4 months ago

Implement a system to monitor and record state changes of the blockchain during block commits.

Just reuse MPT with full history. NeoGo can rollback using exactly that. We will need MPT embedded anyway for #1526.

Jim8y commented 4 months ago

Implement a system to monitor and record state changes of the blockchain during block commits.

Just reuse MPT with full history. NeoGo can rollback using exactly that. We will need MPT embedded anyway for #1526.

unfortunately, we will not have mpt for ordinary node. why? because it would require a whole resync of the chain again to calculate the tree. using mpt is really good, we should have it integrated at the very begining. but now its unnecessary as neo has adopted statevalidators solution, and new 'stateroot' solution should based on that. we shall never require another whole chain resync.

Jim8y commented 4 months ago

oh by the way, i am curious how mpt can achieve rollback? isnt it a store tree? how come it can have history data?

update： get that already.

Jim8y commented 4 months ago

Another BTW, this is a plugin~~~~ technically, user will not use it unless they know that it is and they install it.

cschuchardt88 commented 4 months ago

Do you think core should save the TriggerType.OnPersist and TriggerType.PostPersist application_executed states? Pointless to process them to just reprocess them again in ApplicationLogs

Jim8y commented 4 months ago

Do you think core should save the TriggerType.OnPersist and TriggerType.PostPersist application_executed states? Pointless to process them to just reprocess them again in ApplicationLogs

this pr only solve one problem, which is rollback the ledger. taking care of other plugins can be solved in other prs. that is why i make this feature a plugin, we can keep updating it without breaking anything.

previously i had a complete system that handles all plugin store, but just too big to get reviewed.

shargon commented 4 months ago

Implement a system to monitor and record state changes of the blockchain during block commits.

Just reuse MPT with full history. NeoGo can rollback using exactly that. We will need MPT embedded anyway for #1526.

unfortunately, we will not have mpt for ordinary node. why? because it would require a whole resync of the chain again to calculate the tree. using mpt is really good, we should have it integrated at the very begining. but now its unnecessary as neo has adopted statevalidators solution, and new 'stateroot' solution should based on that. we shall never require another whole chain resync.

You will need to resync

Jim8y commented 4 months ago

Implement a system to monitor and record state changes of the blockchain during block commits.

Just reuse MPT with full history. NeoGo can rollback using exactly that. We will need MPT embedded anyway for #1526.

unfortunately, we will not have mpt for ordinary node. why? because it would require a whole resync of the chain again to calculate the tree. using mpt is really good, we should have it integrated at the very begining. but now its unnecessary as neo has adopted statevalidators solution, and new 'stateroot' solution should based on that. we shall never require another whole chain resync.

You will need to resync

totally up to the solution, if we have a solution that can be applied at any moment and not require a resync, why would we choose a resync one? and this plugin takes effect when it is installed, no resync is needed at all, unless user want to rollback to revious blocks, but that should not be a concern of this plugin cause he can simply resync the chain as before. this plugin is for answering the question of some plugin falls back the mainchain and miss some block states.

roman-khimov commented 4 months ago

but now its unnecessary as neo has adopted statevalidators solution, and new 'stateroot' solution should based on that.

This part I can't understand.

this plugin is for answering the question of some plugin falls back the mainchain and miss some block states.

Plugin to restore plugins? Who is going to use it?

Jim8y commented 4 months ago

but now its unnecessary as neo has adopted statevalidators solution, and new 'stateroot' solution should based on that.

This part I can't understand.

Neo has statevalidators, we can use them to calculate a state root for every single block, then broadcast/share it, other nodes can also calculate its own state root for that block, then compare with statevalidators'. No need to calculate a global stateroot at all, no need to resync at all. As long as we ensure block leverl state consistancy, we ensure the global state consistancy.

We may switch to MPT in the future, but currently, i think we should avoid resync. With block-level MT being calculated and stored, we can have an official snapshot at some height, then those existing node can calculate MPT based on that snapshot and its local MTs without resyning anything.

That is ->

Block leverl MT
Snapshot (MPT) + MT
MPT

this plugin is for answering the question of some plugin falls back the mainchain and miss some block states.

Plugin to restore plugins? Who is going to use it?

Whoever think its useful. its plugin, not something integrated into the core. First of all i dont think plugin restore plugin has any issue at all. As long as it works efficiently and solves the issue.

Then, its not plugin restoring plugin, rollback plugin works on ledger, moves ledger backwards, it does not communicate, nor interoperate with other plugins.

roman-khimov commented 4 months ago

we can use them to calculate a state root for every single block, then broadcast/share it,

This is done already (but https://github.com/neo-project/neo-modules/issues/536 and https://github.com/neo-project/neo/issues/2374#issuecomment-791276069).

other nodes can also calculate its own state root for that block, then compare with statevalidators'

You wouldn't believe, but that's exactly what NeoGo does.

No need to calculate a global stateroot at all

I'd be interested in alternative to #2373 without in-header state root. Snapshots can help somewhat (hi, #1286), but the question is --- why should we trust them?

Although all of this discussion is better for #1526.

Jim8y commented 4 months ago

This is done already (but neo-project/neo-modules#536 and #2374 (comment)).

This is awesome.

You wouldn't believe, but that's exactly what NeoGo does.

Nothing wont i believe neogo has it, you basically have discussed all problems that i am concering now.

I'd be interested in alternative to #2373 without in-header state root. Snapshots can help somewhat (hi, #1286), but the question is --- why should we trust them?

Although all of this discussion is better for #1526.

What i have in mind is exactly not in-header, but directly p2p/rpc validation. as to why should we trust them, very simple, we hardcode snapshot state root into the core. Its public and anyone can challenge that.

roman-khimov commented 4 months ago

as to why should we trust them, very simple, we hardcode snapshot state root into the core

What core? Node source code?

neo-project / neo