search

darwinia-network / darwinia

Darwinia Chain, EVM+, Polkadot parachain and integrating with XCMP.
https://darwinia.network
GNU General Public License v3.0
238 stars 49 forks source link

GRANDPA voter error: Client(Msg("State database error: Trying to canonicalize invalid block")) #583

Closed wi1dcard closed 3 years ago

wi1dcard commented 3 years ago

Full log:


Nov 03 17:11:15.006  INFO   _____                      _       _           
Nov 03 17:11:15.006  INFO  |  __ \                    (_)     (_)          
Nov 03 17:11:15.006  INFO  | |  | | __ _ _ ____      ___ _ __  _  __ _     
Nov 03 17:11:15.006  INFO  | |  | |/ _` | '__\ \ /\ / / | '_ \| |/ _` |    
Nov 03 17:11:15.006  INFO  | |__| | (_| | |   \ V  V /| | | | | | (_| |    
Nov 03 17:11:15.006  INFO  |_____/ \__,_|_|    \_/\_/ |_|_| |_|_|\__,_|    
Nov 03 17:11:15.006  INFO Darwinia    
Nov 03 17:11:15.006  INFO āœŒļø  version 0.8.1-f708513-x86_64-linux-gnu    
Nov 03 17:11:15.006  INFO ā¤ļø  by Darwinia Network <hello@darwinia.network>, 2018-2020    
Nov 03 17:11:15.006  INFO šŸ“‹ Chain specification: Darwinia Crab    
Nov 03 17:11:15.006  INFO šŸ· Node name: crab-fullnode-rpc    
Nov 03 17:11:15.006  INFO šŸ‘¤ Role: FULL    
Nov 03 17:11:15.006  INFO šŸ’¾ Database: RocksDb at /substrate-base/chains/crab/db    
Nov 03 17:11:15.006  INFO ā›“  Native runtime: Crab-27 (Darwinia Crab-0.tx3.au0)    
Nov 03 17:11:15.109  INFO šŸ· Local node identity is: 12D3KooWP11jFzcbCRbqRRR73aMRz3CtGLgN1X7VrCXuSDHgeNML    
Nov 03 17:11:15.113  INFO šŸ“¦ Highest known block at #2511304    
Nov 03 17:11:15.113  INFO ć€½ļø Prometheus server started at 0.0.0.0:9615    
Nov 03 17:11:15.128  INFO Listening for new connections on 0.0.0.0:9944.    
Nov 03 17:11:15.133 ERROR GRANDPA voter error: Client(Msg("State database error: Trying to canonicalize invalid block"))    
Nov 03 17:11:15.134 ERROR Essential task `grandpa-voter` failed. Shutting down service.    
Error: Input("Essential task failed.")

One of our nodes has this error repeatedly on every start. BTW, is there any workaround or quick fix? Do I have to re-sync all data of this node?

AurevoirXavier commented 3 years ago

Try ./darwinia revert -d /path

wi1dcard commented 3 years ago

@AurevoirXavier Looks the same:

root@debug-alpine-54d4c57c8-djvf9:/darwinia# darwinia --chain crab -d /darwinia
Nov 04 07:45:02.484  INFO   _____                      _       _
Nov 04 07:45:02.484  INFO  |  __ \                    (_)     (_)
Nov 04 07:45:02.484  INFO  | |  | | __ _ _ ____      ___ _ __  _  __ _
Nov 04 07:45:02.484  INFO  | |  | |/ _` | '__\ \ /\ / / | '_ \| |/ _` |
Nov 04 07:45:02.484  INFO  | |__| | (_| | |   \ V  V /| | | | | | (_| |
Nov 04 07:45:02.484  INFO  |_____/ \__,_|_|    \_/\_/ |_|_| |_|_|\__,_|
Nov 04 07:45:02.484  INFO Darwinia
Nov 04 07:45:02.484  INFO āœŒļø  version 0.8.1-f708513-x86_64-linux-gnu
Nov 04 07:45:02.484  INFO ā¤ļø  by Darwinia Network <hello@darwinia.network>, 2018-2020
Nov 04 07:45:02.484  INFO šŸ“‹ Chain specification: Darwinia Crab
Nov 04 07:45:02.484  INFO šŸ· Node name: childlike-rainstorm-0726
Nov 04 07:45:02.484  INFO šŸ‘¤ Role: FULL
Nov 04 07:45:02.484  INFO šŸ’¾ Database: RocksDb at /darwinia/chains/crab/db
Nov 04 07:45:02.484  INFO ā›“  Native runtime: Crab-27 (Darwinia Crab-0.tx3.au0)
Nov 04 07:45:02.594  INFO šŸ· Local node identity is: 12D3KooWP11jFzcbCRbqRRR73aMRz3CtGLgN1X7VrCXuSDHgeNML
Nov 04 07:45:02.597  INFO šŸ“¦ Highest known block at #2511304
Nov 04 07:45:02.598  INFO ć€½ļø Prometheus server started at 127.0.0.1:9615
Nov 04 07:45:02.602  INFO Listening for new connections on 127.0.0.1:9944.
Nov 04 07:45:02.609 ERROR GRANDPA voter error: Client(Msg("State database error: Trying to canonicalize invalid block"))
Nov 04 07:45:02.609 ERROR Essential task `grandpa-voter` failed. Shutting down service.
Error: Input("Essential task failed.")
root@debug-alpine-54d4c57c8-djvf9:/darwinia# darwinia revert --chain crab -d /darwinia
Nov 04 07:45:20.035  INFO There aren't any non-finalized blocks to revert.
root@debug-alpine-54d4c57c8-djvf9:/darwinia# darwinia revert -d /darwinia
Nov 04 07:45:26.697  INFO šŸ‘¶ Creating empty BABE epoch changes on what appears to be first startup.
Nov 04 07:45:26.700  INFO There aren't any non-finalized blocks to revert.
root@debug-alpine-54d4c57c8-djvf9:/darwinia# darwinia revert --chain crab -d /darwinia/chains/crab/
Nov 04 07:45:44.573  INFO šŸ”Ø Initializing Genesis block/state (state: 0x34d4ā€¦110f, header-hash: 0x34f6ā€¦3c65)
Nov 04 07:45:44.630  INFO šŸ‘“ Loading GRANDPA authority set from genesis on what appears to be first startup.
Nov 04 07:45:44.812  INFO ā±  Loaded block-time = 6000 milliseconds from genesis on first-launch
Nov 04 07:45:44.813  INFO šŸ‘¶ Creating empty BABE epoch changes on what appears to be first startup.
Nov 04 07:45:44.813  INFO There aren't any non-finalized blocks to revert.
root@debug-alpine-54d4c57c8-djvf9:/darwinia# darwinia revert -d /darwinia/chains/crab/
Nov 04 07:45:54.247  INFO šŸ”Ø Initializing Genesis block/state (state: 0x192dā€¦81ef, header-hash: 0x729cā€¦1db8)
Nov 04 07:45:54.282  INFO šŸ‘“ Loading GRANDPA authority set from genesis on what appears to be first startup.
Nov 04 07:45:54.600  INFO ā±  Loaded block-time = 6000 milliseconds from genesis on first-launch
Nov 04 07:45:54.601  INFO šŸ‘¶ Creating empty BABE epoch changes on what appears to be first startup.
Nov 04 07:45:54.601  INFO There aren't any non-finalized blocks to revert.
root@debug-alpine-54d4c57c8-djvf9:/darwinia# darwinia --chain crab -d /darwinia
Nov 04 07:46:01.545  INFO   _____                      _       _
Nov 04 07:46:01.545  INFO  |  __ \                    (_)     (_)
Nov 04 07:46:01.545  INFO  | |  | | __ _ _ ____      ___ _ __  _  __ _
Nov 04 07:46:01.545  INFO  | |  | |/ _` | '__\ \ /\ / / | '_ \| |/ _` |
Nov 04 07:46:01.545  INFO  | |__| | (_| | |   \ V  V /| | | | | | (_| |
Nov 04 07:46:01.545  INFO  |_____/ \__,_|_|    \_/\_/ |_|_| |_|_|\__,_|
Nov 04 07:46:01.545  INFO Darwinia
Nov 04 07:46:01.545  INFO āœŒļø  version 0.8.1-f708513-x86_64-linux-gnu
Nov 04 07:46:01.546  INFO ā¤ļø  by Darwinia Network <hello@darwinia.network>, 2018-2020
Nov 04 07:46:01.546  INFO šŸ“‹ Chain specification: Darwinia Crab
Nov 04 07:46:01.546  INFO šŸ· Node name: polite-harmony-8846
Nov 04 07:46:01.546  INFO šŸ‘¤ Role: FULL
Nov 04 07:46:01.546  INFO šŸ’¾ Database: RocksDb at /darwinia/chains/crab/db
Nov 04 07:46:01.546  INFO ā›“  Native runtime: Crab-27 (Darwinia Crab-0.tx3.au0)
Nov 04 07:46:01.654  INFO šŸ· Local node identity is: 12D3KooWP11jFzcbCRbqRRR73aMRz3CtGLgN1X7VrCXuSDHgeNML
Nov 04 07:46:01.659  INFO šŸ“¦ Highest known block at #2511304
Nov 04 07:46:01.659  INFO ć€½ļø Prometheus server started at 127.0.0.1:9615
Nov 04 07:46:01.664  INFO Listening for new connections on 127.0.0.1:9944.
Nov 04 07:46:01.673 ERROR GRANDPA voter error: Client(Msg("State database error: Trying to canonicalize invalid block"))
Nov 04 07:46:01.673 ERROR Essential task `grandpa-voter` failed. Shutting down service.
Error: Input("Essential task failed.")
AurevoirXavier commented 3 years ago 
Nov 04 07:45:20.035  INFO There aren't any non-finalized blocks to revert.

I think you have to resync. Or copy the data from another local node.

wi1dcard commented 3 years ago 
Nov 04 07:45:20.035  INFO There aren't any non-finalized blocks to revert.

I think you have to resync. Or copy the data from another local node.

Ok, sure. However, I'm wondering how has this happened...