search

maticnetwork / heimdall

Validator node for Polygon PoS
https://polygon.technology/
GNU General Public License v3.0
255 stars 178 forks source link

Always crashing with "panic: Failed to process committed block" #1173

Open donbeave opened 2 months ago

donbeave commented 2 months ago

Heimdall always crashes with the error:

panic: Failed to process committed block (107121:8E3CF3D710B5410CD5A7D5F9554CDF4035C0B2E699C458ED35146FF63EF63C69): Invalid commit -- invalid signature: Vote{1:42EEFCDA06EA 107120/00/2(Precommit) 7DD5F11F3A3E 885E02DC416E @ 2020-06-06T11:39:49.791792191Z [no-proposals]}

Heimdall version:

1.0.7

Environment:

Have you tried the latest version: yes

Logs (paste a small part showing an error (< 10 lines) or link a pastebin, gist, etc. containing more of the log file):

INFO [2024-06-27|14:07:48.407] Executed block                               module=state height=107118 validTxs=0 invalidTxs=0
INFO [2024-06-27|14:07:48.512] Committed state                              module=state height=107118 txs=0 appHash=C13D46A4EF1AE0D5B184137CC9B6F4AAC6BA7A83002CF172DED2F64BDFD5426E
INFO [2024-06-27|14:07:48.513] Indexed block                                module=txindex height=107118
INFO [2024-06-27|14:07:48.521] Executed block                               module=state height=107119 validTxs=0 invalidTxs=0
INFO [2024-06-27|14:07:48.572] Committed state                              module=state height=107119 txs=0 appHash=C13D46A4EF1AE0D5B184137CC9B6F4AAC6BA7A83002CF172DED2F64BDFD5426E
INFO [2024-06-27|14:07:48.572] Indexed block                                module=txindex height=107119
INFO [2024-06-27|14:07:48.576] Executed block                               module=state height=107120 validTxs=0 invalidTxs=0
INFO [2024-06-27|14:07:48.625] Committed state                              module=state height=107120 txs=0 appHash=C13D46A4EF1AE0D5B184137CC9B6F4AAC6BA7A83002CF172DED2F64BDFD5426E
INFO [2024-06-27|14:07:48.626] Indexed block                                module=txindex height=107120
panic: Failed to process committed block (107121:8E3CF3D710B5410CD5A7D5F9554CDF4035C0B2E699C458ED35146FF63EF63C69): Invalid commit -- invalid signature: Vote{1:42EEFCDA06EA 107120/00/2(Precommit) 7DD5F11F3A3E 885E02DC416E @ 2020-06-06T11:39:49.791792191Z [no-proposals]}

goroutine 196 [running]:
github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0x4001400e00)
    /go/pkg/mod/github.com/maticnetwork/tendermint@v0.33.2/blockchain/v0/reactor.go:344 +0xbe0
created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart in goroutine 1
    /go/pkg/mod/github.com/maticnetwork/tendermint@v0.33.2/blockchain/v0/reactor.go:118 +0x78

Config (you can paste only the changes you've made):

No changes, basic mainnet config.

node command runtime flags:

  --chain=mainnet
  --home=/data/heimdalld
  --home-client=/data/heimdallcli
  --log_level=*:info
github-actions[bot] commented 1 month ago

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.

donbeave commented 1 month ago

Sorry for pinging, any way to take a look at this issue?

github-actions[bot] commented 1 month ago

This issue is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.

donbeave commented 1 month ago

Is heimdall dead?

ssandeep commented 1 month ago

Hey @donbeave , did you start syncing the node from genesis? If yes, are you aware that it will take at least a few months to sync till the chain tip?

Raneet10 commented 1 month ago

@donbeave Did you start syncing from genesis using v1.0.7 ?

marcello33 commented 3 weeks ago

@donbeave is this fixed?

donbeave commented 3 weeks ago

@ssandeep Yes, I'm syncing the node from genesis. And it started crashing in a few hours or less. @Raneet10 Yes, I'm using the latest version 1.0.7. @marcello33 Nope, I still have this issue. I tried to run it on an x86_64 (AMD64) server and it works fine, but whenever I try to do it on an ARM server it always crashes after a while.

Raneet10 commented 2 weeks ago

Thanks @donbeave for the answers! We'll have a look at this; have flagged internally.

ssandeep commented 4 days ago

Hi @donbeave ,

We tried syncing a node from genesis and our node has had no issues getting to the blockheight for heimdall mentioned in the issue. We are using ubuntu 22.04 on arm and deployed via ansible. We also encourage node operators to use this method as well. We provide these same tools in the https://github.com/maticnetwork/node-ansible/tree/master repository

Sep 03 18:06:47 ip-10-8-1-222 heimdalld[370067]: INFO [2024-09-03|18:06:47.805] Committed state                              module=state height=107205 txs=0 appHash=C13D46A4EF1AE0D5B184137CC9B6F4AAC6BA7A83002CF172DED2F64BDFD5426E
Sep 03 18:06:47 ip-10-8-1-222 heimdalld[370067]: INFO [2024-09-03|18:06:47.816] Executed block                               module=state height=107206 validTxs=0 invalidTxs=0
Sep 03 18:06:47 ip-10-8-1-222 heimdalld[370067]: INFO [2024-09-03|18:06:47.845] Committed state                              module=state height=107206 txs=0 appHash=C13D46A4EF1AE0D5B184137CC9B6F4AAC6BA7A83002CF172DED2F64BDFD5426E
Sep 03 18:06:47 ip-10-8-1-222 heimdalld[370067]: INFO [2024-09-03|18:06:47.856] Executed block                               module=state height=107207 validTxs=0 invalidTxs=0
Sep 03 18:06:47 ip-10-8-1-222 heimdalld[370067]: INFO [2024-09-03|18:06:47.884] Committed state                              module=state height=107207 txs=0 appHash=C13D46A4EF1AE0D5B184137CC9B6F4AAC6BA7A83002CF172DED2F64BDFD5426E
Sep 03 18:06:47 ip-10-8-1-222 heimdalld[370067]: INFO [2024-09-03|18:06:47.895] Executed block                               module=state height=107208 validTxs=0 invalidTxs=0
Sep 03 18:06:47 ip-10-8-1-222 heimdalld[370067]: INFO [2024-09-03|18:06:47.924] Committed state                              module=state height=107208 txs=0 appHash=C13D46A4EF1AE0D5B184137CC9B6F4AAC6BA7A83002CF172DED2F64BDFD5426E
Sep 03 18:06:47 ip-10-8-1-222 heimdalld[370067]: INFO [2024-09-03|18:06:47.935] Executed block                               module=state height=107209 validTxs=0 invalidTxs=0
Sep 03 18:06:47 ip-10-8-1-222 heimdalld[370067]: INFO [2024-09-03|18:06:47.965] Committed state                              module=state height=107209 txs=0 appHash=C13D46A4EF1AE0D5B184137CC9B6F4AAC6BA7A83002CF172DED2F64BDFD5426E
Sep 03 18:06:47 ip-10-8-1-222 heimdalld[370067]: INFO [2024-09-03|18:06:47.977] Executed block                               module=state height=107210 validTxs=0 invalidTxs=0
Sep 03 18:06:48 ip-10-8-1-222 heimdalld[370067]: INFO [2024-09-03|18:06:48.008] Committed state                              module=state height=107210 txs=0 appHash=C13D46A4EF1AE0D5B184137CC9B6F4AAC6BA7A83002CF172DED2F64BDFD5426E
Sep 03 18:06:48 ip-10-8-1-222 heimdalld[370067]: INFO [2024-09-03|18:06:48.019] Executed block                               module=state height=107211 validTxs=0 invalidTxs=0
Sep 03 18:06:48 ip-10-8-1-222 heimdalld[370067]: INFO [2024-09-03|18:06:48.049] Committed state                              module=state height=107211 txs=0 appHash=C13D46A4EF1AE0D5B184137CC9B6F4AAC6BA7A83002CF172DED2F64BDFD5426E
Sep 03 18:06:48 ip-10-8-1-222 heimdalld[370067]: INFO [2024-09-03|18:06:48.060] Executed block                               module=state height=107212 validTxs=0 invalidTxs=0
Sep 03 18:06:48 ip-10-8-1-222 heimdalld[370067]: INFO [2024-09-03|18:06:48.090] Committed state                              module=state height=107212 txs=0 appHash=C13D46A4EF1AE0D5B184137CC9B6F4AAC6BA7A83002CF172DED2F64BDFD5426E
Sep 03 18:06:48 ip-10-8-1-222 heimdalld[370067]: INFO [2024-09-03|18:06:48.101] Executed block                               module=state height=107213 validTxs=0 invalidTxs=0
Sep 03 18:06:48 ip-10-8-1-222 heimdalld[370067]: INFO [2024-09-03|18:06:48.132] Committed state                              module=state height=107213 txs=0 appHash=C13D46A4EF1AE0D5B184137CC9B6F4AAC6BA7A83002CF172DED2F64BDFD5426E
Sep 03 18:06:48 ip-10-8-1-222 heimdalld[370067]: INFO [2024-09-03|18:06:48.143] Executed block                               module=state height=107214 validTxs=0 invalidTxs=0

Can you confirm which hardware you are using, how did you install heimdall?

cc: @djpolygon

donbeave commented 2 days ago

@ssandeep I used the Hetzner ARM server, this machine: https://www.hetzner.com/dedicated-rootserver/rx220/configurator

I already stopped it and switched to the x86_64 server.

This is how I install heimdall: https://github.com/ChainArgos/blockchain-nodes/blob/main/docker-heimdall/Dockerfile.arm64