search

crypto-org-chain / cronos

Cronos is the first Ethereum-compatible blockchain network built on Cosmos SDK technology. Cronos aims to massively scale the DeFi, GameFi, and overall Web3 user community by providing builders with the ability to instantly port apps and crypto assets from other chains while benefiting from low transaction fees, high throughput, and fast finality.
Other
290 stars 235 forks source link

wrong Block.Header.AppHash #692

Closed PPianAIC closed 2 years ago

PPianAIC commented 2 years ago

I encountered this problem when snapshots synchronized. How to solve it, please.

12:26PM INF received complete proposal block hash=78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2 height=4475276 module=consensus server=node panic: +2/3 committed an invalid block: wrong Block.Header.AppHash. Expected DAA9072EC7B586617F866645D613C3AE17102594FFCDE76AF467EBB104A727D3, got 4E97CB5B45C698F75F3CC9C19455B25B75513812EEF0DB937C49D97EBB8CEBFE

JayT106 commented 2 years ago

Hi @PPianAIC can you provide which binary version you are running? Also, can you post more log details? So your node was able to do state-sync at a certain height (before 4475276), and sync some blocks, and finally got this panic at height 4475276?

yihuang commented 2 years ago

Likely a wrong version of binary?

PPianAIC commented 2 years ago

I have this problem when the version is latest [root@ip-172-31-29-219 cronos-1]# bin/cronosd version v0.8.0 3:23AM INF Unlocking keyring 3:23AM INF starting ABCI with Tendermint Error: invalid json-rpc config value: JSON-RPC feehistory-cap cannot be negative or 0: error in app.toml

The cronosd not running last for half an hour when I set feehistory-cap=10 [root@ip-172-31-29-219 cronos-1]# /data/cronos-1/bin/cronosd start --home /data/cronos-1/.cronos 3:25AM INF Unlocking keyring 3:25AM INF starting ABCI with Tendermint

Then I change the version to 0.7.1 [root@ip-172-31-29-219 cronos-1]# bin-7.1/bin/cronosd version v0.7.1 3:42AM INF Unlocking keyring 3:42AM INF starting ABCI with Tendermint 3:45AM INF service start impl=multiAppConn module=proxy msg={} server=node 3:45AM INF service start connection=query impl=localClient module=abci-client msg={} server=node 3:45AM INF service start connection=snapshot impl=localClient module=abci-client msg={} server=node 3:45AM INF service start connection=mempool impl=localClient module=abci-client msg={} server=node 3:45AM INF service start connection=consensus impl=localClient module=abci-client msg={} server=node 3:45AM INF service start impl=EventBus module=events msg={} server=node 3:45AM INF service start impl=PubSub module=pubsub msg={} server=node 3:46AM INF service start impl=IndexerService module=txindex msg={} server=node 3:46AM INF ABCI Handshake App Info hash="ک\a.ǵ�a\u007f�fE�\x13î\x17\x10%����j�g��\x04�'�" height=4475275 module=consensus protocol-version=0 server=node software-version=v0.7.1 3:46AM INF ABCI Replay Blocks appHeight=4475275 module=consensus server=node stateHeight=4475275 storeHeight=4475275 3:46AM INF Completed ABCI Handshake - Tendermint and App are synced appHash="ک\a.ǵ�a\u007f�fE�\x13î\x17\x10%����j�g��\x04�'�" appHeight=4475275 module=consensus server=node 3:46AM INF Version info block=11 p2p=8 server=node tendermint_version=0.34.20 3:46AM INF This node is not a validator addr=BF734249BD0001D856DB6CA3C221F8B114908A66 module=consensus pubKey=3M6qUKPGqClUGaJAmYjsYZnHk7zKaDNlbSkj8U26aEI= server=node 3:46AM INF P2P Node ID ID=59157c187f733dc20dc77f3efc2ddaf01c531368 file=/data/cronos-1/.cronos/config/node_key.json module=p2p server=node 3:46AM INF Adding persistent peers addrs=[] module=p2p server=node 3:46AM INF Adding unconditional peer ids ids=[] module=p2p server=node 3:46AM INF Add our address to book addr={"id":"59157c187f733dc20dc77f3efc2ddaf01c531368","ip":"0.0.0.0","port":25656} book=/data/cronos-1/.cronos/config/addrbook.json module=p2p server=node 3:46AM INF service start impl=Node msg={} server=node 3:46AM INF Starting pprof server laddr=localhost:5060 server=node 3:46AM INF service start impl="P2P Switch" module=p2p msg={} server=node 3:46AM INF service start impl=ConsensusReactor module=consensus msg={} server=node 3:46AM INF Reactor module=consensus server=node waitSync=false 3:46AM INF service start impl=ConsensusState module=consensus msg={} server=node 3:46AM INF serve module=rpc-server msg={} server=node 3:46AM INF service start impl=baseWAL module=consensus msg={} server=node wal=/data/cronos-1/.cronos/data/cs.wal/wal 3:46AM INF service start impl=Group module=consensus msg={} server=node wal=/data/cronos-1/.cronos/data/cs.wal/wal 3:46AM INF service start impl=TimeoutTicker module=consensus msg={} server=node 3:46AM INF Searching for height height=4475276 max=710307 min=710208 module=consensus server=node wal=/data/cronos-1/.cronos/data/cs.wal/wal 3:46AM INF Searching for height height=4475275 max=710307 min=710208 module=consensus server=node wal=/data/cronos-1/.cronos/data/cs.wal/wal 3:46AM INF Found height=4475275 index=710307 module=consensus server=node wal=/data/cronos-1/.cronos/data/cs.wal/wal 3:46AM INF Catchup by replaying consensus messages height=4475276 module=consensus server=node 3:46AM INF Replay: New Step height=4475276 module=consensus round=0 server=node step=RoundStepNewHeight 3:46AM INF Replay: Timeout dur=3000 height=4475275 module=consensus round=0 server=node step=3 3:46AM INF Replay: Vote blockID={"hash":"FFB08F49B2228038C05681C61E0BE2EF15D1DB24A0B1ED43473471FEF423E894","parts":{"hash":"4DF8CDDABB629E60E46EF3641DA80C984C7AC1C2AC41ADF18AB1CAD107DDBD7D","total":1}} height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"FFB08F49B2228038C05681C61E0BE2EF15D1DB24A0B1ED43473471FEF423E894","parts":{"hash":"4DF8CDDABB629E60E46EF3641DA80C984C7AC1C2AC41ADF18AB1CAD107DDBD7D","total":1}} height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Timeout dur=-8237.686159 height=4475276 module=consensus round=0 server=node step=1 3:46AM INF Replay: New Step height=4475276 module=consensus round=0 server=node step=RoundStepPropose 3:46AM INF Replay: Vote blockID={"hash":"FFB08F49B2228038C05681C61E0BE2EF15D1DB24A0B1ED43473471FEF423E894","parts":{"hash":"4DF8CDDABB629E60E46EF3641DA80C984C7AC1C2AC41ADF18AB1CAD107DDBD7D","total":1}} height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"FFB08F49B2228038C05681C61E0BE2EF15D1DB24A0B1ED43473471FEF423E894","parts":{"hash":"4DF8CDDABB629E60E46EF3641DA80C984C7AC1C2AC41ADF18AB1CAD107DDBD7D","total":1}} height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"FFB08F49B2228038C05681C61E0BE2EF15D1DB24A0B1ED43473471FEF423E894","parts":{"hash":"4DF8CDDABB629E60E46EF3641DA80C984C7AC1C2AC41ADF18AB1CAD107DDBD7D","total":1}} height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"FFB08F49B2228038C05681C61E0BE2EF15D1DB24A0B1ED43473471FEF423E894","parts":{"hash":"4DF8CDDABB629E60E46EF3641DA80C984C7AC1C2AC41ADF18AB1CAD107DDBD7D","total":1}} height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"FFB08F49B2228038C05681C61E0BE2EF15D1DB24A0B1ED43473471FEF423E894","parts":{"hash":"4DF8CDDABB629E60E46EF3641DA80C984C7AC1C2AC41ADF18AB1CAD107DDBD7D","total":1}} height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"FFB08F49B2228038C05681C61E0BE2EF15D1DB24A0B1ED43473471FEF423E894","parts":{"hash":"4DF8CDDABB629E60E46EF3641DA80C984C7AC1C2AC41ADF18AB1CAD107DDBD7D","total":1}} height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"FFB08F49B2228038C05681C61E0BE2EF15D1DB24A0B1ED43473471FEF423E894","parts":{"hash":"4DF8CDDABB629E60E46EF3641DA80C984C7AC1C2AC41ADF18AB1CAD107DDBD7D","total":1}} height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"FFB08F49B2228038C05681C61E0BE2EF15D1DB24A0B1ED43473471FEF423E894","parts":{"hash":"4DF8CDDABB629E60E46EF3641DA80C984C7AC1C2AC41ADF18AB1CAD107DDBD7D","total":1}} height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: BlockPart height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node 3:46AM INF Replay: BlockPart height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF commit is for a block we do not know about; set ProposalBlock=nil commit=78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2 commit_round=0 height=4475276 module=consensus proposal={} server=node 3:46AM INF Replay: New Step height=4475276 module=consensus round=0 server=node step=RoundStepPrecommit 3:46AM INF Replay: New Step height=4475276 module=consensus round=0 server=node step=RoundStepCommit 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 3:46AM INF Replay: BlockPart height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node 3:46AM INF received complete proposal block hash=78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2 height=4475276 module=consensus server=node panic: +2/3 committed an invalid block: wrong Block.Header.AppHash. Expected DAA9072EC7B586617F866645D613C3AE17102594FFCDE76AF467EBB104A727D3, got 4E97CB5B45C698F75F3CC9C19455B25B75513812EEF0DB937C49D97EBB8CEBFE

goroutine 1 [running]: github.com/tendermint/tendermint/consensus.(State).finalizeCommit(0x4000027c00, 0x44498c) /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/vendor/github.com/tendermint/tendermint/consensus/state.go:1595 +0x1074 github.com/tendermint/tendermint/consensus.(State).tryFinalizeCommit(0x4000027c00, 0x44498c) /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/vendor/github.com/tendermint/tendermint/consensus/state.go:1564 +0x30c github.com/tendermint/tendermint/consensus.(State).handleCompleteProposal(0x4000027c00, 0x44498c) /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/vendor/github.com/tendermint/tendermint/consensus/state.go:1943 +0x37c github.com/tendermint/tendermint/consensus.(State).handleMsg(0x4000027c00, {{0x2cf2700, 0x4042150870}, {0x410c6f64e0, 0x28}}) /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/vendor/github.com/tendermint/tendermint/consensus/state.go:837 +0x160 github.com/tendermint/tendermint/consensus.(State).readReplayMessage(0x4000027c00, 0x4023a7eea0, {0x0, 0x0}) /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/vendor/github.com/tendermint/tendermint/consensus/replay.go:81 +0x8b8 github.com/tendermint/tendermint/consensus.(State).catchupReplay(0x4000027c00, 0x44498c) /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/vendor/github.com/tendermint/tendermint/consensus/replay.go:160 +0x6ac github.com/tendermint/tendermint/consensus.(State).OnStart(0x4000027c00) /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/vendor/github.com/tendermint/tendermint/consensus/state.go:324 +0x18c github.com/tendermint/tendermint/libs/service.(BaseService).Start(0x4000027c00) /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/vendor/github.com/tendermint/tendermint/libs/service/service.go:144 +0x3c8 github.com/tendermint/tendermint/consensus.(Reactor).OnStart(0x400135c480) /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/vendor/github.com/tendermint/tendermint/consensus/reactor.go:86 +0x148 github.com/tendermint/tendermint/libs/service.(BaseService).Start(0x400135c480) /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/vendor/github.com/tendermint/tendermint/libs/service/service.go:144 +0x3c8 github.com/tendermint/tendermint/p2p.(Switch).OnStart(0x40b0c34900) /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/vendor/github.com/tendermint/tendermint/p2p/switch.go:227 +0x90 github.com/tendermint/tendermint/libs/service.(BaseService).Start(0x40b0c34900) /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/vendor/github.com/tendermint/tendermint/libs/service/service.go:144 +0x3c8 github.com/tendermint/tendermint/node.(Node).OnStart(0x40fdae82c0) /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/vendor/github.com/tendermint/tendermint/node/node.go:973 +0x398 github.com/tendermint/tendermint/libs/service.(BaseService).Start(0x40fdae82c0) /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/vendor/github.com/tendermint/tendermint/libs/service/service.go:144 +0x3c8 github.com/tharsis/ethermint/server.startInProcess(0x4000200e40, {{0x0, 0x0, 0x0}, {0x2d7b138, 0x40002ab800}, {0x0, 0x0}, {0x2d43e80, 0x40002091c0}, ...}, ...) /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/vendor/github.com/tharsis/ethermint/server/start.go:317 +0xd14 github.com/tharsis/ethermint/server.StartCmd.func2(0x4000258280, {0x40002010c0, 0x0, 0x2}) /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/vendor/github.com/tharsis/ethermint/server/start.go:117 +0x220 github.com/spf13/cobra.(Command).execute(0x4000258280, {0x4000201080, 0x2, 0x2}) /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/vendor/github.com/spf13/cobra/command.go:872 +0x694 github.com/spf13/cobra.(Command).ExecuteC(0x4000782280) /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/vendor/github.com/spf13/cobra/command.go:990 +0x424 github.com/spf13/cobra.(Command).Execute(...) /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/vendor/github.com/spf13/cobra/command.go:918 github.com/spf13/cobra.(Command).ExecuteContext(...) /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/vendor/github.com/spf13/cobra/command.go:911 github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x4000782280, {0x4001680040, 0xd}) /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/vendor/github.com/cosmos/cosmos-sdk/server/cmd/execute.go:36 +0x238 main.main() /build/rzflfgscfghw4fmlz1y37v5n0x90nk30-source/cmd/cronosd/main.go:13 +0x44

Then I change the version to 0.6.5 3:36AM ERR ante handler panicked error="runtime error: invalid memory address or nil pointer dereference" stack trace="goroutine 263 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x88\ngithub.com/tharsis/ethermint/app/ante.Recover({0x2709a88, 0x4000157500}, 0x416fa1c2b0)\n\t/go/pkg/mod/github.com/crypto-org-chain/ethermint@v0.7.2-cronos-6/app/ante/ante.go:132 +0x104\npanic({0x1aaa5a0, 0x361a0a0})\n\t/usr/local/go/src/runtime/panic.go:1038 +0x224\ngithub.com/ethereum/go-ethereum/core/types.NewTx({0x0, 0x0})\n\t/go/pkg/mod/github.com/crypto-org-chain/go-ethereum@v1.10.3-patched/core/types/transaction.go:61 +0x38\ngithub.com/tharsis/ethermint/x/evm/types.MsgEthereumTx.AsTransaction({0x416f994720, 0x0, {0x416d9baf00, 0x42}, {0x0, 0x0}})\n\t/go/pkg/mod/github.com/crypto-org-chain/ethermint@v0.7.2-cronos-6/x/evm/types/msg.go:241 +0x68\ngithub.com/tharsis/ethermint/app/ante.EthSigVerificationDecorator.AnteHandle({{0x2745b98, 0x4000e54700}}, {{0x2708ac8, 0x40000560d0}, {0x27341a0, 0x416f9ac9c0}, {{0xb, 0x0}, {0x400a6f6888, 0x12}, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/crypto-org-chain/ethermint@v0.7.2-cronos-6/app/ante/eth.go:85 +0x3cc\ngithub.com/cosmos/cosmos-sdk/types.ChainAnteDecorators.func1({{0x2708ac8, 0x40000560d0}, {0x27341a0, 0x416f9ac9c0}, {{0xb, 0x0}, {0x400a6f6888, 0x12}, 0x44498b, {0xdc0f38, ...}, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.2/types/handler.go:40 +0xf8\ngithub.com/tharsis/ethermint/app/ante.EthValidateBasicDecorator.AnteHandle({{0x2745b98, 0x4000e54700}}, {{0x2708ac8, 0x40000560d0}, {0x27341a0, 0x416f9ac9c0}, {{0xb, 0x0}, {0x400a6f6888, 0x12}, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/crypto-org-chain/ethermint@v0.7.2-cronos-6/app/ante/eth.go:616 +0x908\ngithub.com/cosmos/cosmos-sdk/types.ChainAnteDecorators.func1({{0x2708ac8, 0x40000560d0}, {0x27341a0, 0x416f9ac9c0}, {{0xb, 0x0}, {0x400a6f6888, 0x12}, 0x44498b, {0xdc0f38, ...}, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.2/types/handler.go:40 +0xf8\ngithub.com/cosmos/cosmos-sdk/x/auth/ante.ValidateMemoDecorator.AnteHandle({{0xfffef5028fd8, 0x40001e1ef0}}, {{0x2708ac8, 0x40000560d0}, {0x27341a0, 0x416f9ac9c0}, {{0xb, 0x0}, {0x400a6f6888, 0x12}, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.2/x/auth/ante/basic.go:66 +0x23c\ngithub.com/cosmos/cosmos-sdk/types.ChainAnteDecorators.func1({{0x2708ac8, 0x40000560d0}, {0x27341a0, 0x416f9ac9c0}, {{0xb, 0x0}, {0x400a6f6888, 0x12}, 0x44498b, {0xdc0f38, ...}, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.2/types/handler.go:40 +0xf8\ngithub.com/cosmos/cosmos-sdk/x/auth/ante.TxTimeoutHeightDecorator.AnteHandle({}, {{0x2708ac8, 0x40000560d0}, {0x27341a0, 0x416f9ac9c0}, {{0xb, 0x0}, {0x400a6f6888, 0x12}, 0x44498b, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.2/x/auth/ante/basic.go:205 +0x21c\ngithub.com/cosmos/cosmos-sdk/types.ChainAnteDecorators.func1({{0x2708ac8, 0x40000560d0}, {0x27341a0, 0x416f9ac9c0}, {{0xb, 0x0}, {0x400a6f6888, 0x12}, 0x44498b, {0xdc0f38, ...}, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.2/types/handler.go:40 +0xf8\ngithub.com/cosmos/cosmos-sdk/x/auth/ante.MempoolFeeDecorator.AnteHandle({}, {{0x2708ac8, 0x40000560d0}, {0x27341a0, 0x416f9ac9c0}, {{0xb, 0x0}, {0x400a6f6888, 0x12}, 0x44498b, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.2/x/auth/ante/fee.go:54 +0x28c\ngithub.com/cosmos/cosmos-sdk/types.ChainAnteDecorators.func1({{0x2708ac8, 0x40000560d0}, {0x27341a0, 0x416f9ac9c0}, {{0xb, 0x0}, {0x400a6f6888, 0x12}, 0x44498b, {0xdc0f38, ...}, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.2/types/handler.go:40 +0xf8\ngithub.com/tharsis/ethermint/app/ante.EthSetupContextDecorator.AnteHandle({}, {{0x2708ac8, 0x40000560d0}, {0x27341a0, 0x416f9ac9c0}, {{0xb, 0x0}, {0x400a6f6888, 0x12}, 0x44498b, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/crypto-org-chain/ethermint@v0.7.2-cronos-6/app/ante/eth.go:635 +0x140\ngithub.com/cosmos/cosmos-sdk/types.ChainAnteDecorators.func1({{0x2708ac8, 0x40000560d0}, {0x27341a0, 0x416f9ac9c0}, {{0xb, 0x0}, {0x400a6f6888, 0x12}, 0x44498b, {0xdc0f38, ...}, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.2/types/handler.go:40 +0xf8\ngithub.com/tharsis/ethermint/app/ante.NewAnteHandler.func1({{0x2708ac8, 0x40000560d0}, {0x27341a0, 0x416f9ac9c0}, {{0xb, 0x0}, {0x400a6f6888, 0x12}, 0x44498b, {0xdc0f38, ...}, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/crypto-org-chain/ethermint@v0.7.2-cronos-6/app/ante/ante.go:77 +0x6e4\ngithub.com/cosmos/cosmos-sdk/baseapp.(BaseApp).runTx(0x400113f380, 0x3, {0x416dc2a000, 0x1b1, 0x1b1})\n\t/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.2/baseapp/baseapp.go:647 +0x500\ngithub.com/cosmos/cosmos-sdk/baseapp.(BaseApp).DeliverTx(0x400113f380, {{0x416dc2a000, 0x1b1, 0x1b1}})\n\t/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.2/baseapp/abci.go:275 +0x158\ngithub.com/tendermint/tendermint/abci/client.(localClient).DeliverTxAsync(0x4027e98960, {{0x416dc2a000, 0x1b1, 0x1b1}})\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/abci/client/local_client.go:93 +0xec\ngithub.com/tendermint/tendermint/proxy.(appConnConsensus).DeliverTxAsync(0x4012a486c0, {{0x416dc2a000, 0x1b1, 0x1b1}})\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/proxy/app_conn.go:85 +0x4c\ngithub.com/tendermint/tendermint/state.execBlockOnProxyApp({0x2709a88, 0x40011b7da0}, {0x271ef50, 0x4012a486c0}, 0x416dc38000, {0x272d6f8, 0x40128958d0}, 0x1)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/state/execution.go:320 +0x7dc\ngithub.com/tendermint/tendermint/state.(BlockExecutor).ApplyBlock(0x408b0cc3f0, {{{0xb, 0x0}, {0x4168072270, 0x7}}, {0x40162081c8, 0x12}, 0x1, 0x44498a, {{0x408b65c820, ...}, ...}, ...}, ...)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/state/execution.go:140 +0x108\ngithub.com/tendermint/tendermint/consensus.(State).finalizeCommit(0x4000243880, 0x44498b)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/state.go:1635 +0x964\ngithub.com/tendermint/tendermint/consensus.(State).tryFinalizeCommit(0x4000243880, 0x44498b)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/state.go:1546 +0x324\ngithub.com/tendermint/tendermint/consensus.(State).addProposalBlockPart(0x4000243880, 0x40145e80f0, {0x401404e690, 0x28})\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/state.go:1919 +0x8fc\ngithub.com/tendermint/tendermint/consensus.(State).handleMsg(0x4000243880, {{0x26c3d60, 0x40145e80f0}, {0x401404e690, 0x28}})\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/state.go:820 +0x100\ngithub.com/tendermint/tendermint/consensus.(State).receiveRoutine(0x4000243880, 0x0)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/state.go:762 +0x324\ncreated by github.com/tendermint/tendermint/consensus.(*State).OnStart\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/state.go:378 +0x120\n"

JayT106 commented 2 years ago

for the block height after 3982500, see "Huygen" Network upgrade, you need to use cronos v0.8.x. Otherwise, the apphash will be mismatched (consensus breaking)

for The cronosd not running last for half an hour when I set feehistory-cap=10, I think it should be a different issue, can you post the log when you were using v0.8.0 at the beginning?

sotatek-manhnguyen8 commented 2 years ago

I got same issues with @PPianAIC. After got snapshot, I started with v0.8.x and stuck for several hours. Only get this log:

7:29AM INF Unlocking keyring
7:29AM INF starting ABCI with Tendermint
yihuang commented 2 years ago

I got same issues with @PPianAIC. After got snapshot, I started with v0.8.x and stuck for several hours. Only get this log:

7:29AM INF Unlocking keyring
7:29AM INF starting ABCI with Tendermint

this is the iavl 0.19 indexing, it only happens for the first time start up.

PPianAIC commented 2 years ago

@yihuang @JayT106 It really took me too much time, It stays in this state for 4-5 days, and then panic, I'm collapsed 7:29AM INF Unlocking keyring 7:29AM INF starting ABCI with Tendermint

This is the version [root@ip-172-31-29-219 cronos-1]# bin/cronosd version v0.8.0

[root@ip-172-31-29-219 cronos-1]# bin/cronosd start --home /data/cronos-1/.cronos 3:02AM INF Unlocking keyring 3:02AM INF starting ABCI with Tendermint 6:17PM INF service start impl=multiAppConn module=proxy msg={} server=node 6:17PM INF service start connection=query impl=localClient module=abci-client msg={} server=node 6:17PM INF service start connection=snapshot impl=localClient module=abci-client msg={} server=node 6:17PM INF service start connection=mempool impl=localClient module=abci-client msg={} server=node 6:17PM INF service start connection=consensus impl=localClient module=abci-client msg={} server=node 6:17PM INF service start impl=EventBus module=events msg={} server=node 6:17PM INF service start impl=PubSub module=pubsub msg={} server=node 6:18PM INF service start impl=IndexerService module=txindex msg={} server=node 6:18PM INF ABCI Handshake App Info hash="ک\a.ǵ�a\u007f�fE�\x13î\x17\x10%����j�g��\x04�'�" height=4475275 module=consensus protocol-version=0 server=node software -version=v0.8.0 6:18PM INF ABCI Replay Blocks appHeight=4475275 module=consensus server=node stateHeight=4475275 storeHeight=4475275 6:18PM INF Completed ABCI Handshake - Tendermint and App are synced appHash="ک\a.ǵ�a\u007f�fE�\x13î\x17\x10%����j�g��\x04�'�" appHeight=4475275 module=consensus server=node 6:18PM INF Version info block=11 p2p=8 server=node tendermint_version=0.34.20 6:18PM INF This node is not a validator addr=C67488AD3E4A9278B7B8CF0A016A404829592F11 module=consensus pubKey=1I4AHZzBMQQimHv5rxQAlXqukQDJW6b4gLO04fIwaIo= server =node 6:18PM INF P2P Node ID ID=ef500d424ba5b4356f4940516091146b35cb102a file=/data/cronos-1/.cronos/config/node_key.json module=p2p server=node 6:18PM INF Adding persistent peers addrs=[] module=p2p server=node 6:18PM INF Adding unconditional peer ids ids=[] module=p2p server=node 6:18PM INF Add our address to book addr={"id":"ef500d424ba5b4356f4940516091146b35cb102a","ip":"0.0.0.0","port":26656} book=/data/cronos-1/.cronos/config/addrbook.json module=p2p server=node 6:18PM INF service start impl=Node msg={} server=node 6:18PM INF Starting pprof server laddr=localhost:6060 server=node 6:18PM INF service start impl="P2P Switch" module=p2p msg={} server=node 6:18PM INF service start impl=BlockchainReactor module=blockchain msg={} server=node 6:18PM INF service start impl=ConsensusReactor module=consensus msg={} server=node 6:18PM INF Reactor module=consensus server=node waitSync=false 6:18PM INF service start impl=ConsensusState module=consensus msg={} server=node 6:18PM INF serve module=rpc-server msg={} server=node 6:18PM INF service start impl=baseWAL module=consensus msg={} server=node wal=/data/cronos-1/.cronos/data/cs.wal/wal 6:18PM INF service start impl=Group module=consensus msg={} server=node wal=/data/cronos-1/.cronos/data/cs.wal/wal 6:18PM INF service start impl=TimeoutTicker module=consensus msg={} server=node 6:18PM INF Searching for height height=4475276 max=710307 min=710208 module=consensus server=node wal=/data/cronos-1/.cronos/data/cs.wal/wal 6:18PM INF Searching for height height=4475275 max=710307 min=710208 module=consensus server=node wal=/data/cronos-1/.cronos/data/cs.wal/wal 6:18PM INF Found height=4475275 index=710307 module=consensus server=node wal=/data/cronos-1/.cronos/data/cs.wal/wal 6:18PM INF Catchup by replaying consensus messages height=4475276 module=consensus server=node 6:18PM INF Replay: New Step height=4475276 module=consensus round=0 server=node step=RoundStepNewHeight 6:18PM INF Replay: Timeout dur=3000 height=4475275 module=consensus round=0 server=node step=3 6:18PM INF Replay: Vote blockID={"hash":"FFB08F49B2228038C05681C61E0BE2EF15D1DB24A0B1ED43473471FEF423E894","parts":{"hash":"4DF8CDDABB629E60E46EF3641DA80C984C7AC1C2AC41ADF18AB1CAD107DDBD7D","total":1}} height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"FFB08F49B2228038C05681C61E0BE2EF15D1DB24A0B1ED43473471FEF423E894","parts":{"hash":"4DF8CDDABB629E60E46EF3641DA80C984C7AC1C2AC41ADF18AB1CAD107DDBD7D","total":1}} height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Timeout dur=-8237.686159 height=4475276 module=consensus round=0 server=node step=1 6:18PM INF Replay: New Step height=4475276 module=consensus round=0 server=node step=RoundStepPropose 6:18PM INF Replay: Vote blockID={"hash":"FFB08F49B2228038C05681C61E0BE2EF15D1DB24A0B1ED43473471FEF423E894","parts":{"hash":"4DF8CDDABB629E60E46EF3641DA80C984C7AC1C2AC41ADF18AB1CAD107DDBD7D","total":1}} height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"FFB08F49B2228038C05681C61E0BE2EF15D1DB24A0B1ED43473471FEF423E894","parts":{"hash":"4DF8CDDABB629E60E46EF3641DA80C984C7AC1C2AC41ADF18AB1CAD107DDBD7D","total":1}} height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"FFB08F49B2228038C05681C61E0BE2EF15D1DB24A0B1ED43473471FEF423E894","parts":{"hash":"4DF8CDDABB629E60E46EF3641DA80C984C7AC1C2AC41ADF18AB1CAD107DDBD7D","total":1}} height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"FFB08F49B2228038C05681C61E0BE2EF15D1DB24A0B1ED43473471FEF423E894","parts":{"hash":"4DF8CDDABB629E60E46EF3641DA80C984C7AC1C2AC41ADF18AB1CAD107DDBD7D","total":1}} height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"FFB08F49B2228038C05681C61E0BE2EF15D1DB24A0B1ED43473471FEF423E894","parts":{"hash":"4DF8CDDABB629E60E46EF3641DA80C984C7AC1C2AC41ADF18AB1CAD107DDBD7D","total":1}} height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"FFB08F49B2228038C05681C61E0BE2EF15D1DB24A0B1ED43473471FEF423E894","parts":{"hash":"4DF8CDDABB629E60E46EF3641DA80C984C7AC1C2AC41ADF18AB1CAD107DDBD7D","total":1}} height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"FFB08F49B2228038C05681C61E0BE2EF15D1DB24A0B1ED43473471FEF423E894","parts":{"hash":"4DF8CDDABB629E60E46EF3641DA80C984C7AC1C2AC41ADF18AB1CAD107DDBD7D","total":1}} height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"FFB08F49B2228038C05681C61E0BE2EF15D1DB24A0B1ED43473471FEF423E894","parts":{"hash":"4DF8CDDABB629E60E46EF3641DA80C984C7AC1C2AC41ADF18AB1CAD107DDBD7D","total":1}} height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: BlockPart height=4475275 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node 6:18PM INF Replay: BlockPart height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7DA710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF commit is for a block we do not know about; set ProposalBlock=nil commit=78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2 commit_round=0 height=4475276 module=consensus proposal={} server=node 6:18PM INF Replay: New Step height=4475276 module=consensus round=0 server=node step=RoundStepPrecommit 6:18PM INF Replay: New Step height=4475276 module=consensus round=0 server=node step=RoundStepCommit 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7D A710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7D A710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7D A710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7D A710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7D A710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7D A710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7D A710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7D A710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7D A710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7D A710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7D A710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7D A710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7D A710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7D A710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7D A710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: Vote blockID={"hash":"78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2","parts":{"hash":"0F093DD3C85F1F56E662C531E946246B40B7D A710DB91D5C9E992E7B7728EAF5","total":1}} height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node type=2 6:18PM INF Replay: BlockPart height=4475276 module=consensus peer=86c60c98519f3a7944baeb37a7ed7d2d6f56872f round=0 server=node 6:18PM INF received complete proposal block hash=78A747E699577841B9C19D0B5A92FE6C7DBD2BD8E47681C65C6CCDC5E7F764C2 height=4475276 module=consensus server=node panic: +2/3 committed an invalid block: wrong Block.Header.AppHash. Expected DAA9072EC7B586617F866645D613C3AE17102594FFCDE76AF467EBB104A727D3, got 4E97CB5B45C698F75F3CC9C19455B25B75513812EEF0DB937C49D97EBB8CEBFE

goroutine 1 [running]: github.com/tendermint/tendermint/consensus.(State).finalizeCommit(0x400022e000, 0x44498c) /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/vendor/github.com/tendermint/tendermint/consensus/state.go:1595 +0xe84 github.com/tendermint/tendermint/consensus.(State).tryFinalizeCommit(0x400022e000, 0x44498c) /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/vendor/github.com/tendermint/tendermint/consensus/state.go:1564 +0x294 github.com/tendermint/tendermint/consensus.(State).handleCompleteProposal(0x400022e000, 0x2bff080?) /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/vendor/github.com/tendermint/tendermint/consensus/state.go:1943 +0x32c github.com/tendermint/tendermint/consensus.(State).handleMsg(0x400022e000, {{0x2bf7c28?, 0x4000159c08?}, {0x40cf087d10?, 0x40e2e956d8?}}) /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/vendor/github.com/tendermint/tendermint/consensus/state.go:837 +0x150 github.com/tendermint/tendermint/consensus.(State).readReplayMessage(0x400022e000, 0x1d3c996?, {0x0?, 0x0?}) /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/vendor/github.com/tendermint/tendermint/consensus/replay.go:81 +0x8d4 github.com/tendermint/tendermint/consensus.(State).catchupReplay(0x400022e000, 0x44498c) /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/vendor/github.com/tendermint/tendermint/consensus/replay.go:160 +0x590 github.com/tendermint/tendermint/consensus.(State).OnStart(0x400022e000) /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/vendor/github.com/tendermint/tendermint/consensus/state.go:324 +0x14c github.com/tendermint/tendermint/libs/service.(BaseService).Start(0x400022e000) /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/vendor/github.com/tendermint/tendermint/libs/service/service.go:144 +0x35c github.com/tendermint/tendermint/consensus.(Reactor).OnStart(0x40010fc480) /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/vendor/github.com/tendermint/tendermint/consensus/reactor.go:86 +0x150 github.com/tendermint/tendermint/libs/service.(BaseService).Start(0x40010fc480) /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/vendor/github.com/tendermint/tendermint/libs/service/service.go:144 +0x35c github.com/tendermint/tendermint/p2p.(Switch).OnStart(0x4000e205a0) /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/vendor/github.com/tendermint/tendermint/p2p/switch.go:227 +0x7c github.com/tendermint/tendermint/libs/service.(BaseService).Start(0x4000e205a0) /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/vendor/github.com/tendermint/tendermint/libs/service/service.go:144 +0x35c github.com/tendermint/tendermint/node.(Node).OnStart(0x40915a8000) /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/vendor/github.com/tendermint/tendermint/node/node.go:973 +0x2e0 github.com/tendermint/tendermint/libs/service.(BaseService).Start(0x40915a8000) /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/vendor/github.com/tendermint/tendermint/libs/service/service.go:144 +0x35c github.com/tharsis/ethermint/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x2c27f80, 0x400152d0e0}, {0x0, 0x0}, {0x2c15df8, 0x40000abbb0}, ...}, ...) /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/vendor/github.com/tharsis/ethermint/server/start.go:321 +0xbac github.com/tharsis/ethermint/server.StartCmd.func2(0x4001131900?, {0x40000b4f80?, 0x0?, 0x2?}) /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/vendor/github.com/tharsis/ethermint/server/start.go:117 +0x1ac github.com/spf13/cobra.(Command).execute(0x4001131900, {0x40000b4f40, 0x2, 0x2}) /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/vendor/github.com/spf13/cobra/command.go:872 +0x4e8 github.com/spf13/cobra.(Command).ExecuteC(0x4000246f00) /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/vendor/github.com/spf13/cobra/command.go:990 +0x360 github.com/spf13/cobra.(Command).Execute(...) /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/vendor/github.com/spf13/cobra/command.go:918 github.com/spf13/cobra.(Command).ExecuteContext(...) /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/vendor/github.com/spf13/cobra/command.go:911 github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x4000094768?, {0x4001148f50, 0xd}) /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/vendor/github.com/cosmos/cosmos-sdk/server/cmd/execute.go:36 +0x1a0 main.main() /build/ijzfm1x2rly7m1rghdyxxr1hxapz5vqm-source/cmd/cronosd/main.go:13 +0x3c

yihuang commented 2 years ago

It really took me too much time, It stays in this state for 4-5 days, and then panic

@PPianAIC app hash mismatch is due to the wrong version of the node, you need to recover by resync from a recent snapshot.

yihuang commented 2 years ago

closing for now, feel free to reopen if is persisted.