Closed shufps closed 1 year ago
A little bit more information:
2023-05-12T11:55:46Z DEBUG Coordinator DeliverTx {"tx": {"Index":5097492,"Parent":"379a4cf4138f7671c58a2f45a0f746267d31df927630538b513962a760bb5e56"}, "issuer": "s7TJIJCXILpffDDd3A+Rab+CQ7UptgH8R3a4ywqMolM="}
2023-05-12T11:55:46Z DEBUG Coordinator DeliverTx {"tx": {"Index":5097492,"Parent":"379a4cf4138f7671c58a2f45a0f746267d31df927630538b513962a760bb5e56"}, "issuer": "Yflf7TC26b8LLQOTj1bTV4n/fw6hItAcXBt+hpUl4hg="}
2023-05-12T11:55:46Z DEBUG Coordinator DeliverTx {"tx": {"Index":5097492,"Parent":"379a4cf4138f7671c58a2f45a0f746267d31df927630538b513962a760bb5e56"}, "issuer": "7el2DH8qqkYYpYoTV3Bc3Bh0lirTaTCVQyMDlLt3VIs="}
2023-05-12T11:55:46Z DEBUG Coordinator create milestone {"state": {"MilestoneHeight":4472423,"MilestoneIndex":5097492,"LastMilestoneID":"f7046e2abae25f8a2751cf3513e3921c05ef35dcba1375911961b0ab95f054ba","LastMilestoneBlockID":"12b4cd7e4fbbc7f4fedb59c9612d58091bb6aacf64c5d15da24e557beaa024a9"}, "parents": ["12b4cd7e4fbbc7f4fedb59c9612d58091bb6aacf64c5d15da24e557beaa024a9","379a4cf4138f7671c58a2f45a0f746267d31df927630538b513962a760bb5e56"]}
2023-05-12T11:55:46Z INFO Tendermint executed block {"module": "state", "height": 4472426, "num_valid_txs": 19, "num_invalid_txs": 0}
2023-05-12T11:55:46Z DEBUG Coordinator broadcast tx {"partial": {"Signature":"BBmrarQ53/pPfI0O/wm720tlnP3wiroQRKi8Fa25JBHQALG40atFf/jytkzy9b5iH7KKIeS7ejwBQeClM7rUAw=="}}
2023-05-12T11:55:46Z INFO Tendermint committed state {"module": "state", "height": 4472426, "num_txs": 19, "app_hash": "EDA0E564BCCEBC8675249D19EE00F512BCA1F806568E15E589B37593FE2935DF"}
2023-05-12T11:55:46Z INFO Tendermint indexed block exents {"module": "txindex", "height": 4472426}
2023-05-12T11:55:46Z INFO Tendermint Timed out {"module": "consensus", "dur": -1.393214164, "height": 4472427, "round": 0, "step": "RoundStepNewHeight"}
2023-05-12T11:55:46Z INFO Coordinator New confirmed milestone: 5097492
2023-05-12T11:55:46Z INFO Tendermint commit is for a block we do not know about; set ProposalBlock=nil {"module": "consensus", "height": 4472427, "commit_round": 0, "proposal": "", "commit": "48D4CB2AC39B52EB52054352B208AB699680DC3A986862ED13D9313DBE02C0A0"}
2023-05-12T11:55:46Z INFO Tendermint received complete proposal block {"module": "consensus", "height": 4472427, "hash": "48D4CB2AC39B52EB52054352B208AB699680DC3A986862ED13D9313DBE02C0A0"}
2023-05-12T11:55:46Z ERROR Tendermint CONSENSUS FAILURE!!! {"module": "consensus", "err": "+2/3 committed an invalid block: wrong Block.Header.AppHash. Expected EDA0E564BCCEBC8675249D19EE00F512BCA1F806568E15E589B37593FE2935DF, got 51DB8330B9D79F3DBD61B181307126A0DB7CF54A652D2AB9CFAC205FD6B62A6F", "stack": "goroutine 216 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2()\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:724 +0x4c\npanic({0x108cbc0, 0xc00264d880})\n\t/usr/local/go/src/runtime/panic.go:884 +0x212\ngithub.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc00030b880, 0x443e6b)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:1594 +0x1105\ngithub.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc00030b880, 0x443e6b)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:1563 +0x2ff\ngithub.com/tendermint/tendermint/consensus.(*State).handleCompleteProposal(0xc00030b880, 0xc9c?)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:1942 +0x399\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc00030b880, {{0x14f4b00?, 0xc003c6a1b0?}, {0xc005edd380?, 0x0?}})\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:834 +0x290\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc00030b880, 0x0)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:760 +0x3f9\ncreated by github.com/tendermint/tendermint/consensus.(*State).OnStart\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:379 +0x12d\n"}
2023-05-12T11:55:46Z INFO Tendermint service stop {"module": "consensus", "wal": "tendermint/data/cs.wal/wal", "msg": "Stopping baseWAL service", "impl": "baseWAL"}
2023-05-12T11:55:46Z INFO Tendermint service stop {"module": "consensus", "wal": "tendermint/data/cs.wal/wal", "msg": "Stopping Group service", "impl": "Group"}
2023-05-12T11:55:48Z INFO Coordinator Proposing parent for milestone 5097493
2023-05-12T11:55:49Z INFO Tendermint Timed out {"module": "consensus", "dur": 3, "height": 4472427, "round": 0, "step": "RoundStepPropose"}
2023-05-12T11:55:56Z INFO Coordinator New confirmed milestone: 5097493
2023-05-12T11:55:59Z INFO Coordinator Proposing parent for milestone 5097494
2023-05-12T11:56:01Z INFO Coordinator New confirmed milestone: 5097494
2023-05-12T11:56:04Z INFO Coordinator Proposing parent for milestone 5097495
2023-05-12T11:56:11Z INFO Coordinator New confirmed milestone: 5097495
2023-05-12T11:56:11Z INFO Coordinator Proposing parent for milestone 5097496
2023-05-12T11:56:11Z WARN Coordinator defaulting to last milestone: tip selection failed: no tips available
Interesting is the: commit is for a block we do not know about
And this also looks interesting: "consensus", "dur": -1.393214164
another notes:
committed state {"module": "state", "height": 4472426, "num_txs": 19, "app_hash": "EDA0E564BCCEBC8675249D19EE00F512BCA1F806568E15E589B37593FE2935DF"}
already contains the wrong app-state and here:
CONSENSUS FAILURE!!! {"module": "consensus", "err": "+2/3 committed an invalid block: wrong Block.Header.AppHash. Expected EDA0E564BCCEBC8675249D19EE00F512BCA1F806568E15E589B37593FE2935DF, got 51DB8330B9D79F3DBD61B181307126A0DB7CF54A652D2AB9CFAC205FD6B62A6F", ...
the "Expected" is wrong and the "got" is correct.
mmhmm, at this place, the state already should be wrong:
https://github.com/iotaledger/inx-tendercoo/blob/develop/pkg/decoo/app.go#L235
According to the hornet logs, hornet had connections issues / took longer for validating a new milestone and the milestone came late:
Coordinator New confirmed milestone: 5097492
It already was confirmed when inx-tendercoo was not yet finished.
edit: Provoking this error by adding a random delay (1-10s) into the confirmedMilestoneListener running on a private tangle didn't work.
Added logging of state: https://github.com/iotaledger/inx-tendercoo/commit/ff9b3ee17f1adb64c77f3411bea316f6371323f6
This happens after a restart from the previously mentioned condition:
2023-05-29T16:54:35Z INFO INX Connecting to node and reading node configuration ...
2023-05-29T16:54:35Z INFO INX Reading node status ...
2023-05-29T16:54:35Z INFO Coordinator Providing Coordinator ...
2023-05-29T16:54:35Z INFO Coordinator Providing Coordinator ... done
2023-05-29T16:54:35Z INFO App Loading components ...
2023-05-29T16:54:35Z INFO App Loading components: INX ... done
2023-05-29T16:54:35Z INFO App Loading components: Coordinator ... done
2023-05-29T16:54:35Z INFO App Loading components: Shutdown ... done
2023-05-29T16:54:35Z INFO App Loading components: Profiling ... done
2023-05-29T16:54:35Z INFO App Executing components ...
2023-05-29T16:54:35Z INFO App Starting component: INX ...
2023-05-29T16:54:35Z INFO App Starting component: Coordinator ...
2023-05-29T16:54:35Z INFO App Starting component: Shutdown ...
2023-05-29T16:54:35Z INFO App Starting component: Profiling ...
2023-05-29T16:54:35Z INFO App Starting background workers ...
2023-05-29T16:54:35Z INFO Coordinator Starting Decentralized Coordinator ...
2023-05-29T16:54:35Z INFO INX Starting NodeBridge ...
2023-05-29T16:54:35Z INFO Coordinator Starting TangleListener ... done
2023-05-29T16:54:35Z INFO Profiling You can now access the profiling server using: http://0.0.0.0:6060/debug/pprof/
2023-05-29T16:54:35Z INFO Coordinator Found private validator {"keyFile": "tendermint/config/priv_validator_key.json", "stateFile": "tendermint/data/priv_validator_state.json"}
2023-05-29T16:54:35Z INFO Coordinator Found node key {"path": "tendermint/config/node_key.json"}
2023-05-29T16:54:35Z INFO Coordinator Found genesis file {"path": "tendermint/config/genesis.json"}
2023-05-29T16:54:35Z INFO Coordinator Node appears to be connected
2023-05-29T16:54:35Z INFO Coordinator Node appears to be synced; latest=9855998 confirmed=9855998
2023-05-29T16:54:40Z INFO Coordinator Coordinator resumed {"state": {"MilestoneHeight":5642115,"MilestoneIndex":9846344,"LastMilestoneID":"b3fa2eb4f8d3d9889a57205f4e15b86278b83882ba28839d343ba7a735ad335e","LastMilestoneBlockID":"2812358e31d74c687ce4f9677448039494257e4d339a8bf8128deb6db6551b5b"}}
2023-05-29T16:54:40Z INFO Tendermint service start {"module": "proxy", "msg": "Starting multiAppConn service", "impl": "multiAppConn"}
2023-05-29T16:54:40Z INFO Tendermint service start {"module": "proxy", "module": "abci-client", "connection": "query", "msg": "Starting localClient service", "impl": "localClient"}
2023-05-29T16:54:40Z INFO Tendermint service start {"module": "proxy", "module": "abci-client", "connection": "snapshot", "msg": "Starting localClient service", "impl": "localClient"}
2023-05-29T16:54:40Z INFO Tendermint service start {"module": "proxy", "module": "abci-client", "connection": "mempool", "msg": "Starting localClient service", "impl": "localClient"}
2023-05-29T16:54:40Z INFO Tendermint service start {"module": "proxy", "module": "abci-client", "connection": "consensus", "msg": "Starting localClient service", "impl": "localClient"}
2023-05-29T16:54:40Z INFO Tendermint service start {"module": "events", "msg": "Starting EventBus service", "impl": "EventBus"}
2023-05-29T16:54:40Z INFO Tendermint service start {"module": "events", "module": "pubsub", "msg": "Starting PubSub service", "impl": "PubSub"}
2023-05-29T16:54:40Z INFO Tendermint service start {"module": "txindex", "msg": "Starting IndexerService service", "impl": "IndexerService"}
2023-05-29T16:54:40Z DEBUG Coordinator ABCI Info {"req": {"version":"0.34.24","block_version":11,"p2p_version":8}}
2023-05-29T16:54:40Z INFO Tendermint ABCI Handshake App Info {"module": "consensus", "height": 5642115, "hash": "XPXZQSZyGaTwSDLYRmjIB4cRCOx8IVnbDy31DFq+MpM=", "software-version": "", "protocol-version": 1}
2023-05-29T16:54:40Z INFO Tendermint ABCI Replay Blocks {"module": "consensus", "appHeight": 5642115, "storeHeight": 5642118, "stateHeight": 5642118}
2023-05-29T16:54:40Z INFO Tendermint Applying block {"module": "consensus", "height": 5642116}
2023-05-29T16:54:40Z INFO Tendermint executed block {"module": "consensus", "height": 5642116, "num_valid_txs": 0, "num_invalid_txs": 0}
2023-05-29T16:54:40Z DEBUG Coordinator Commit-aftercommit {"stateHash": "0x5cf5d941267219a4f04832d84668c807871108ec7c2159db0f2df50c5abe3293", "state": {"MilestoneHeight":5642115,"MilestoneIndex":9846344,"LastMilestoneID":"b3fa2eb4f8d3d9889a57205f4e15b86278b83882ba28839d343ba7a735ad335e","LastMilestoneBlockID":"2812358e31d74c687ce4f9677448039494257e4d339a8bf8128deb6db6551b5b","ParentByIssuer":{},"IssuerCountByParent":{},"ProofIssuersByBlockID":{},"Milestone":null,"SignaturesByIssuer":{}}}
2023-05-29T16:54:40Z INFO Tendermint Applying block {"module": "consensus", "height": 5642117}
2023-05-29T16:54:40Z DEBUG Coordinator DeliverTx {"tx": {"Index":9846344,"BlockID":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "PfgJZMyDkh4IwfoKT1/AWBCmNNpFRhsrMV/L/WL3yrc="}
2023-05-29T16:54:40Z DEBUG Coordinator DeliverTx {"tx": {"Index":9846344,"BlockID":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "2ZIoGaOelN3zkH9LnI35PznwJiRPy2CSBbmoeQIlmfI="}
2023-05-29T16:54:40Z DEBUG Coordinator DeliverTx {"tx": {"Index":9846344,"BlockID":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "jiIq5+Ktz7h6KYShmq1SsZee0UcsPLFyOac+8dNEw1o="}
2023-05-29T16:54:40Z DEBUG Coordinator DeliverTx {"tx": {"Index":9846344,"BlockID":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "ZLx8094lTMOipuCq7luEHQpnGyK0h3iPWU+/HvMpk1w="}
2023-05-29T16:54:40Z DEBUG Coordinator DeliverTx {"tx": {"Index":9846344,"BlockID":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "+dllamAEkIPu9hSHYyGHs1EpTB+iPRGAYNgT220D6LY="}
2023-05-29T16:54:40Z DEBUG Coordinator DeliverTx {"tx": {"Index":9846344,"BlockID":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "r+rCxSoK09pTe6h9AXJyD9VfpP2zxEsP0gvIEtGKiFY="}
2023-05-29T16:54:40Z DEBUG Coordinator DeliverTx {"tx": {"Index":9846344,"BlockID":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "o74D/yOQIQNUqa9r60GKlXdmsFD1p8YmYoA8wmepRPY="}
2023-05-29T16:54:40Z DEBUG Coordinator DeliverTx {"tx": {"Index":9846344,"BlockID":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "2F5bFZDYmNHgzeuy47UzfIt2JwFCZj14gRaDukfBfJg="}
2023-05-29T16:54:40Z DEBUG Coordinator DeliverTx {"tx": {"Index":9846344,"BlockID":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "GVltVw6VNfEcw0CR25jsRKUm44ZX+Ca11rYVLlM7jAY="}
2023-05-29T16:54:40Z DEBUG Coordinator DeliverTx {"tx": {"Index":9846344,"BlockID":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "Kml39uVHUY1XLXumP2tQtbuvPegCPcVCdRcj85Epe4s="}
2023-05-29T16:54:40Z DEBUG Coordinator awaiting parent {"blockID": "3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}
2023-05-29T16:54:40Z INFO Tendermint executed block {"module": "consensus", "height": 5642117, "num_valid_txs": 10, "num_invalid_txs": 0}
2023-05-29T16:54:40Z DEBUG Coordinator Commit-aftercommit {"stateHash": "0x175f6ff18437540c7628ef198cfa96e0b67e9509a63ab8f8490aa60c4a0209d7", "state": {"MilestoneHeight":5642115,"MilestoneIndex":9846344,"LastMilestoneID":"b3fa2eb4f8d3d9889a57205f4e15b86278b83882ba28839d343ba7a735ad335e","LastMilestoneBlockID":"2812358e31d74c687ce4f9677448039494257e4d339a8bf8128deb6db6551b5b","ParentByIssuer":{"19596d570e9535f11cc34091db98ec44a526e38657f826b5d6b6152e533b8c06":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","2a6977f6e547518d572d7ba63f6b50b5bbaf3de8023dc542751723f391297b8b":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","3df80964cc83921e08c1fa0a4f5fc05810a634da45461b2b315fcbfd62f7cab7":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","64bc7cd3de254cc3a2a6e0aaee5b841d0a671b22b487788f594fbf1ef329935c":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","8e222ae7e2adcfb87a2984a19aad52b1979ed1472c3cb17239a73ef1d344c35a":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","a3be03ff2390210354a9af6beb418a957766b050f5a7c62662803cc267a944f6":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","afeac2c52a0ad3da537ba87d0172720fd55fa4fdb3c44b0fd20bc812d18a8856":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","d85e5b1590d898d1e0cdebb2e3b5337c8b76270142663d78811683ba47c17c98":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","d9922819a39e94ddf3907f4b9c8df93f39f026244fcb609205b9a879022599f2":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","f9d9656a60049083eef61487632187b351294c1fa23d118060d813db6d03e8b6":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"},"IssuerCountByParent":{"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5":10},"ProofIssuersByBlockID":{},"Milestone":null,"SignaturesByIssuer":{}}}
2023-05-29T16:54:40Z INFO Tendermint Applying block {"module": "consensus", "height": 5642118}
2023-05-29T16:54:40Z DEBUG Coordinator DeliverTx {"tx": {"Index":9846344,"Parent":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "r+rCxSoK09pTe6h9AXJyD9VfpP2zxEsP0gvIEtGKiFY="}
panic: state.AppHash does not match AppHash after replay. Got
2023-05-29T16:54:40Z DEBUG Coordinator DeliverTx {"tx": {"Index":9846344,"Parent":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "Kml39uVHUY1XLXumP2tQtbuvPegCPcVCdRcj85Epe4s="}
2023-05-29T16:54:40Z DEBUG Coordinator DeliverTx {"tx": {"Index":9846344,"Parent":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "o74D/yOQIQNUqa9r60GKlXdmsFD1p8YmYoA8wmepRPY="}
2C3D69BB07BA4DDDB514C124BD8CF395E8AFCD04CE3A99422C891084DCA7B22A, expected D1F7546E034D1BCB5D161CF11F12C95014C0E4EB41161A0C27ACFF3AEE17744F.
2023-05-29T16:54:40Z DEBUG Coordinator DeliverTx {"tx": {"Index":9846344,"Parent":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "2ZIoGaOelN3zkH9LnI35PznwJiRPy2CSBbmoeQIlmfI="}
State: {{{11 1} 0.34.24} alphanet-10 1 5642118 9CCEB8CB938D9232080DB913FE946E419F8B13CA320DE25313BFB1C65100DD06:1:0043345B4213 2023-05-29 00:00:06.274769542 +0000 UTC ValidatorSet{
2023-05-29T16:54:40Z DEBUG Coordinator DeliverTx {"tx": {"Index":9846344,"Parent":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "ZLx8094lTMOipuCq7luEHQpnGyK0h3iPWU+/HvMpk1w="}
2023-05-29T16:54:40Z DEBUG Coordinator DeliverTx {"tx": {"Index":9846344,"Parent":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "PfgJZMyDkh4IwfoKT1/AWBCmNNpFRhsrMV/L/WL3yrc="}
Proposer: Validator{9234C27379290332CA13772144DB4E12FCA1FFAD PubKeyEd25519{AFEAC2C52A0AD3DA537BA87D0172720FD55FA4FDB3C44B0FD20BC812D18A8856} VP:1 A:0}
2023-05-29T16:54:40Z DEBUG Coordinator DeliverTx {"tx": {"Index":9846344,"Parent":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "+dllamAEkIPu9hSHYyGHs1EpTB+iPRGAYNgT220D6LY="}
Validators:
Validator{008CB3E446C123CD0BE3D1C8BBDFA555CFB600E7 PubKeyEd25519{19596D570E9535F11CC34091DB98EC44A526E38657F826B5D6B6152E533B8C06} VP:1 A:0}
2023-05-29T16:54:40Z DEBUG Coordinator DeliverTx {"tx": {"Index":9846344,"Parent":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "GVltVw6VNfEcw0CR25jsRKUm44ZX+Ca11rYVLlM7jAY="}
Validator{0223EB172703E02C2B73D9710F535B8C01480235 PubKeyEd25519{64BC7CD3DE254CC3A2A6E0AAEE5B841D0A671B22B487788F594FBF1EF329935C} VP:1 A:0}
2023-05-29T16:54:40Z DEBUG Coordinator DeliverTx {"tx": {"Index":9846344,"Parent":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "jiIq5+Ktz7h6KYShmq1SsZee0UcsPLFyOac+8dNEw1o="}
Validator{04A5FE0DA11994D8B399860F7A704BF46E1FEBED PubKeyEd25519{A3BE03FF2390210354A9AF6BEB418A957766B050F5A7C62662803CC267A944F6} VP:1 A:0}
2023-05-29T16:54:40Z DEBUG Coordinator DeliverTx {"tx": {"Index":9846344,"Parent":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "2F5bFZDYmNHgzeuy47UzfIt2JwFCZj14gRaDukfBfJg="}
2023-05-29T16:54:40Z DEBUG Coordinator create milestone {"state": {"MilestoneHeight":5642115,"MilestoneIndex":9846344,"LastMilestoneID":"b3fa2eb4f8d3d9889a57205f4e15b86278b83882ba28839d343ba7a735ad335e","LastMilestoneBlockID":"2812358e31d74c687ce4f9677448039494257e4d339a8bf8128deb6db6551b5b"}, "parents": ["2812358e31d74c687ce4f9677448039494257e4d339a8bf8128deb6db6551b5b","3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"]}
Validator{0E27BBBE79F7D010439C78273D1B35F1CE4CFAF0 PubKeyEd25519{D9922819A39E94DDF3907F4B9C8DF93F39F026244FCB609205B9A879022599F2} VP:1 A:0}
2023-05-29T16:54:40Z INFO Tendermint executed block {"module": "consensus", "height": 5642118, "num_valid_txs": 10, "num_invalid_txs": 0}
Validator{10849BB652FED684976249438116C6FD3552C035 PubKeyEd25519{D85E5B1590D898D1E0CDEBB2E3B5337C8B76270142663D78811683BA47C17C98} VP:1 A:0}
2023-05-29T16:54:40Z DEBUG Coordinator broadcast tx {"partial": {"Signature":"2RuLdDgCIlZKd+EEYaraC56ZcwyElJSHP2CL6unw7QsBEbD0FRf4o/92YVLGxJ6Sm4/mtF+mSm4yqwlLZL6gBQ=="}}
2023-05-29T16:54:40Z DEBUG Coordinator Commit-aftercommit {"stateHash": "0x2c3d69bb07ba4dddb514c124bd8cf395e8afcd04ce3a99422c891084dca7b22a", "state": {"MilestoneHeight":5642115,"MilestoneIndex":9846344,"LastMilestoneID":"b3fa2eb4f8d3d9889a57205f4e15b86278b83882ba28839d343ba7a735ad335e","LastMilestoneBlockID":"2812358e31d74c687ce4f9677448039494257e4d339a8bf8128deb6db6551b5b","ParentByIssuer":{"19596d570e9535f11cc34091db98ec44a526e38657f826b5d6b6152e533b8c06":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","2a6977f6e547518d572d7ba63f6b50b5bbaf3de8023dc542751723f391297b8b":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","3df80964cc83921e08c1fa0a4f5fc05810a634da45461b2b315fcbfd62f7cab7":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","64bc7cd3de254cc3a2a6e0aaee5b841d0a671b22b487788f594fbf1ef329935c":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","8e222ae7e2adcfb87a2984a19aad52b1979ed1472c3cb17239a73ef1d344c35a":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","a3be03ff2390210354a9af6beb418a957766b050f5a7c62662803cc267a944f6":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","afeac2c52a0ad3da537ba87d0172720fd55fa4fdb3c44b0fd20bc812d18a8856":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","d85e5b1590d898d1e0cdebb2e3b5337c8b76270142663d78811683ba47c17c98":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","d9922819a39e94ddf3907f4b9c8df93f39f026244fcb609205b9a879022599f2":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","f9d9656a60049083eef61487632187b351294c1fa23d118060d813db6d03e8b6":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"},"IssuerCountByParent":{"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5":10},"ProofIssuersByBlockID":{"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5":{"19596d570e9535f11cc34091db98ec44a526e38657f826b5d6b6152e533b8c06":{},"2a6977f6e547518d572d7ba63f6b50b5bbaf3de8023dc542751723f391297b8b":{},"3df80964cc83921e08c1fa0a4f5fc05810a634da45461b2b315fcbfd62f7cab7":{},"64bc7cd3de254cc3a2a6e0aaee5b841d0a671b22b487788f594fbf1ef329935c":{},"8e222ae7e2adcfb87a2984a19aad52b1979ed1472c3cb17239a73ef1d344c35a":{},"a3be03ff2390210354a9af6beb418a957766b050f5a7c62662803cc267a944f6":{},"afeac2c52a0ad3da537ba87d0172720fd55fa4fdb3c44b0fd20bc812d18a8856":{},"d85e5b1590d898d1e0cdebb2e3b5337c8b76270142663d78811683ba47c17c98":{},"d9922819a39e94ddf3907f4b9c8df93f39f026244fcb609205b9a879022599f2":{},"f9d9656a60049083eef61487632187b351294c1fa23d118060d813db6d03e8b6":{}}},"Milestone":{"type":7,"index":9846344,"timestamp":1685318406,"protocolVersion":2,"previousMilestoneId":"0xb3fa2eb4f8d3d9889a57205f4e15b86278b83882ba28839d343ba7a735ad335e","parents":["0x2812358e31d74c687ce4f9677448039494257e4d339a8bf8128deb6db6551b5b","0x3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"],"inclusionMerkleRoot":"0x46b68298fe97239abedbd44063e1aefe50e5299be9c813ffaffa45bf762b55e7","appliedMerkleRoot":"0x0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8","metadata":"0x83175600000000002812358e31d74c687ce4f9677448039494257e4d339a8bf8128deb6db6551b5b","signatures":[]},"SignaturesByIssuer":{}}}
Validator{1093D97B9310C072EED7A4957DFEDE346C76B13D PubKeyEd25519{8E222AE7E2ADCFB87A2984A19AAD52B1979ED1472C3CB17239A73EF1D344C35A} VP:1 A:0}
Validator{355DBE857FED7715F9C2AC12AE1A0E9E7C316EBD PubKeyEd25519{3DF80964CC83921E08C1FA0A4F5FC05810A634DA45461B2B315FCBFD62F7CAB7} VP:1 A:0}
Validator{367CA8B17443585A2D013DA2916C10BFEC29D7F7 PubKeyEd25519{2A6977F6E547518D572D7BA63F6B50B5BBAF3DE8023DC542751723F391297B8B} VP:1 A:0}
Validator{840399906E468F3E87FA8C0865638D6F29D5C2EA PubKeyEd25519{F9D9656A60049083EEF61487632187B351294C1FA23D118060D813DB6D03E8B6} VP:1 A:0}
Validator{9234C27379290332CA13772144DB4E12FCA1FFAD PubKeyEd25519{AFEAC2C52A0AD3DA537BA87D0172720FD55FA4FDB3C44B0FD20BC812D18A8856} VP:1 A:0}
} ValidatorSet{
Proposer: Validator{840399906E468F3E87FA8C0865638D6F29D5C2EA PubKeyEd25519{F9D9656A60049083EEF61487632187B351294C1FA23D118060D813DB6D03E8B6} VP:1 A:-1}
Validators:
Validator{008CB3E446C123CD0BE3D1C8BBDFA555CFB600E7 PubKeyEd25519{19596D570E9535F11CC34091DB98EC44A526E38657F826B5D6B6152E533B8C06} VP:1 A:-1}
Validator{0223EB172703E02C2B73D9710F535B8C01480235 PubKeyEd25519{64BC7CD3DE254CC3A2A6E0AAEE5B841D0A671B22B487788F594FBF1EF329935C} VP:1 A:-1}
Validator{04A5FE0DA11994D8B399860F7A704BF46E1FEBED PubKeyEd25519{A3BE03FF2390210354A9AF6BEB418A957766B050F5A7C62662803CC267A944F6} VP:1 A:-1}
Validator{0E27BBBE79F7D010439C78273D1B35F1CE4CFAF0 PubKeyEd25519{D9922819A39E94DDF3907F4B9C8DF93F39F026244FCB609205B9A879022599F2} VP:1 A:-1}
Validator{10849BB652FED684976249438116C6FD3552C035 PubKeyEd25519{D85E5B1590D898D1E0CDEBB2E3B5337C8B76270142663D78811683BA47C17C98} VP:1 A:-1}
Validator{1093D97B9310C072EED7A4957DFEDE346C76B13D PubKeyEd25519{8E222AE7E2ADCFB87A2984A19AAD52B1979ED1472C3CB17239A73EF1D344C35A} VP:1 A:-1}
Validator{355DBE857FED7715F9C2AC12AE1A0E9E7C316EBD PubKeyEd25519{3DF80964CC83921E08C1FA0A4F5FC05810A634DA45461B2B315FCBFD62F7CAB7} VP:1 A:-1}
Validator{367CA8B17443585A2D013DA2916C10BFEC29D7F7 PubKeyEd25519{2A6977F6E547518D572D7BA63F6B50B5BBAF3DE8023DC542751723F391297B8B} VP:1 A:-1}
Validator{840399906E468F3E87FA8C0865638D6F29D5C2EA PubKeyEd25519{F9D9656A60049083EEF61487632187B351294C1FA23D118060D813DB6D03E8B6} VP:1 A:-1}
Validator{9234C27379290332CA13772144DB4E12FCA1FFAD PubKeyEd25519{AFEAC2C52A0AD3DA537BA87D0172720FD55FA4FDB3C44B0FD20BC812D18A8856} VP:1 A:9}
} ValidatorSet{
Proposer: Validator{367CA8B17443585A2D013DA2916C10BFEC29D7F7 PubKeyEd25519{2A6977F6E547518D572D7BA63F6B50B5BBAF3DE8023DC542751723F391297B8B} VP:1 A:-2}
Validators:
Validator{008CB3E446C123CD0BE3D1C8BBDFA555CFB600E7 PubKeyEd25519{19596D570E9535F11CC34091DB98EC44A526E38657F826B5D6B6152E533B8C06} VP:1 A:-2}
Validator{0223EB172703E02C2B73D9710F535B8C01480235 PubKeyEd25519{64BC7CD3DE254CC3A2A6E0AAEE5B841D0A671B22B487788F594FBF1EF329935C} VP:1 A:-2}
Validator{04A5FE0DA11994D8B399860F7A704BF46E1FEBED PubKeyEd25519{A3BE03FF2390210354A9AF6BEB418A957766B050F5A7C62662803CC267A944F6} VP:1 A:-2}
Validator{0E27BBBE79F7D010439C78273D1B35F1CE4CFAF0 PubKeyEd25519{D9922819A39E94DDF3907F4B9C8DF93F39F026244FCB609205B9A879022599F2} VP:1 A:-2}
Validator{10849BB652FED684976249438116C6FD3552C035 PubKeyEd25519{D85E5B1590D898D1E0CDEBB2E3B5337C8B76270142663D78811683BA47C17C98} VP:1 A:-2}
Validator{1093D97B9310C072EED7A4957DFEDE346C76B13D PubKeyEd25519{8E222AE7E2ADCFB87A2984A19AAD52B1979ED1472C3CB17239A73EF1D344C35A} VP:1 A:-2}
Validator{355DBE857FED7715F9C2AC12AE1A0E9E7C316EBD PubKeyEd25519{3DF80964CC83921E08C1FA0A4F5FC05810A634DA45461B2B315FCBFD62F7CAB7} VP:1 A:-2}
Validator{367CA8B17443585A2D013DA2916C10BFEC29D7F7 PubKeyEd25519{2A6977F6E547518D572D7BA63F6B50B5BBAF3DE8023DC542751723F391297B8B} VP:1 A:-2}
Validator{840399906E468F3E87FA8C0865638D6F29D5C2EA PubKeyEd25519{F9D9656A60049083EEF61487632187B351294C1FA23D118060D813DB6D03E8B6} VP:1 A:8}
Validator{9234C27379290332CA13772144DB4E12FCA1FFAD PubKeyEd25519{AFEAC2C52A0AD3DA537BA87D0172720FD55FA4FDB3C44B0FD20BC812D18A8856} VP:1 A:8}
} 1 {{22020096 -1 500} {100000 48h0m0s 1048576} {[ed25519]} {0}} 1 [205 200 243 152 215 234 219 86 61 184 125 106 182 17 235 31 154 35 138 248 190 252 154 83 54 183 228 193 159 16 214 179] [209 247 84 110 3 77 27 203 93 22 28 241 31 18 201 80 20 192 228 235 65 22 26 12 39 172 255 58 238 23 116 79]}
Did you reset Tendermint without resetting your application's data?
goroutine 33 [running]:
github.com/tendermint/tendermint/consensus.assertAppHashEqualsOneFromState({_, _, _}, {{{0xb, 0x1}, {0xc0015b3f98, 0x7}}, {0xc0015b3fc0, 0xb}, 0x1, ...})
/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/replay.go:525 +0x12a
github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlocks(_, {{{0xb, 0x1}, {0xc0015b3f98, 0x7}}, {0xc0015b3fc0, 0xb}, 0x1, 0x561786, {{0xc000ecc720, ...}, ...}, ...}, ...)
/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/replay.go:488 +0x3e9
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x1}, {0xc0015b3f98, 0x7}}, {0xc0015b3fc0, 0xb}, 0x1, 0x561786, {{0xc000ecc720, ...}, ...}, ...}, ...)
/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/replay.go:393 +0x479
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc001016e48, {0x1539968, 0xc0010ead00})
/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/replay.go:268 +0x3d4
github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x1}, {0xc0015b3f98, 0x7}}, {0xc0015b3fc0, 0xb}, 0x1, 0x561786, ...}, ...)
/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/node/node.go:329 +0x1b8
github.com/tendermint/tendermint/node.NewNode(0xc0000c8140, {0x152b2d0, 0xc0002b6640}, 0xc0010ef2c0, {0x15222c0, 0xc0010f2c60}, 0x11b71ba93?, 0x1db0080?, 0xc0010ef300, {0x152fc40, ...}, ...)
/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/node/node.go:777 +0x597
github.com/iotaledger/inx-tendercoo/components/decoo.newTendermintNode(0xc0000c8140)
/scratch/components/decoo/tendermint.go:204 +0x4b6
github.com/iotaledger/inx-tendercoo/components/decoo.run.func2({0x152e350, 0xc0000e2320})
/scratch/components/decoo/component.go:216 +0x1a8
github.com/iotaledger/hive.go/app/daemon.(*OrderedDaemon).runBackgroundWorker.func1()
/go/pkg/mod/github.com/iotaledger/hive.go/app@v0.0.0-20230417125513-e2e89991217f/daemon/daemon.go:163 +0xf2
created by github.com/iotaledger/hive.go/app/daemon.(*OrderedDaemon).runBackgroundWorker
/go/pkg/mod/github.com/iotaledger/hive.go/app@v0.0.0-20230417125513-e2e89991217f/daemon/daemon.go:158 +0x13a
this time with more logging outputs:
2023-06-02T00:09:26Z DEBUG Coordinator create milestone {"state": {"MilestoneHeight":5906520,"MilestoneIndex":9912319,"LastMilestoneID":"04f35ecb367d18ecd75d260fb12c38afa984212d587ac803c33f7f06b3d5d8a2","LastMilestoneBlockID":"56def2f78d6073371a5f07945a4e
6cfebb298bb1918413846bbb980190799195"}, "parents": ["56def2f78d6073371a5f07945a4e6cfebb298bb1918413846bbb980190799195","ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02"]}
2023-06-02T00:09:26Z INFO Tendermint executed block {"module": "state", "height": 5906523, "num_valid_txs": 17, "num_invalid_txs": 0}
2023-06-02T00:09:26Z DEBUG Coordinator broadcast tx {"partial": {"Signature":"ou7uNWvq/j5Rt1HnR63Z174CLDlPFzFj4IJIOPctznp2+v0vXCLlNzEPdpSYyjDS7XjfalNxkcR6MNDTbI13Cg=="}}
2023-06-02T00:09:26Z DEBUG Coordinator Commit-aftercommit {"stateHash": "0x5437222422ac24d5d0af94daac8a9b92ce5081f36a2c07e39c998778c533e980", "state": {"MilestoneHeight":5906520,"MilestoneIndex":9912319,"LastMilestoneID":"04f35ecb367d18ecd75d260fb12c38afa98
4212d587ac803c33f7f06b3d5d8a2","LastMilestoneBlockID":"56def2f78d6073371a5f07945a4e6cfebb298bb1918413846bbb980190799195","ParentByIssuer":{"19596d570e9535f11cc34091db98ec44a526e38657f826b5d6b6152e533b8c06":"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02
","2a6977f6e547518d572d7ba63f6b50b5bbaf3de8023dc542751723f391297b8b":"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02","3df80964cc83921e08c1fa0a4f5fc05810a634da45461b2b315fcbfd62f7cab7":"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02","
64bc7cd3de254cc3a2a6e0aaee5b841d0a671b22b487788f594fbf1ef329935c":"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02","8e222ae7e2adcfb87a2984a19aad52b1979ed1472c3cb17239a73ef1d344c35a":"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02","a3b
e03ff2390210354a9af6beb418a957766b050f5a7c62662803cc267a944f6":"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02","afeac2c52a0ad3da537ba87d0172720fd55fa4fdb3c44b0fd20bc812d18a8856":"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02","d85e5b
1590d898d1e0cdebb2e3b5337c8b76270142663d78811683ba47c17c98":"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02","f9d9656a60049083eef61487632187b351294c1fa23d118060d813db6d03e8b6":"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02"},"IssuerCo
untByParent":{"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02":9},"ProofIssuersByBlockID":{"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02":{"19596d570e9535f11cc34091db98ec44a526e38657f826b5d6b6152e533b8c06":{},"2a6977f6e547518d572d7ba
63f6b50b5bbaf3de8023dc542751723f391297b8b":{},"3df80964cc83921e08c1fa0a4f5fc05810a634da45461b2b315fcbfd62f7cab7":{},"64bc7cd3de254cc3a2a6e0aaee5b841d0a671b22b487788f594fbf1ef329935c":{},"8e222ae7e2adcfb87a2984a19aad52b1979ed1472c3cb17239a73ef1d344c35a":{},"a3be03ff239021
0354a9af6beb418a957766b050f5a7c62662803cc267a944f6":{},"afeac2c52a0ad3da537ba87d0172720fd55fa4fdb3c44b0fd20bc812d18a8856":{},"d85e5b1590d898d1e0cdebb2e3b5337c8b76270142663d78811683ba47c17c98":{},"f9d9656a60049083eef61487632187b351294c1fa23d118060d813db6d03e8b6":{}}},"Mil
estone":{"type":7,"index":9912319,"timestamp":1685664559,"protocolVersion":2,"previousMilestoneId":"0x04f35ecb367d18ecd75d260fb12c38afa984212d587ac803c33f7f06b3d5d8a2","parents":["0x56def2f78d6073371a5f07945a4e6cfebb298bb1918413846bbb980190799195","0xba6dee36651960e60dda
4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02"],"inclusionMerkleRoot":"0x8be25cc05159ffedab154e251971e1c28a83821e6515a13b8944fa7c795734fa","appliedMerkleRoot":"0x2e22bd402ccccfd221546829a848d83b315b629f4565aa09b597b89ca76760f8","metadata":"0x58205a000000000056def2f78d6073
371a5f07945a4e6cfebb298bb1918413846bbb980190799195","signatures":[]},"SignaturesByIssuer":{}}}
2023-06-02T00:09:26Z INFO Tendermint committed state {"module": "state", "height": 5906523, "num_txs": 17, "app_hash": "5437222422AC24D5D0AF94DAAC8A9B92CE5081F36A2C07E39C998778C533E980"}
2023-06-02T00:09:26Z INFO Tendermint indexed block exents {"module": "txindex", "height": 5906523}
2023-06-02T00:09:26Z INFO Coordinator New confirmed milestone: 9912319
2023-06-02T00:09:26Z INFO Coordinator Proposing parent for milestone 9912320
2023-06-02T00:09:26Z WARN Coordinator defaulting to last milestone: tip selection failed: no tips available
2023-06-02T00:09:26Z INFO Tendermint commit is for a block we do not know about; set ProposalBlock=nil {"module": "consensus", "height": 5906524, "commit_round": 1, "proposal": "", "commit": "C6F348DECCF270BF2B111BD1E015781DAEB03398EA7ACAB44437D8417F0642
54"}
2023-06-02T00:09:26Z INFO Tendermint received complete proposal block {"module": "consensus", "height": 5906524, "hash": "C6F348DECCF270BF2B111BD1E015781DAEB03398EA7ACAB44437D8417F064254"}
2023-06-02T00:09:26Z ERROR Tendermint CONSENSUS FAILURE!!! {"module": "consensus", "err": "+2/3 committed an invalid block: wrong Block.Header.AppHash. Expected 5437222422AC24D5D0AF94DAAC8A9B92CE5081F36A2C07E39C998778C533E980, got E0269E863385CEE700A77FC0AF
5CF975F88C79BD8A21234F2FC0F28BD3C066C1", "stack": "goroutine 7609 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2()\n\t/go/pkg/mod/github.com/tendermint/tende
rmint@v0.34.24/consensus/state.go:724 +0x4c\npanic({0x10ca000, 0xc004c7c8c0})\n\t/usr/local/go/src/runtime/panic.go:884 +0x213\ngithub.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc002a32700, 0x5a205c)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.3
4.24/consensus/state.go:1594 +0x1105\ngithub.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc002a32700, 0x5a205c)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:1563 +0x2ff\ngithub.com/tendermint/tendermint/consensus.(*State
).handleCompleteProposal(0xc002a32700, 0xc004e84c80?)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:1942 +0x399\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc002a32700, {{0x15220c0, 0xc0049acd50}, {0xc001fc7020, 0x28}})\n
\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:834 +0x1b7\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc002a32700, 0x0)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:760 +0x3f0\ncreated by
github.com/tendermint/tendermint/consensus.(*State).OnStart\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:379 +0x12d\n"}
The states differ ...
Good:
{
"stateHash": "0xe0269e863385cee700a77fc0af5cf975f88c79bd8a21234f2fc0f28bd3c066c1",
"state": {
"MilestoneHeight": 5906520,
"MilestoneIndex": 9912319,
"LastMilestoneID": "04f35ecb367d18ecd75d260fb12c38afa984212d587ac803c33f7f06b3d5d8a2",
"LastMilestoneBlockID": "56def2f78d6073371a5f07945a4e6cfebb298bb1918413846bbb980190799195",
"ParentByIssuer": {
"19596d570e9535f11cc34091db98ec44a526e38657f826b5d6b6152e533b8c06": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
"2a6977f6e547518d572d7ba63f6b50b5bbaf3de8023dc542751723f391297b8b": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
"3df80964cc83921e08c1fa0a4f5fc05810a634da45461b2b315fcbfd62f7cab7": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
"64bc7cd3de254cc3a2a6e0aaee5b841d0a671b22b487788f594fbf1ef329935c": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
"8e222ae7e2adcfb87a2984a19aad52b1979ed1472c3cb17239a73ef1d344c35a": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
"a3be03ff2390210354a9af6beb418a957766b050f5a7c62662803cc267a944f6": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
"afeac2c52a0ad3da537ba87d0172720fd55fa4fdb3c44b0fd20bc812d18a8856": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
"d85e5b1590d898d1e0cdebb2e3b5337c8b76270142663d78811683ba47c17c98": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
"f9d9656a60049083eef61487632187b351294c1fa23d118060d813db6d03e8b6": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02"
},
"IssuerCountByParent": {
"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02": 9
},
"ProofIssuersByBlockID": {
"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02": {
"19596d570e9535f11cc34091db98ec44a526e38657f826b5d6b6152e533b8c06": {},
"2a6977f6e547518d572d7ba63f6b50b5bbaf3de8023dc542751723f391297b8b": {},
"3df80964cc83921e08c1fa0a4f5fc05810a634da45461b2b315fcbfd62f7cab7": {},
"64bc7cd3de254cc3a2a6e0aaee5b841d0a671b22b487788f594fbf1ef329935c": {},
"8e222ae7e2adcfb87a2984a19aad52b1979ed1472c3cb17239a73ef1d344c35a": {},
"a3be03ff2390210354a9af6beb418a957766b050f5a7c62662803cc267a944f6": {},
"afeac2c52a0ad3da537ba87d0172720fd55fa4fdb3c44b0fd20bc812d18a8856": {},
"d85e5b1590d898d1e0cdebb2e3b5337c8b76270142663d78811683ba47c17c98": {},
"f9d9656a60049083eef61487632187b351294c1fa23d118060d813db6d03e8b6": {}
}
},
"Milestone": {
"type": 7,
"index": 9912319,
"timestamp": 1685664559,
"protocolVersion": 2,
"previousMilestoneId": "0x04f35ecb367d18ecd75d260fb12c38afa984212d587ac803c33f7f06b3d5d8a2",
"parents": [
"0x56def2f78d6073371a5f07945a4e6cfebb298bb1918413846bbb980190799195",
"0xba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02"
],
"inclusionMerkleRoot": "0x15405e649707955e66c4e9e9ba62fb5436b7e2acf3ff6e3de1ec7421273fac9d",
"appliedMerkleRoot": "0x2e22bd402ccccfd221546829a848d83b315b629f4565aa09b597b89ca76760f8",
"metadata": "0x58205a000000000056def2f78d6073371a5f07945a4e6cfebb298bb1918413846bbb980190799195",
"signatures": []
},
"SignaturesByIssuer": {}
}
}
bad:
{
"stateHash": "0x5437222422ac24d5d0af94daac8a9b92ce5081f36a2c07e39c998778c533e980",
"state": {
"MilestoneHeight": 5906520,
"MilestoneIndex": 9912319,
"LastMilestoneID": "04f35ecb367d18ecd75d260fb12c38afa984212d587ac803c33f7f06b3d5d8a2",
"LastMilestoneBlockID": "56def2f78d6073371a5f07945a4e6cfebb298bb1918413846bbb980190799195",
"ParentByIssuer": {
"19596d570e9535f11cc34091db98ec44a526e38657f826b5d6b6152e533b8c06": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
"2a6977f6e547518d572d7ba63f6b50b5bbaf3de8023dc542751723f391297b8b": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
"3df80964cc83921e08c1fa0a4f5fc05810a634da45461b2b315fcbfd62f7cab7": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
"64bc7cd3de254cc3a2a6e0aaee5b841d0a671b22b487788f594fbf1ef329935c": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
"8e222ae7e2adcfb87a2984a19aad52b1979ed1472c3cb17239a73ef1d344c35a": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
"a3be03ff2390210354a9af6beb418a957766b050f5a7c62662803cc267a944f6": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
"afeac2c52a0ad3da537ba87d0172720fd55fa4fdb3c44b0fd20bc812d18a8856": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
"d85e5b1590d898d1e0cdebb2e3b5337c8b76270142663d78811683ba47c17c98": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
"f9d9656a60049083eef61487632187b351294c1fa23d118060d813db6d03e8b6": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02"
},
"IssuerCountByParent": {
"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02": 9
},
"ProofIssuersByBlockID": {
"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02": {
"19596d570e9535f11cc34091db98ec44a526e38657f826b5d6b6152e533b8c06": {},
"2a6977f6e547518d572d7ba63f6b50b5bbaf3de8023dc542751723f391297b8b": {},
"3df80964cc83921e08c1fa0a4f5fc05810a634da45461b2b315fcbfd62f7cab7": {},
"64bc7cd3de254cc3a2a6e0aaee5b841d0a671b22b487788f594fbf1ef329935c": {},
"8e222ae7e2adcfb87a2984a19aad52b1979ed1472c3cb17239a73ef1d344c35a": {},
"a3be03ff2390210354a9af6beb418a957766b050f5a7c62662803cc267a944f6": {},
"afeac2c52a0ad3da537ba87d0172720fd55fa4fdb3c44b0fd20bc812d18a8856": {},
"d85e5b1590d898d1e0cdebb2e3b5337c8b76270142663d78811683ba47c17c98": {},
"f9d9656a60049083eef61487632187b351294c1fa23d118060d813db6d03e8b6": {}
}
},
"Milestone": {
"type": 7,
"index": 9912319,
"timestamp": 1685664559,
"protocolVersion": 2,
"previousMilestoneId": "0x04f35ecb367d18ecd75d260fb12c38afa984212d587ac803c33f7f06b3d5d8a2",
"parents": [
"0x56def2f78d6073371a5f07945a4e6cfebb298bb1918413846bbb980190799195",
"0xba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02"
],
"inclusionMerkleRoot": "0x8be25cc05159ffedab154e251971e1c28a83821e6515a13b8944fa7c795734fa",
"appliedMerkleRoot": "0x2e22bd402ccccfd221546829a848d83b315b629f4565aa09b597b89ca76760f8",
"metadata": "0x58205a000000000056def2f78d6073371a5f07945a4e6cfebb298bb1918413846bbb980190799195",
"signatures": []
},
"SignaturesByIssuer": {}
}
}
Mmhmm comparing the good and the bad gives:
diff good_jq.txt bad_jq.txt
2c2
< "stateHash": "0xe0269e863385cee700a77fc0af5cf975f88c79bd8a21234f2fc0f28bd3c066c1",
---
> "stateHash": "0x5437222422ac24d5d0af94daac8a9b92ce5081f36a2c07e39c998778c533e980",
45c45
< "inclusionMerkleRoot": "0x15405e649707955e66c4e9e9ba62fb5436b7e2acf3ff6e3de1ec7421273fac9d",
---
> "inclusionMerkleRoot": "0x8be25cc05159ffedab154e251971e1c28a83821e6515a13b8944fa7c795734fa",
so the reason for the changed app state is connected to a different inclusionMerkleRoot :thinking:
I added additional logging to whiteflag computation for further analysis:
https://github.com/iotaledger/inx-tendercoo/commit/f7a3e4086557b9d3e55d37f5da4d0ff32bc25237
so the main question here is, how can the inclMerkleRoot be wrong at this place? (parameters of the call are correct)
https://github.com/iotaledger/inx-tendercoo/blob/develop/pkg/decoo/app.go#L352
Hornet (white_flag):
// compute past cone merkle tree root hash
confirmedMerkleHash := merklehasher.NewHasher(crypto.BLAKE2b_256).HashBlockIDs(wfConf.ReferencedBlocks.BlockIDs())
copy(wfConf.InclusionMerkleRoot[:], confirmedMerkleHash)
// compute inclusion merkle tree root hash
appliedMerkleHash := merklehasher.NewHasher(crypto.BLAKE2b_256).HashBlockIDs(wfConf.ReferencedBlocks.IncludedTransactionBlockIDs())
copy(wfConf.AppliedMerkleRoot[:], appliedMerkleHash)
applied was okay but inclusion was different :thinking:
the question would be: Can a solid cone of a parent be different (e.g. has more or less blocks) on different nodes if a node lags behind, leading to different inclusionMerkleRoot :thinking:
New information:
coo-1 (broken):
2023-06-20T09:55:11Z DEBUG Coordinator WhiteFlag-Start {"cmi": 10225138, "index": 10225139, "timestamp": 1687254908, "parents": ["d93a622f4b31e489c337c9fa656527901a39396861a98ae968e7cf54e5d52cb3","de626c6a82eb4e979e82da69fcb73bd8d300924136ee7d8fbdf2904d625689a3"], "previousMilestoneId": "492cdd603c017115b092a7bee2c6d322733022ae4b6587cdd110736902f2e1f2"}
2023-06-20T09:55:11Z DEBUG Coordinator WhiteFlag-Compute {"index": 10225139, "timestamp": 1687254908, "parents": ["d93a622f4b31e489c337c9fa656527901a39396861a98ae968e7cf54e5d52cb3","de626c6a82eb4e979e82da69fcb73bd8d300924136ee7d8fbdf2904d625689a3"], "previousMilestoneId": "492cdd603c017115b092a7bee2c6d322733022ae4b6587cdd110736902f2e1f2"}
2023-06-20T09:55:11Z DEBUG Coordinator WhiteFlag-Return {"inclusionMerkleRoot": "0x0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8", "appliedMerkleRoot": "0x0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8", "err": null}
coo-0 (good)
2023-06-20T09:55:10Z DEBUG Coordinator WhiteFlag-Start {"cmi": 10225138, "index": 10225139, "timestamp": 1687254908, "parents": ["d93a622f4b31e489c337c9fa656527901a39396861a98ae968e7cf54e5d52cb3","de626c6a82eb4e979e82da69fcb73bd8d300924136ee7d8fbdf2904d625689a3"], "previousMilestoneId": "492cdd603c017115b092a7bee2c6d322733022ae4b6587cdd110736902f2e1f2"}
2023-06-20T09:55:10Z DEBUG Coordinator WhiteFlag-Compute {"index": 10225139, "timestamp": 1687254908, "parents": ["d93a622f4b31e489c337c9fa656527901a39396861a98ae968e7cf54e5d52cb3","de626c6a82eb4e979e82da69fcb73bd8d300924136ee7d8fbdf2904d625689a3"], "previousMilestoneId": "492cdd603c017115b092a7bee2c6d322733022ae4b6587cdd110736902f2e1f2"}
2023-06-20T09:55:10Z DEBUG Coordinator WhiteFlag-Return {"inclusionMerkleRoot": "0xd632fa4d5d275a1aa38e60283bdb3b2f881cf304e6aa6ea2f0ff4d2fd3073829", "appliedMerkleRoot": "0x0f5fb22ca84363f14d80b3ff85e45222594dfd17a1266a70be7022e94e08550e", "err": null}
Hornet logs for this section: coo-1 (bad):
2023-06-20T09:55:11Z INFO Tangle Valid milestone detected! Index: 10225139
2023-06-20T09:55:11Z INFO Tangle Run solidity check for Milestone (10225139) ...
2023-06-20T09:55:11Z INFO Tangle Solidifier finished: blocks: 0, collect: 0s, solidity 0s, propagation: 0s, total: 0s
2023-06-20T09:55:11Z INFO Glogger WhiteFlag Included Hash: 0xd632fa4d5d275a1aa38e60283bdb3b2f881cf304e6aa6ea2f0ff4d2fd3073829, Blocks: [0xd93a622f4b31e489c337c9fa656527901a39396861a98ae968e7cf54e5d52cb3 0x1f801a6c330e658996a9c2e812495ddedd980e1279dea8b66173af0e0b44
96b8 0x81778629b8b793117e4f5e536724dc09f4f2342f956f5add16139bd67a43a53b 0x8c9eed94fb47cc02299b48f9bcaae178a6a8cffe3876b2224cfcd71e850711b8 0x4790efdb1a2199f233ddae9a75c71bea6bfe7803766552e3e95056c87636cdcd 0x79512b1179ce915049d17d8eb0477615113ef67d03a1eb8e9d74f7d43dcfb5a
b 0xbb2e615cf8626aeb2cdc058bc88b30b1ad52ae13b263c50770eccf094f13d60d 0xd100da96a75a0b1a52280559d80949b96751db437d6b334ac9c7cb77df7b2da8 0x4ac1ad0bad6577a2e70a35a077a3526545031acd89ee6d3e6e408c78716acd36 0x4566d98ab6a7a8f4a8c5d367223ffa9462b46e97647c0ec8964386d3aa680530 0
x16f1e6f6c8cb83d3772199583f76edffe796ab95c0aba89c5c5efc1d3f287f67 0x5f9550df4661861fdc485a0e4303b1625af43df0891bfbd0006ba63956f6b622 0xd2f74cc2ddebe85991fa3573ede6e52b71999a806fe8fb88f255730deec57f1d 0xde626c6a82eb4e979e82da69fcb73bd8d300924136ee7d8fbdf2904d625689a3]
2023-06-20T09:55:11Z INFO Glogger WhiteFlag Applied Hash: 0x0f5fb22ca84363f14d80b3ff85e45222594dfd17a1266a70be7022e94e08550e, Blocks: [0x5f9550df4661861fdc485a0e4303b1625af43df0891bfbd0006ba63956f6b622]
2023-06-20T09:55:11Z INFO Tangle Solidifier finished: blocks: 0, collect: 0s, solidity 0s, propagation: 4ms, total: 4ms
2023-06-20T09:55:11Z INFO Glogger WhiteFlag Included Hash: 0x0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8, Blocks: []
2023-06-20T09:55:11Z INFO Glogger WhiteFlag Applied Hash: 0x0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8, Blocks: []
2023-06-20T09:55:11Z INFO Tangle Milestone confirmed (10225139): txsReferenced: 14, txsValue: 1, txsZeroValue: 13, txsConflicting: 0, collect: 1ms, total: 7ms
2023-06-20T09:55:11Z INFO Tangle New confirmed milestone: 10225139, 0.67 BPS, 1.56 RBPS, 233.33% ref.rate
coo-0 (good):
2023-06-20T09:55:11Z INFO Glogger WhiteFlag Included Hash: 0xd632fa4d5d275a1aa38e60283bdb3b2f881cf304e6aa6ea2f0ff4d2fd3073829, Blocks: [0xd93a622f4b31e489c337c9fa656527901a39396861a98ae968e7cf54e5d52cb3 0x1f801a6c330e658996a9c2e812495ddedd980e1279dea8b66173af0e0b4496b8 0x81778629b8b793117e4f5e536724dc09f4f2342f956f5add16139bd67a43a53b 0x8c9eed94fb47cc02299b48f9bcaae178a6a8cffe3876b2224cfcd71e850711b8 0x4790efdb1a2199f233ddae9a75c71bea6bfe7803766552e3e95056c87636cdcd 0x79512b1179ce915049d17d8eb0477615113ef67d03a1eb8e9d74f7d43dcfb5ab 0xbb2e615cf8626aeb2cdc058bc88b30b1ad52ae13b263c50770eccf094f13d60d 0xd100da96a75a0b1a52280559d80949b96751db437d6b334ac9c7cb77df7b2da8 0x4ac1ad0bad6577a2e70a35a077a3526545031acd89ee6d3e6e408c78716acd36 0x4566d98ab6a7a8f4a8c5d367223ffa9462b46e97647c0ec8964386d3aa680530 0x16f1e6f6c8cb83d3772199583f76edffe796ab95c0aba89c5c5efc1d3f287f67 0x5f9550df4661861fdc485a0e4303b1625af43df0891bfbd0006ba63956f6b622 0xd2f74cc2ddebe85991fa3573ede6e52b71999a806fe8fb88f255730deec57f1d 0xde626c6a82eb4e979e82da69fcb73bd8d300924136ee7d8fbdf2904d625689a3]
2023-06-20T09:55:11Z INFO Glogger WhiteFlag Applied Hash: 0x0f5fb22ca84363f14d80b3ff85e45222594dfd17a1266a70be7022e94e08550e, Blocks: [0x5f9550df4661861fdc485a0e4303b1625af43df0891bfbd0006ba63956f6b622]
Max found a possible explanation (missing mutex on whiteflag calculation) and provided a fix for the issue:
https://github.com/iotaledger/hornet/commit/7fad952203ed1678054607f715224c317b6f6917
With the fix, problem seems to be solve.
After this error:
The tendercoo-part of inx-tendercoo has crashed completly.
Only the coordinator tries to do something - but unsuccessfully:
Probably would be better if the entire inx-tendercoo would crash not only the tendermint thread.