iotaledger / inx-tendercoo

INX-Tendercoo enables a committee of validators to operate as a distributed Coordinator using Tendermint Core BFT consensus.
Apache License 2.0
1 stars 0 forks source link

consensus failure: +2/3 committed an invalid block: wrong Block.Header.AppHash #64

Closed shufps closed 1 year ago

shufps commented 1 year ago

After this error:

2023-04-24T00:00:31Z ERROR Tendermint CONSENSUS FAILURE!!! {"module": "consensus", "err": "+2/3 committed an invalid block: wrong Block.Header.AppHash. Expected ADEE1F2A17B88FA7874C6252809BA8F6E770FB88AF69D271F43B84B43BE52770, got 367391666238528CB4A7D7CB7B04266160E901D9340A616A9D8CD36E8C80F8C0", "stack": "goroutine 184 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x65
github.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2()
        /go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:724 +0x4c
panic({0x108cbc0, 0xc002a341c0})
        /usr/local/go/src/runtime/panic.go:884 +0x212
github.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc000d89180, 0x633c5)
        /go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:1594 +0x1105
github.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc000d89180, 0x633c5)
        /go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:1563 +0x2ff
github.com/tendermint/tendermint/consensus.(*State).handleCompleteProposal(0xc000d89180, 0xa31?)
        /go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:1942 +0x399
github.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc000d89180, {{0x14f4b00?, 0xc003aeeee8?}, {0xc0026beae0?, 0x0?}})
        /go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:834 +0x290
github.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc000d89180, 0x0)
        /go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:760 +0x3f9
created by github.com/tendermint/tendermint/consensus.(*State).OnStart
        /go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:379 +0x12d
"}

The tendercoo-part of inx-tendercoo has crashed completly.

Only the coordinator tries to do something - but unsuccessfully:

2023-04-28T13:22:34Z    WARN    Coordinator     defaulting to last milestone: tip selection failed: no tips available
2023-04-28T13:22:38Z    INFO    Coordinator     New confirmed milestone: 3651606
2023-04-28T13:22:40Z    INFO    Coordinator     Proposing parent for milestone 3651607
2023-04-28T13:22:43Z    INFO    Coordinator     New confirmed milestone: 3651607
2023-04-28T13:22:45Z    INFO    Coordinator     Proposing parent for milestone 3651608
2023-04-28T13:22:52Z    INFO    Coordinator     New confirmed milestone: 3651608
2023-04-28T13:22:54Z    INFO    Coordinator     Proposing parent for milestone 3651609
2023-04-28T13:22:57Z    INFO    Coordinator     New confirmed milestone: 3651609
2023-04-28T13:22:59Z    INFO    Coordinator     Proposing parent for milestone 3651610

Probably would be better if the entire inx-tendercoo would crash not only the tendermint thread.

shufps commented 1 year ago

A little bit more information:

2023-05-12T11:55:46Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":5097492,"Parent":"379a4cf4138f7671c58a2f45a0f746267d31df927630538b513962a760bb5e56"}, "issuer": "s7TJIJCXILpffDDd3A+Rab+CQ7UptgH8R3a4ywqMolM="}
2023-05-12T11:55:46Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":5097492,"Parent":"379a4cf4138f7671c58a2f45a0f746267d31df927630538b513962a760bb5e56"}, "issuer": "Yflf7TC26b8LLQOTj1bTV4n/fw6hItAcXBt+hpUl4hg="}
2023-05-12T11:55:46Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":5097492,"Parent":"379a4cf4138f7671c58a2f45a0f746267d31df927630538b513962a760bb5e56"}, "issuer": "7el2DH8qqkYYpYoTV3Bc3Bh0lirTaTCVQyMDlLt3VIs="}
2023-05-12T11:55:46Z    DEBUG   Coordinator     create milestone        {"state": {"MilestoneHeight":4472423,"MilestoneIndex":5097492,"LastMilestoneID":"f7046e2abae25f8a2751cf3513e3921c05ef35dcba1375911961b0ab95f054ba","LastMilestoneBlockID":"12b4cd7e4fbbc7f4fedb59c9612d58091bb6aacf64c5d15da24e557beaa024a9"}, "parents": ["12b4cd7e4fbbc7f4fedb59c9612d58091bb6aacf64c5d15da24e557beaa024a9","379a4cf4138f7671c58a2f45a0f746267d31df927630538b513962a760bb5e56"]}
2023-05-12T11:55:46Z    INFO    Tendermint      executed block  {"module": "state", "height": 4472426, "num_valid_txs": 19, "num_invalid_txs": 0}
2023-05-12T11:55:46Z    DEBUG   Coordinator     broadcast tx    {"partial": {"Signature":"BBmrarQ53/pPfI0O/wm720tlnP3wiroQRKi8Fa25JBHQALG40atFf/jytkzy9b5iH7KKIeS7ejwBQeClM7rUAw=="}}
2023-05-12T11:55:46Z    INFO    Tendermint      committed state {"module": "state", "height": 4472426, "num_txs": 19, "app_hash": "EDA0E564BCCEBC8675249D19EE00F512BCA1F806568E15E589B37593FE2935DF"}
2023-05-12T11:55:46Z    INFO    Tendermint      indexed block exents    {"module": "txindex", "height": 4472426}
2023-05-12T11:55:46Z    INFO    Tendermint      Timed out       {"module": "consensus", "dur": -1.393214164, "height": 4472427, "round": 0, "step": "RoundStepNewHeight"}
2023-05-12T11:55:46Z    INFO    Coordinator     New confirmed milestone: 5097492
2023-05-12T11:55:46Z    INFO    Tendermint      commit is for a block we do not know about; set ProposalBlock=nil       {"module": "consensus", "height": 4472427, "commit_round": 0, "proposal": "", "commit": "48D4CB2AC39B52EB52054352B208AB699680DC3A986862ED13D9313DBE02C0A0"}
2023-05-12T11:55:46Z    INFO    Tendermint      received complete proposal block        {"module": "consensus", "height": 4472427, "hash": "48D4CB2AC39B52EB52054352B208AB699680DC3A986862ED13D9313DBE02C0A0"}
2023-05-12T11:55:46Z    ERROR   Tendermint      CONSENSUS FAILURE!!!    {"module": "consensus", "err": "+2/3 committed an invalid block: wrong Block.Header.AppHash.  Expected EDA0E564BCCEBC8675249D19EE00F512BCA1F806568E15E589B37593FE2935DF, got 51DB8330B9D79F3DBD61B181307126A0DB7CF54A652D2AB9CFAC205FD6B62A6F", "stack": "goroutine 216 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2()\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:724 +0x4c\npanic({0x108cbc0, 0xc00264d880})\n\t/usr/local/go/src/runtime/panic.go:884 +0x212\ngithub.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc00030b880, 0x443e6b)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:1594 +0x1105\ngithub.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc00030b880, 0x443e6b)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:1563 +0x2ff\ngithub.com/tendermint/tendermint/consensus.(*State).handleCompleteProposal(0xc00030b880, 0xc9c?)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:1942 +0x399\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc00030b880, {{0x14f4b00?, 0xc003c6a1b0?}, {0xc005edd380?, 0x0?}})\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:834 +0x290\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc00030b880, 0x0)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:760 +0x3f9\ncreated by github.com/tendermint/tendermint/consensus.(*State).OnStart\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:379 +0x12d\n"}
2023-05-12T11:55:46Z    INFO    Tendermint      service stop    {"module": "consensus", "wal": "tendermint/data/cs.wal/wal", "msg": "Stopping baseWAL service", "impl": "baseWAL"}
2023-05-12T11:55:46Z    INFO    Tendermint      service stop    {"module": "consensus", "wal": "tendermint/data/cs.wal/wal", "msg": "Stopping Group service", "impl": "Group"}
2023-05-12T11:55:48Z    INFO    Coordinator     Proposing parent for milestone 5097493
2023-05-12T11:55:49Z    INFO    Tendermint      Timed out       {"module": "consensus", "dur": 3, "height": 4472427, "round": 0, "step": "RoundStepPropose"}
2023-05-12T11:55:56Z    INFO    Coordinator     New confirmed milestone: 5097493
2023-05-12T11:55:59Z    INFO    Coordinator     Proposing parent for milestone 5097494
2023-05-12T11:56:01Z    INFO    Coordinator     New confirmed milestone: 5097494
2023-05-12T11:56:04Z    INFO    Coordinator     Proposing parent for milestone 5097495
2023-05-12T11:56:11Z    INFO    Coordinator     New confirmed milestone: 5097495
2023-05-12T11:56:11Z    INFO    Coordinator     Proposing parent for milestone 5097496
2023-05-12T11:56:11Z    WARN    Coordinator     defaulting to last milestone: tip selection failed: no tips available

Interesting is the: commit is for a block we do not know about

And this also looks interesting: "consensus", "dur": -1.393214164

shufps commented 1 year ago

another notes:

committed state {"module": "state", "height": 4472426, "num_txs": 19, "app_hash": "EDA0E564BCCEBC8675249D19EE00F512BCA1F806568E15E589B37593FE2935DF"}

already contains the wrong app-state and here:

CONSENSUS FAILURE!!!    {"module": "consensus", "err": "+2/3 committed an invalid block: wrong Block.Header.AppHash.  Expected EDA0E564BCCEBC8675249D19EE00F512BCA1F806568E15E589B37593FE2935DF, got 51DB8330B9D79F3DBD61B181307126A0DB7CF54A652D2AB9CFAC205FD6B62A6F", ...

the "Expected" is wrong and the "got" is correct.

shufps commented 1 year ago

mmhmm, at this place, the state already should be wrong:

https://github.com/iotaledger/inx-tendercoo/blob/develop/pkg/decoo/app.go#L235

shufps commented 1 year ago

According to the hornet logs, hornet had connections issues / took longer for validating a new milestone and the milestone came late:

Coordinator     New confirmed milestone: 5097492

It already was confirmed when inx-tendercoo was not yet finished.

edit: Provoking this error by adding a random delay (1-10s) into the confirmedMilestoneListener running on a private tangle didn't work.

shufps commented 1 year ago

Added logging of state: https://github.com/iotaledger/inx-tendercoo/commit/ff9b3ee17f1adb64c77f3411bea316f6371323f6

shufps commented 1 year ago

This happens after a restart from the previously mentioned condition:

2023-05-29T16:54:35Z    INFO    INX     Connecting to node and reading node configuration ...
2023-05-29T16:54:35Z    INFO    INX     Reading node status ...
2023-05-29T16:54:35Z    INFO    Coordinator     Providing Coordinator ...
2023-05-29T16:54:35Z    INFO    Coordinator     Providing Coordinator ... done
2023-05-29T16:54:35Z    INFO    App     Loading components ...
2023-05-29T16:54:35Z    INFO    App     Loading components: INX ... done
2023-05-29T16:54:35Z    INFO    App     Loading components: Coordinator ... done
2023-05-29T16:54:35Z    INFO    App     Loading components: Shutdown ... done
2023-05-29T16:54:35Z    INFO    App     Loading components: Profiling ... done
2023-05-29T16:54:35Z    INFO    App     Executing components ...
2023-05-29T16:54:35Z    INFO    App     Starting component: INX ...
2023-05-29T16:54:35Z    INFO    App     Starting component: Coordinator ...
2023-05-29T16:54:35Z    INFO    App     Starting component: Shutdown ...
2023-05-29T16:54:35Z    INFO    App     Starting component: Profiling ...
2023-05-29T16:54:35Z    INFO    App     Starting background workers ...
2023-05-29T16:54:35Z    INFO    Coordinator     Starting Decentralized Coordinator ...
2023-05-29T16:54:35Z    INFO    INX     Starting NodeBridge ...
2023-05-29T16:54:35Z    INFO    Coordinator     Starting TangleListener ... done
2023-05-29T16:54:35Z    INFO    Profiling       You can now access the profiling server using: http://0.0.0.0:6060/debug/pprof/
2023-05-29T16:54:35Z    INFO    Coordinator     Found private validator {"keyFile": "tendermint/config/priv_validator_key.json", "stateFile": "tendermint/data/priv_validator_state.json"}
2023-05-29T16:54:35Z    INFO    Coordinator     Found node key  {"path": "tendermint/config/node_key.json"}
2023-05-29T16:54:35Z    INFO    Coordinator     Found genesis file      {"path": "tendermint/config/genesis.json"}
2023-05-29T16:54:35Z    INFO    Coordinator     Node appears to be connected
2023-05-29T16:54:35Z    INFO    Coordinator     Node appears to be synced; latest=9855998 confirmed=9855998
2023-05-29T16:54:40Z    INFO    Coordinator     Coordinator resumed     {"state": {"MilestoneHeight":5642115,"MilestoneIndex":9846344,"LastMilestoneID":"b3fa2eb4f8d3d9889a57205f4e15b86278b83882ba28839d343ba7a735ad335e","LastMilestoneBlockID":"2812358e31d74c687ce4f9677448039494257e4d339a8bf8128deb6db6551b5b"}}
2023-05-29T16:54:40Z    INFO    Tendermint      service start   {"module": "proxy", "msg": "Starting multiAppConn service", "impl": "multiAppConn"}
2023-05-29T16:54:40Z    INFO    Tendermint      service start   {"module": "proxy", "module": "abci-client", "connection": "query", "msg": "Starting localClient service", "impl": "localClient"}
2023-05-29T16:54:40Z    INFO    Tendermint      service start   {"module": "proxy", "module": "abci-client", "connection": "snapshot", "msg": "Starting localClient service", "impl": "localClient"}
2023-05-29T16:54:40Z    INFO    Tendermint      service start   {"module": "proxy", "module": "abci-client", "connection": "mempool", "msg": "Starting localClient service", "impl": "localClient"}
2023-05-29T16:54:40Z    INFO    Tendermint      service start   {"module": "proxy", "module": "abci-client", "connection": "consensus", "msg": "Starting localClient service", "impl": "localClient"}
2023-05-29T16:54:40Z    INFO    Tendermint      service start   {"module": "events", "msg": "Starting EventBus service", "impl": "EventBus"}
2023-05-29T16:54:40Z    INFO    Tendermint      service start   {"module": "events", "module": "pubsub", "msg": "Starting PubSub service", "impl": "PubSub"}
2023-05-29T16:54:40Z    INFO    Tendermint      service start   {"module": "txindex", "msg": "Starting IndexerService service", "impl": "IndexerService"}
2023-05-29T16:54:40Z    DEBUG   Coordinator     ABCI Info       {"req": {"version":"0.34.24","block_version":11,"p2p_version":8}}
2023-05-29T16:54:40Z    INFO    Tendermint      ABCI Handshake App Info {"module": "consensus", "height": 5642115, "hash": "XPXZQSZyGaTwSDLYRmjIB4cRCOx8IVnbDy31DFq+MpM=", "software-version": "", "protocol-version": 1}
2023-05-29T16:54:40Z    INFO    Tendermint      ABCI Replay Blocks      {"module": "consensus", "appHeight": 5642115, "storeHeight": 5642118, "stateHeight": 5642118}
2023-05-29T16:54:40Z    INFO    Tendermint      Applying block  {"module": "consensus", "height": 5642116}
2023-05-29T16:54:40Z    INFO    Tendermint      executed block  {"module": "consensus", "height": 5642116, "num_valid_txs": 0, "num_invalid_txs": 0}
2023-05-29T16:54:40Z    DEBUG   Coordinator     Commit-aftercommit      {"stateHash": "0x5cf5d941267219a4f04832d84668c807871108ec7c2159db0f2df50c5abe3293", "state": {"MilestoneHeight":5642115,"MilestoneIndex":9846344,"LastMilestoneID":"b3fa2eb4f8d3d9889a57205f4e15b86278b83882ba28839d343ba7a735ad335e","LastMilestoneBlockID":"2812358e31d74c687ce4f9677448039494257e4d339a8bf8128deb6db6551b5b","ParentByIssuer":{},"IssuerCountByParent":{},"ProofIssuersByBlockID":{},"Milestone":null,"SignaturesByIssuer":{}}}
2023-05-29T16:54:40Z    INFO    Tendermint      Applying block  {"module": "consensus", "height": 5642117}
2023-05-29T16:54:40Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":9846344,"BlockID":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "PfgJZMyDkh4IwfoKT1/AWBCmNNpFRhsrMV/L/WL3yrc="}
2023-05-29T16:54:40Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":9846344,"BlockID":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "2ZIoGaOelN3zkH9LnI35PznwJiRPy2CSBbmoeQIlmfI="}
2023-05-29T16:54:40Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":9846344,"BlockID":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "jiIq5+Ktz7h6KYShmq1SsZee0UcsPLFyOac+8dNEw1o="}
2023-05-29T16:54:40Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":9846344,"BlockID":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "ZLx8094lTMOipuCq7luEHQpnGyK0h3iPWU+/HvMpk1w="}
2023-05-29T16:54:40Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":9846344,"BlockID":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "+dllamAEkIPu9hSHYyGHs1EpTB+iPRGAYNgT220D6LY="}
2023-05-29T16:54:40Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":9846344,"BlockID":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "r+rCxSoK09pTe6h9AXJyD9VfpP2zxEsP0gvIEtGKiFY="}
2023-05-29T16:54:40Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":9846344,"BlockID":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "o74D/yOQIQNUqa9r60GKlXdmsFD1p8YmYoA8wmepRPY="}
2023-05-29T16:54:40Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":9846344,"BlockID":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "2F5bFZDYmNHgzeuy47UzfIt2JwFCZj14gRaDukfBfJg="}
2023-05-29T16:54:40Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":9846344,"BlockID":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "GVltVw6VNfEcw0CR25jsRKUm44ZX+Ca11rYVLlM7jAY="}
2023-05-29T16:54:40Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":9846344,"BlockID":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "Kml39uVHUY1XLXumP2tQtbuvPegCPcVCdRcj85Epe4s="}
2023-05-29T16:54:40Z    DEBUG   Coordinator     awaiting parent {"blockID": "3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}
2023-05-29T16:54:40Z    INFO    Tendermint      executed block  {"module": "consensus", "height": 5642117, "num_valid_txs": 10, "num_invalid_txs": 0}
2023-05-29T16:54:40Z    DEBUG   Coordinator     Commit-aftercommit      {"stateHash": "0x175f6ff18437540c7628ef198cfa96e0b67e9509a63ab8f8490aa60c4a0209d7", "state": {"MilestoneHeight":5642115,"MilestoneIndex":9846344,"LastMilestoneID":"b3fa2eb4f8d3d9889a57205f4e15b86278b83882ba28839d343ba7a735ad335e","LastMilestoneBlockID":"2812358e31d74c687ce4f9677448039494257e4d339a8bf8128deb6db6551b5b","ParentByIssuer":{"19596d570e9535f11cc34091db98ec44a526e38657f826b5d6b6152e533b8c06":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","2a6977f6e547518d572d7ba63f6b50b5bbaf3de8023dc542751723f391297b8b":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","3df80964cc83921e08c1fa0a4f5fc05810a634da45461b2b315fcbfd62f7cab7":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","64bc7cd3de254cc3a2a6e0aaee5b841d0a671b22b487788f594fbf1ef329935c":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","8e222ae7e2adcfb87a2984a19aad52b1979ed1472c3cb17239a73ef1d344c35a":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","a3be03ff2390210354a9af6beb418a957766b050f5a7c62662803cc267a944f6":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","afeac2c52a0ad3da537ba87d0172720fd55fa4fdb3c44b0fd20bc812d18a8856":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","d85e5b1590d898d1e0cdebb2e3b5337c8b76270142663d78811683ba47c17c98":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","d9922819a39e94ddf3907f4b9c8df93f39f026244fcb609205b9a879022599f2":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","f9d9656a60049083eef61487632187b351294c1fa23d118060d813db6d03e8b6":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"},"IssuerCountByParent":{"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5":10},"ProofIssuersByBlockID":{},"Milestone":null,"SignaturesByIssuer":{}}}
2023-05-29T16:54:40Z    INFO    Tendermint      Applying block  {"module": "consensus", "height": 5642118}
2023-05-29T16:54:40Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":9846344,"Parent":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "r+rCxSoK09pTe6h9AXJyD9VfpP2zxEsP0gvIEtGKiFY="}
panic: state.AppHash does not match AppHash after replay. Got
2023-05-29T16:54:40Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":9846344,"Parent":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "Kml39uVHUY1XLXumP2tQtbuvPegCPcVCdRcj85Epe4s="}
2023-05-29T16:54:40Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":9846344,"Parent":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "o74D/yOQIQNUqa9r60GKlXdmsFD1p8YmYoA8wmepRPY="}
2C3D69BB07BA4DDDB514C124BD8CF395E8AFCD04CE3A99422C891084DCA7B22A, expected D1F7546E034D1BCB5D161CF11F12C95014C0E4EB41161A0C27ACFF3AEE17744F.

2023-05-29T16:54:40Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":9846344,"Parent":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "2ZIoGaOelN3zkH9LnI35PznwJiRPy2CSBbmoeQIlmfI="}
State: {{{11 1} 0.34.24} alphanet-10 1 5642118 9CCEB8CB938D9232080DB913FE946E419F8B13CA320DE25313BFB1C65100DD06:1:0043345B4213 2023-05-29 00:00:06.274769542 +0000 UTC ValidatorSet{
2023-05-29T16:54:40Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":9846344,"Parent":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "ZLx8094lTMOipuCq7luEHQpnGyK0h3iPWU+/HvMpk1w="}
2023-05-29T16:54:40Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":9846344,"Parent":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "PfgJZMyDkh4IwfoKT1/AWBCmNNpFRhsrMV/L/WL3yrc="}
  Proposer: Validator{9234C27379290332CA13772144DB4E12FCA1FFAD PubKeyEd25519{AFEAC2C52A0AD3DA537BA87D0172720FD55FA4FDB3C44B0FD20BC812D18A8856} VP:1 A:0}
2023-05-29T16:54:40Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":9846344,"Parent":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "+dllamAEkIPu9hSHYyGHs1EpTB+iPRGAYNgT220D6LY="}
  Validators:
    Validator{008CB3E446C123CD0BE3D1C8BBDFA555CFB600E7 PubKeyEd25519{19596D570E9535F11CC34091DB98EC44A526E38657F826B5D6B6152E533B8C06} VP:1 A:0}
2023-05-29T16:54:40Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":9846344,"Parent":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "GVltVw6VNfEcw0CR25jsRKUm44ZX+Ca11rYVLlM7jAY="}
    Validator{0223EB172703E02C2B73D9710F535B8C01480235 PubKeyEd25519{64BC7CD3DE254CC3A2A6E0AAEE5B841D0A671B22B487788F594FBF1EF329935C} VP:1 A:0}
2023-05-29T16:54:40Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":9846344,"Parent":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "jiIq5+Ktz7h6KYShmq1SsZee0UcsPLFyOac+8dNEw1o="}
    Validator{04A5FE0DA11994D8B399860F7A704BF46E1FEBED PubKeyEd25519{A3BE03FF2390210354A9AF6BEB418A957766B050F5A7C62662803CC267A944F6} VP:1 A:0}
2023-05-29T16:54:40Z    DEBUG   Coordinator     DeliverTx       {"tx": {"Index":9846344,"Parent":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"}, "issuer": "2F5bFZDYmNHgzeuy47UzfIt2JwFCZj14gRaDukfBfJg="}
2023-05-29T16:54:40Z    DEBUG   Coordinator     create milestone        {"state": {"MilestoneHeight":5642115,"MilestoneIndex":9846344,"LastMilestoneID":"b3fa2eb4f8d3d9889a57205f4e15b86278b83882ba28839d343ba7a735ad335e","LastMilestoneBlockID":"2812358e31d74c687ce4f9677448039494257e4d339a8bf8128deb6db6551b5b"}, "parents": ["2812358e31d74c687ce4f9677448039494257e4d339a8bf8128deb6db6551b5b","3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"]}
    Validator{0E27BBBE79F7D010439C78273D1B35F1CE4CFAF0 PubKeyEd25519{D9922819A39E94DDF3907F4B9C8DF93F39F026244FCB609205B9A879022599F2} VP:1 A:0}
2023-05-29T16:54:40Z    INFO    Tendermint      executed block  {"module": "consensus", "height": 5642118, "num_valid_txs": 10, "num_invalid_txs": 0}
    Validator{10849BB652FED684976249438116C6FD3552C035 PubKeyEd25519{D85E5B1590D898D1E0CDEBB2E3B5337C8B76270142663D78811683BA47C17C98} VP:1 A:0}
2023-05-29T16:54:40Z    DEBUG   Coordinator     broadcast tx    {"partial": {"Signature":"2RuLdDgCIlZKd+EEYaraC56ZcwyElJSHP2CL6unw7QsBEbD0FRf4o/92YVLGxJ6Sm4/mtF+mSm4yqwlLZL6gBQ=="}}
2023-05-29T16:54:40Z    DEBUG   Coordinator     Commit-aftercommit      {"stateHash": "0x2c3d69bb07ba4dddb514c124bd8cf395e8afcd04ce3a99422c891084dca7b22a", "state": {"MilestoneHeight":5642115,"MilestoneIndex":9846344,"LastMilestoneID":"b3fa2eb4f8d3d9889a57205f4e15b86278b83882ba28839d343ba7a735ad335e","LastMilestoneBlockID":"2812358e31d74c687ce4f9677448039494257e4d339a8bf8128deb6db6551b5b","ParentByIssuer":{"19596d570e9535f11cc34091db98ec44a526e38657f826b5d6b6152e533b8c06":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","2a6977f6e547518d572d7ba63f6b50b5bbaf3de8023dc542751723f391297b8b":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","3df80964cc83921e08c1fa0a4f5fc05810a634da45461b2b315fcbfd62f7cab7":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","64bc7cd3de254cc3a2a6e0aaee5b841d0a671b22b487788f594fbf1ef329935c":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","8e222ae7e2adcfb87a2984a19aad52b1979ed1472c3cb17239a73ef1d344c35a":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","a3be03ff2390210354a9af6beb418a957766b050f5a7c62662803cc267a944f6":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","afeac2c52a0ad3da537ba87d0172720fd55fa4fdb3c44b0fd20bc812d18a8856":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","d85e5b1590d898d1e0cdebb2e3b5337c8b76270142663d78811683ba47c17c98":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","d9922819a39e94ddf3907f4b9c8df93f39f026244fcb609205b9a879022599f2":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5","f9d9656a60049083eef61487632187b351294c1fa23d118060d813db6d03e8b6":"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"},"IssuerCountByParent":{"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5":10},"ProofIssuersByBlockID":{"3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5":{"19596d570e9535f11cc34091db98ec44a526e38657f826b5d6b6152e533b8c06":{},"2a6977f6e547518d572d7ba63f6b50b5bbaf3de8023dc542751723f391297b8b":{},"3df80964cc83921e08c1fa0a4f5fc05810a634da45461b2b315fcbfd62f7cab7":{},"64bc7cd3de254cc3a2a6e0aaee5b841d0a671b22b487788f594fbf1ef329935c":{},"8e222ae7e2adcfb87a2984a19aad52b1979ed1472c3cb17239a73ef1d344c35a":{},"a3be03ff2390210354a9af6beb418a957766b050f5a7c62662803cc267a944f6":{},"afeac2c52a0ad3da537ba87d0172720fd55fa4fdb3c44b0fd20bc812d18a8856":{},"d85e5b1590d898d1e0cdebb2e3b5337c8b76270142663d78811683ba47c17c98":{},"d9922819a39e94ddf3907f4b9c8df93f39f026244fcb609205b9a879022599f2":{},"f9d9656a60049083eef61487632187b351294c1fa23d118060d813db6d03e8b6":{}}},"Milestone":{"type":7,"index":9846344,"timestamp":1685318406,"protocolVersion":2,"previousMilestoneId":"0xb3fa2eb4f8d3d9889a57205f4e15b86278b83882ba28839d343ba7a735ad335e","parents":["0x2812358e31d74c687ce4f9677448039494257e4d339a8bf8128deb6db6551b5b","0x3cb7537893c83d6cdc86f27ecedb50d48776756d8849c68c71b65fe02352b2e5"],"inclusionMerkleRoot":"0x46b68298fe97239abedbd44063e1aefe50e5299be9c813ffaffa45bf762b55e7","appliedMerkleRoot":"0x0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8","metadata":"0x83175600000000002812358e31d74c687ce4f9677448039494257e4d339a8bf8128deb6db6551b5b","signatures":[]},"SignaturesByIssuer":{}}}
    Validator{1093D97B9310C072EED7A4957DFEDE346C76B13D PubKeyEd25519{8E222AE7E2ADCFB87A2984A19AAD52B1979ED1472C3CB17239A73EF1D344C35A} VP:1 A:0}
    Validator{355DBE857FED7715F9C2AC12AE1A0E9E7C316EBD PubKeyEd25519{3DF80964CC83921E08C1FA0A4F5FC05810A634DA45461B2B315FCBFD62F7CAB7} VP:1 A:0}
    Validator{367CA8B17443585A2D013DA2916C10BFEC29D7F7 PubKeyEd25519{2A6977F6E547518D572D7BA63F6B50B5BBAF3DE8023DC542751723F391297B8B} VP:1 A:0}
    Validator{840399906E468F3E87FA8C0865638D6F29D5C2EA PubKeyEd25519{F9D9656A60049083EEF61487632187B351294C1FA23D118060D813DB6D03E8B6} VP:1 A:0}
    Validator{9234C27379290332CA13772144DB4E12FCA1FFAD PubKeyEd25519{AFEAC2C52A0AD3DA537BA87D0172720FD55FA4FDB3C44B0FD20BC812D18A8856} VP:1 A:0}
} ValidatorSet{
  Proposer: Validator{840399906E468F3E87FA8C0865638D6F29D5C2EA PubKeyEd25519{F9D9656A60049083EEF61487632187B351294C1FA23D118060D813DB6D03E8B6} VP:1 A:-1}
  Validators:
    Validator{008CB3E446C123CD0BE3D1C8BBDFA555CFB600E7 PubKeyEd25519{19596D570E9535F11CC34091DB98EC44A526E38657F826B5D6B6152E533B8C06} VP:1 A:-1}
    Validator{0223EB172703E02C2B73D9710F535B8C01480235 PubKeyEd25519{64BC7CD3DE254CC3A2A6E0AAEE5B841D0A671B22B487788F594FBF1EF329935C} VP:1 A:-1}
    Validator{04A5FE0DA11994D8B399860F7A704BF46E1FEBED PubKeyEd25519{A3BE03FF2390210354A9AF6BEB418A957766B050F5A7C62662803CC267A944F6} VP:1 A:-1}
    Validator{0E27BBBE79F7D010439C78273D1B35F1CE4CFAF0 PubKeyEd25519{D9922819A39E94DDF3907F4B9C8DF93F39F026244FCB609205B9A879022599F2} VP:1 A:-1}
    Validator{10849BB652FED684976249438116C6FD3552C035 PubKeyEd25519{D85E5B1590D898D1E0CDEBB2E3B5337C8B76270142663D78811683BA47C17C98} VP:1 A:-1}
    Validator{1093D97B9310C072EED7A4957DFEDE346C76B13D PubKeyEd25519{8E222AE7E2ADCFB87A2984A19AAD52B1979ED1472C3CB17239A73EF1D344C35A} VP:1 A:-1}
    Validator{355DBE857FED7715F9C2AC12AE1A0E9E7C316EBD PubKeyEd25519{3DF80964CC83921E08C1FA0A4F5FC05810A634DA45461B2B315FCBFD62F7CAB7} VP:1 A:-1}
    Validator{367CA8B17443585A2D013DA2916C10BFEC29D7F7 PubKeyEd25519{2A6977F6E547518D572D7BA63F6B50B5BBAF3DE8023DC542751723F391297B8B} VP:1 A:-1}
    Validator{840399906E468F3E87FA8C0865638D6F29D5C2EA PubKeyEd25519{F9D9656A60049083EEF61487632187B351294C1FA23D118060D813DB6D03E8B6} VP:1 A:-1}
    Validator{9234C27379290332CA13772144DB4E12FCA1FFAD PubKeyEd25519{AFEAC2C52A0AD3DA537BA87D0172720FD55FA4FDB3C44B0FD20BC812D18A8856} VP:1 A:9}
} ValidatorSet{
  Proposer: Validator{367CA8B17443585A2D013DA2916C10BFEC29D7F7 PubKeyEd25519{2A6977F6E547518D572D7BA63F6B50B5BBAF3DE8023DC542751723F391297B8B} VP:1 A:-2}
  Validators:
    Validator{008CB3E446C123CD0BE3D1C8BBDFA555CFB600E7 PubKeyEd25519{19596D570E9535F11CC34091DB98EC44A526E38657F826B5D6B6152E533B8C06} VP:1 A:-2}
    Validator{0223EB172703E02C2B73D9710F535B8C01480235 PubKeyEd25519{64BC7CD3DE254CC3A2A6E0AAEE5B841D0A671B22B487788F594FBF1EF329935C} VP:1 A:-2}
    Validator{04A5FE0DA11994D8B399860F7A704BF46E1FEBED PubKeyEd25519{A3BE03FF2390210354A9AF6BEB418A957766B050F5A7C62662803CC267A944F6} VP:1 A:-2}
    Validator{0E27BBBE79F7D010439C78273D1B35F1CE4CFAF0 PubKeyEd25519{D9922819A39E94DDF3907F4B9C8DF93F39F026244FCB609205B9A879022599F2} VP:1 A:-2}
    Validator{10849BB652FED684976249438116C6FD3552C035 PubKeyEd25519{D85E5B1590D898D1E0CDEBB2E3B5337C8B76270142663D78811683BA47C17C98} VP:1 A:-2}
    Validator{1093D97B9310C072EED7A4957DFEDE346C76B13D PubKeyEd25519{8E222AE7E2ADCFB87A2984A19AAD52B1979ED1472C3CB17239A73EF1D344C35A} VP:1 A:-2}
    Validator{355DBE857FED7715F9C2AC12AE1A0E9E7C316EBD PubKeyEd25519{3DF80964CC83921E08C1FA0A4F5FC05810A634DA45461B2B315FCBFD62F7CAB7} VP:1 A:-2}
    Validator{367CA8B17443585A2D013DA2916C10BFEC29D7F7 PubKeyEd25519{2A6977F6E547518D572D7BA63F6B50B5BBAF3DE8023DC542751723F391297B8B} VP:1 A:-2}
    Validator{840399906E468F3E87FA8C0865638D6F29D5C2EA PubKeyEd25519{F9D9656A60049083EEF61487632187B351294C1FA23D118060D813DB6D03E8B6} VP:1 A:8}
    Validator{9234C27379290332CA13772144DB4E12FCA1FFAD PubKeyEd25519{AFEAC2C52A0AD3DA537BA87D0172720FD55FA4FDB3C44B0FD20BC812D18A8856} VP:1 A:8}
} 1 {{22020096 -1 500} {100000 48h0m0s 1048576} {[ed25519]} {0}} 1 [205 200 243 152 215 234 219 86 61 184 125 106 182 17 235 31 154 35 138 248 190 252 154 83 54 183 228 193 159 16 214 179] [209 247 84 110 3 77 27 203 93 22 28 241 31 18 201 80 20 192 228 235 65 22 26 12 39 172 255 58 238 23 116 79]}

Did you reset Tendermint without resetting your application's data?

goroutine 33 [running]:
github.com/tendermint/tendermint/consensus.assertAppHashEqualsOneFromState({_, _, _}, {{{0xb, 0x1}, {0xc0015b3f98, 0x7}}, {0xc0015b3fc0, 0xb}, 0x1, ...})
        /go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/replay.go:525 +0x12a
github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlocks(_, {{{0xb, 0x1}, {0xc0015b3f98, 0x7}}, {0xc0015b3fc0, 0xb}, 0x1, 0x561786, {{0xc000ecc720, ...}, ...}, ...}, ...)
        /go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/replay.go:488 +0x3e9
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x1}, {0xc0015b3f98, 0x7}}, {0xc0015b3fc0, 0xb}, 0x1, 0x561786, {{0xc000ecc720, ...}, ...}, ...}, ...)
        /go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/replay.go:393 +0x479
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc001016e48, {0x1539968, 0xc0010ead00})
        /go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/replay.go:268 +0x3d4
github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x1}, {0xc0015b3f98, 0x7}}, {0xc0015b3fc0, 0xb}, 0x1, 0x561786, ...}, ...)
        /go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/node/node.go:329 +0x1b8
github.com/tendermint/tendermint/node.NewNode(0xc0000c8140, {0x152b2d0, 0xc0002b6640}, 0xc0010ef2c0, {0x15222c0, 0xc0010f2c60}, 0x11b71ba93?, 0x1db0080?, 0xc0010ef300, {0x152fc40, ...}, ...)
        /go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/node/node.go:777 +0x597
github.com/iotaledger/inx-tendercoo/components/decoo.newTendermintNode(0xc0000c8140)
        /scratch/components/decoo/tendermint.go:204 +0x4b6
github.com/iotaledger/inx-tendercoo/components/decoo.run.func2({0x152e350, 0xc0000e2320})
        /scratch/components/decoo/component.go:216 +0x1a8
github.com/iotaledger/hive.go/app/daemon.(*OrderedDaemon).runBackgroundWorker.func1()
        /go/pkg/mod/github.com/iotaledger/hive.go/app@v0.0.0-20230417125513-e2e89991217f/daemon/daemon.go:163 +0xf2
created by github.com/iotaledger/hive.go/app/daemon.(*OrderedDaemon).runBackgroundWorker
        /go/pkg/mod/github.com/iotaledger/hive.go/app@v0.0.0-20230417125513-e2e89991217f/daemon/daemon.go:158 +0x13a
shufps commented 1 year ago

this time with more logging outputs:

2023-06-02T00:09:26Z    DEBUG   Coordinator     create milestone        {"state": {"MilestoneHeight":5906520,"MilestoneIndex":9912319,"LastMilestoneID":"04f35ecb367d18ecd75d260fb12c38afa984212d587ac803c33f7f06b3d5d8a2","LastMilestoneBlockID":"56def2f78d6073371a5f07945a4e
6cfebb298bb1918413846bbb980190799195"}, "parents": ["56def2f78d6073371a5f07945a4e6cfebb298bb1918413846bbb980190799195","ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02"]}
2023-06-02T00:09:26Z    INFO    Tendermint      executed block  {"module": "state", "height": 5906523, "num_valid_txs": 17, "num_invalid_txs": 0}
2023-06-02T00:09:26Z    DEBUG   Coordinator     broadcast tx    {"partial": {"Signature":"ou7uNWvq/j5Rt1HnR63Z174CLDlPFzFj4IJIOPctznp2+v0vXCLlNzEPdpSYyjDS7XjfalNxkcR6MNDTbI13Cg=="}}
2023-06-02T00:09:26Z    DEBUG   Coordinator     Commit-aftercommit      {"stateHash": "0x5437222422ac24d5d0af94daac8a9b92ce5081f36a2c07e39c998778c533e980", "state": {"MilestoneHeight":5906520,"MilestoneIndex":9912319,"LastMilestoneID":"04f35ecb367d18ecd75d260fb12c38afa98
4212d587ac803c33f7f06b3d5d8a2","LastMilestoneBlockID":"56def2f78d6073371a5f07945a4e6cfebb298bb1918413846bbb980190799195","ParentByIssuer":{"19596d570e9535f11cc34091db98ec44a526e38657f826b5d6b6152e533b8c06":"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02
","2a6977f6e547518d572d7ba63f6b50b5bbaf3de8023dc542751723f391297b8b":"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02","3df80964cc83921e08c1fa0a4f5fc05810a634da45461b2b315fcbfd62f7cab7":"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02","
64bc7cd3de254cc3a2a6e0aaee5b841d0a671b22b487788f594fbf1ef329935c":"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02","8e222ae7e2adcfb87a2984a19aad52b1979ed1472c3cb17239a73ef1d344c35a":"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02","a3b
e03ff2390210354a9af6beb418a957766b050f5a7c62662803cc267a944f6":"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02","afeac2c52a0ad3da537ba87d0172720fd55fa4fdb3c44b0fd20bc812d18a8856":"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02","d85e5b
1590d898d1e0cdebb2e3b5337c8b76270142663d78811683ba47c17c98":"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02","f9d9656a60049083eef61487632187b351294c1fa23d118060d813db6d03e8b6":"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02"},"IssuerCo
untByParent":{"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02":9},"ProofIssuersByBlockID":{"ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02":{"19596d570e9535f11cc34091db98ec44a526e38657f826b5d6b6152e533b8c06":{},"2a6977f6e547518d572d7ba
63f6b50b5bbaf3de8023dc542751723f391297b8b":{},"3df80964cc83921e08c1fa0a4f5fc05810a634da45461b2b315fcbfd62f7cab7":{},"64bc7cd3de254cc3a2a6e0aaee5b841d0a671b22b487788f594fbf1ef329935c":{},"8e222ae7e2adcfb87a2984a19aad52b1979ed1472c3cb17239a73ef1d344c35a":{},"a3be03ff239021
0354a9af6beb418a957766b050f5a7c62662803cc267a944f6":{},"afeac2c52a0ad3da537ba87d0172720fd55fa4fdb3c44b0fd20bc812d18a8856":{},"d85e5b1590d898d1e0cdebb2e3b5337c8b76270142663d78811683ba47c17c98":{},"f9d9656a60049083eef61487632187b351294c1fa23d118060d813db6d03e8b6":{}}},"Mil
estone":{"type":7,"index":9912319,"timestamp":1685664559,"protocolVersion":2,"previousMilestoneId":"0x04f35ecb367d18ecd75d260fb12c38afa984212d587ac803c33f7f06b3d5d8a2","parents":["0x56def2f78d6073371a5f07945a4e6cfebb298bb1918413846bbb980190799195","0xba6dee36651960e60dda
4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02"],"inclusionMerkleRoot":"0x8be25cc05159ffedab154e251971e1c28a83821e6515a13b8944fa7c795734fa","appliedMerkleRoot":"0x2e22bd402ccccfd221546829a848d83b315b629f4565aa09b597b89ca76760f8","metadata":"0x58205a000000000056def2f78d6073
371a5f07945a4e6cfebb298bb1918413846bbb980190799195","signatures":[]},"SignaturesByIssuer":{}}}
2023-06-02T00:09:26Z    INFO    Tendermint      committed state {"module": "state", "height": 5906523, "num_txs": 17, "app_hash": "5437222422AC24D5D0AF94DAAC8A9B92CE5081F36A2C07E39C998778C533E980"}
2023-06-02T00:09:26Z    INFO    Tendermint      indexed block exents    {"module": "txindex", "height": 5906523}
2023-06-02T00:09:26Z    INFO    Coordinator     New confirmed milestone: 9912319
2023-06-02T00:09:26Z    INFO    Coordinator     Proposing parent for milestone 9912320
2023-06-02T00:09:26Z    WARN    Coordinator     defaulting to last milestone: tip selection failed: no tips available
2023-06-02T00:09:26Z    INFO    Tendermint      commit is for a block we do not know about; set ProposalBlock=nil       {"module": "consensus", "height": 5906524, "commit_round": 1, "proposal": "", "commit": "C6F348DECCF270BF2B111BD1E015781DAEB03398EA7ACAB44437D8417F0642
54"}
2023-06-02T00:09:26Z    INFO    Tendermint      received complete proposal block        {"module": "consensus", "height": 5906524, "hash": "C6F348DECCF270BF2B111BD1E015781DAEB03398EA7ACAB44437D8417F064254"}
2023-06-02T00:09:26Z    ERROR   Tendermint      CONSENSUS FAILURE!!!    {"module": "consensus", "err": "+2/3 committed an invalid block: wrong Block.Header.AppHash.  Expected 5437222422AC24D5D0AF94DAAC8A9B92CE5081F36A2C07E39C998778C533E980, got E0269E863385CEE700A77FC0AF
5CF975F88C79BD8A21234F2FC0F28BD3C066C1", "stack": "goroutine 7609 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2()\n\t/go/pkg/mod/github.com/tendermint/tende
rmint@v0.34.24/consensus/state.go:724 +0x4c\npanic({0x10ca000, 0xc004c7c8c0})\n\t/usr/local/go/src/runtime/panic.go:884 +0x213\ngithub.com/tendermint/tendermint/consensus.(*State).finalizeCommit(0xc002a32700, 0x5a205c)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.3
4.24/consensus/state.go:1594 +0x1105\ngithub.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit(0xc002a32700, 0x5a205c)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:1563 +0x2ff\ngithub.com/tendermint/tendermint/consensus.(*State
).handleCompleteProposal(0xc002a32700, 0xc004e84c80?)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:1942 +0x399\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc002a32700, {{0x15220c0, 0xc0049acd50}, {0xc001fc7020, 0x28}})\n
\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:834 +0x1b7\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc002a32700, 0x0)\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:760 +0x3f0\ncreated by
 github.com/tendermint/tendermint/consensus.(*State).OnStart\n\t/go/pkg/mod/github.com/tendermint/tendermint@v0.34.24/consensus/state.go:379 +0x12d\n"}

The states differ ...

Good:

{
  "stateHash": "0xe0269e863385cee700a77fc0af5cf975f88c79bd8a21234f2fc0f28bd3c066c1",
  "state": {
    "MilestoneHeight": 5906520,
    "MilestoneIndex": 9912319,
    "LastMilestoneID": "04f35ecb367d18ecd75d260fb12c38afa984212d587ac803c33f7f06b3d5d8a2",
    "LastMilestoneBlockID": "56def2f78d6073371a5f07945a4e6cfebb298bb1918413846bbb980190799195",
    "ParentByIssuer": {
      "19596d570e9535f11cc34091db98ec44a526e38657f826b5d6b6152e533b8c06": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
      "2a6977f6e547518d572d7ba63f6b50b5bbaf3de8023dc542751723f391297b8b": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
      "3df80964cc83921e08c1fa0a4f5fc05810a634da45461b2b315fcbfd62f7cab7": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
      "64bc7cd3de254cc3a2a6e0aaee5b841d0a671b22b487788f594fbf1ef329935c": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
      "8e222ae7e2adcfb87a2984a19aad52b1979ed1472c3cb17239a73ef1d344c35a": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
      "a3be03ff2390210354a9af6beb418a957766b050f5a7c62662803cc267a944f6": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
      "afeac2c52a0ad3da537ba87d0172720fd55fa4fdb3c44b0fd20bc812d18a8856": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
      "d85e5b1590d898d1e0cdebb2e3b5337c8b76270142663d78811683ba47c17c98": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
      "f9d9656a60049083eef61487632187b351294c1fa23d118060d813db6d03e8b6": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02"
    },
    "IssuerCountByParent": {
      "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02": 9
    },
    "ProofIssuersByBlockID": {
      "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02": {
        "19596d570e9535f11cc34091db98ec44a526e38657f826b5d6b6152e533b8c06": {},
        "2a6977f6e547518d572d7ba63f6b50b5bbaf3de8023dc542751723f391297b8b": {},
        "3df80964cc83921e08c1fa0a4f5fc05810a634da45461b2b315fcbfd62f7cab7": {},
        "64bc7cd3de254cc3a2a6e0aaee5b841d0a671b22b487788f594fbf1ef329935c": {},
        "8e222ae7e2adcfb87a2984a19aad52b1979ed1472c3cb17239a73ef1d344c35a": {},
        "a3be03ff2390210354a9af6beb418a957766b050f5a7c62662803cc267a944f6": {},
        "afeac2c52a0ad3da537ba87d0172720fd55fa4fdb3c44b0fd20bc812d18a8856": {},
        "d85e5b1590d898d1e0cdebb2e3b5337c8b76270142663d78811683ba47c17c98": {},
        "f9d9656a60049083eef61487632187b351294c1fa23d118060d813db6d03e8b6": {}
      }
    },
    "Milestone": {
      "type": 7,
      "index": 9912319,
      "timestamp": 1685664559,
      "protocolVersion": 2,
      "previousMilestoneId": "0x04f35ecb367d18ecd75d260fb12c38afa984212d587ac803c33f7f06b3d5d8a2",
      "parents": [
        "0x56def2f78d6073371a5f07945a4e6cfebb298bb1918413846bbb980190799195",
        "0xba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02"
      ],
      "inclusionMerkleRoot": "0x15405e649707955e66c4e9e9ba62fb5436b7e2acf3ff6e3de1ec7421273fac9d",
      "appliedMerkleRoot": "0x2e22bd402ccccfd221546829a848d83b315b629f4565aa09b597b89ca76760f8",
      "metadata": "0x58205a000000000056def2f78d6073371a5f07945a4e6cfebb298bb1918413846bbb980190799195",
      "signatures": []
    },
    "SignaturesByIssuer": {}
  }
}

bad:

{
  "stateHash": "0x5437222422ac24d5d0af94daac8a9b92ce5081f36a2c07e39c998778c533e980",
  "state": {
    "MilestoneHeight": 5906520,
    "MilestoneIndex": 9912319,
    "LastMilestoneID": "04f35ecb367d18ecd75d260fb12c38afa984212d587ac803c33f7f06b3d5d8a2",
    "LastMilestoneBlockID": "56def2f78d6073371a5f07945a4e6cfebb298bb1918413846bbb980190799195",
    "ParentByIssuer": {
      "19596d570e9535f11cc34091db98ec44a526e38657f826b5d6b6152e533b8c06": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
      "2a6977f6e547518d572d7ba63f6b50b5bbaf3de8023dc542751723f391297b8b": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
      "3df80964cc83921e08c1fa0a4f5fc05810a634da45461b2b315fcbfd62f7cab7": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
      "64bc7cd3de254cc3a2a6e0aaee5b841d0a671b22b487788f594fbf1ef329935c": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
      "8e222ae7e2adcfb87a2984a19aad52b1979ed1472c3cb17239a73ef1d344c35a": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
      "a3be03ff2390210354a9af6beb418a957766b050f5a7c62662803cc267a944f6": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
      "afeac2c52a0ad3da537ba87d0172720fd55fa4fdb3c44b0fd20bc812d18a8856": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
      "d85e5b1590d898d1e0cdebb2e3b5337c8b76270142663d78811683ba47c17c98": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02",
      "f9d9656a60049083eef61487632187b351294c1fa23d118060d813db6d03e8b6": "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02"
    },
    "IssuerCountByParent": {
      "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02": 9
    },
    "ProofIssuersByBlockID": {
      "ba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02": {
        "19596d570e9535f11cc34091db98ec44a526e38657f826b5d6b6152e533b8c06": {},
        "2a6977f6e547518d572d7ba63f6b50b5bbaf3de8023dc542751723f391297b8b": {},
        "3df80964cc83921e08c1fa0a4f5fc05810a634da45461b2b315fcbfd62f7cab7": {},
        "64bc7cd3de254cc3a2a6e0aaee5b841d0a671b22b487788f594fbf1ef329935c": {},
        "8e222ae7e2adcfb87a2984a19aad52b1979ed1472c3cb17239a73ef1d344c35a": {},
        "a3be03ff2390210354a9af6beb418a957766b050f5a7c62662803cc267a944f6": {},
        "afeac2c52a0ad3da537ba87d0172720fd55fa4fdb3c44b0fd20bc812d18a8856": {},
        "d85e5b1590d898d1e0cdebb2e3b5337c8b76270142663d78811683ba47c17c98": {},
        "f9d9656a60049083eef61487632187b351294c1fa23d118060d813db6d03e8b6": {}
      }
    },
    "Milestone": {
      "type": 7,
      "index": 9912319,
      "timestamp": 1685664559,
      "protocolVersion": 2,
      "previousMilestoneId": "0x04f35ecb367d18ecd75d260fb12c38afa984212d587ac803c33f7f06b3d5d8a2",
      "parents": [
        "0x56def2f78d6073371a5f07945a4e6cfebb298bb1918413846bbb980190799195",
        "0xba6dee36651960e60dda4a3d62c4f2842ea396eee27449faeb7175e7ea9ecd02"
      ],
      "inclusionMerkleRoot": "0x8be25cc05159ffedab154e251971e1c28a83821e6515a13b8944fa7c795734fa",
      "appliedMerkleRoot": "0x2e22bd402ccccfd221546829a848d83b315b629f4565aa09b597b89ca76760f8",
      "metadata": "0x58205a000000000056def2f78d6073371a5f07945a4e6cfebb298bb1918413846bbb980190799195",
      "signatures": []
    },
    "SignaturesByIssuer": {}
  }
}
shufps commented 1 year ago

Mmhmm comparing the good and the bad gives:

diff good_jq.txt bad_jq.txt 
2c2
<   "stateHash": "0xe0269e863385cee700a77fc0af5cf975f88c79bd8a21234f2fc0f28bd3c066c1",
---
>   "stateHash": "0x5437222422ac24d5d0af94daac8a9b92ce5081f36a2c07e39c998778c533e980",
45c45
<       "inclusionMerkleRoot": "0x15405e649707955e66c4e9e9ba62fb5436b7e2acf3ff6e3de1ec7421273fac9d",
---
>       "inclusionMerkleRoot": "0x8be25cc05159ffedab154e251971e1c28a83821e6515a13b8944fa7c795734fa",

so the reason for the changed app state is connected to a different inclusionMerkleRoot :thinking:

shufps commented 1 year ago

I added additional logging to whiteflag computation for further analysis:

https://github.com/iotaledger/inx-tendercoo/commit/f7a3e4086557b9d3e55d37f5da4d0ff32bc25237

shufps commented 1 year ago

so the main question here is, how can the inclMerkleRoot be wrong at this place? (parameters of the call are correct)

https://github.com/iotaledger/inx-tendercoo/blob/develop/pkg/decoo/app.go#L352

shufps commented 1 year ago

Hornet (white_flag):

// compute past cone merkle tree root hash
confirmedMerkleHash := merklehasher.NewHasher(crypto.BLAKE2b_256).HashBlockIDs(wfConf.ReferencedBlocks.BlockIDs())
copy(wfConf.InclusionMerkleRoot[:], confirmedMerkleHash)

// compute inclusion merkle tree root hash
appliedMerkleHash := merklehasher.NewHasher(crypto.BLAKE2b_256).HashBlockIDs(wfConf.ReferencedBlocks.IncludedTransactionBlockIDs())
copy(wfConf.AppliedMerkleRoot[:], appliedMerkleHash)

applied was okay but inclusion was different :thinking:

the question would be: Can a solid cone of a parent be different (e.g. has more or less blocks) on different nodes if a node lags behind, leading to different inclusionMerkleRoot :thinking:

shufps commented 1 year ago

New information:

coo-1 (broken):

2023-06-20T09:55:11Z    DEBUG   Coordinator     WhiteFlag-Start {"cmi": 10225138, "index": 10225139, "timestamp": 1687254908, "parents": ["d93a622f4b31e489c337c9fa656527901a39396861a98ae968e7cf54e5d52cb3","de626c6a82eb4e979e82da69fcb73bd8d300924136ee7d8fbdf2904d625689a3"], "previousMilestoneId": "492cdd603c017115b092a7bee2c6d322733022ae4b6587cdd110736902f2e1f2"}
2023-06-20T09:55:11Z    DEBUG   Coordinator     WhiteFlag-Compute       {"index": 10225139, "timestamp": 1687254908, "parents": ["d93a622f4b31e489c337c9fa656527901a39396861a98ae968e7cf54e5d52cb3","de626c6a82eb4e979e82da69fcb73bd8d300924136ee7d8fbdf2904d625689a3"], "previousMilestoneId": "492cdd603c017115b092a7bee2c6d322733022ae4b6587cdd110736902f2e1f2"}
2023-06-20T09:55:11Z    DEBUG   Coordinator     WhiteFlag-Return        {"inclusionMerkleRoot": "0x0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8", "appliedMerkleRoot": "0x0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8", "err": null}

coo-0 (good)

2023-06-20T09:55:10Z    DEBUG   Coordinator     WhiteFlag-Start {"cmi": 10225138, "index": 10225139, "timestamp": 1687254908, "parents": ["d93a622f4b31e489c337c9fa656527901a39396861a98ae968e7cf54e5d52cb3","de626c6a82eb4e979e82da69fcb73bd8d300924136ee7d8fbdf2904d625689a3"], "previousMilestoneId": "492cdd603c017115b092a7bee2c6d322733022ae4b6587cdd110736902f2e1f2"}
2023-06-20T09:55:10Z    DEBUG   Coordinator     WhiteFlag-Compute       {"index": 10225139, "timestamp": 1687254908, "parents": ["d93a622f4b31e489c337c9fa656527901a39396861a98ae968e7cf54e5d52cb3","de626c6a82eb4e979e82da69fcb73bd8d300924136ee7d8fbdf2904d625689a3"], "previousMilestoneId": "492cdd603c017115b092a7bee2c6d322733022ae4b6587cdd110736902f2e1f2"}
2023-06-20T09:55:10Z    DEBUG   Coordinator     WhiteFlag-Return        {"inclusionMerkleRoot": "0xd632fa4d5d275a1aa38e60283bdb3b2f881cf304e6aa6ea2f0ff4d2fd3073829", "appliedMerkleRoot": "0x0f5fb22ca84363f14d80b3ff85e45222594dfd17a1266a70be7022e94e08550e", "err": null}
shufps commented 1 year ago

Hornet logs for this section: coo-1 (bad):

2023-06-20T09:55:11Z    INFO    Tangle  Valid milestone detected! Index: 10225139
2023-06-20T09:55:11Z    INFO    Tangle  Run solidity check for Milestone (10225139) ...
2023-06-20T09:55:11Z    INFO    Tangle  Solidifier finished: blocks: 0, collect: 0s, solidity 0s, propagation: 0s, total: 0s
2023-06-20T09:55:11Z    INFO    Glogger WhiteFlag Included Hash: 0xd632fa4d5d275a1aa38e60283bdb3b2f881cf304e6aa6ea2f0ff4d2fd3073829, Blocks: [0xd93a622f4b31e489c337c9fa656527901a39396861a98ae968e7cf54e5d52cb3 0x1f801a6c330e658996a9c2e812495ddedd980e1279dea8b66173af0e0b44
96b8 0x81778629b8b793117e4f5e536724dc09f4f2342f956f5add16139bd67a43a53b 0x8c9eed94fb47cc02299b48f9bcaae178a6a8cffe3876b2224cfcd71e850711b8 0x4790efdb1a2199f233ddae9a75c71bea6bfe7803766552e3e95056c87636cdcd 0x79512b1179ce915049d17d8eb0477615113ef67d03a1eb8e9d74f7d43dcfb5a
b 0xbb2e615cf8626aeb2cdc058bc88b30b1ad52ae13b263c50770eccf094f13d60d 0xd100da96a75a0b1a52280559d80949b96751db437d6b334ac9c7cb77df7b2da8 0x4ac1ad0bad6577a2e70a35a077a3526545031acd89ee6d3e6e408c78716acd36 0x4566d98ab6a7a8f4a8c5d367223ffa9462b46e97647c0ec8964386d3aa680530 0
x16f1e6f6c8cb83d3772199583f76edffe796ab95c0aba89c5c5efc1d3f287f67 0x5f9550df4661861fdc485a0e4303b1625af43df0891bfbd0006ba63956f6b622 0xd2f74cc2ddebe85991fa3573ede6e52b71999a806fe8fb88f255730deec57f1d 0xde626c6a82eb4e979e82da69fcb73bd8d300924136ee7d8fbdf2904d625689a3]
2023-06-20T09:55:11Z    INFO    Glogger WhiteFlag Applied Hash: 0x0f5fb22ca84363f14d80b3ff85e45222594dfd17a1266a70be7022e94e08550e, Blocks: [0x5f9550df4661861fdc485a0e4303b1625af43df0891bfbd0006ba63956f6b622]
2023-06-20T09:55:11Z    INFO    Tangle  Solidifier finished: blocks: 0, collect: 0s, solidity 0s, propagation: 4ms, total: 4ms
2023-06-20T09:55:11Z    INFO    Glogger WhiteFlag Included Hash: 0x0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8, Blocks: []
2023-06-20T09:55:11Z    INFO    Glogger WhiteFlag Applied Hash: 0x0e5751c026e543b2e8ab2eb06099daa1d1e5df47778f7787faab45cdf12fe3a8, Blocks: []
2023-06-20T09:55:11Z    INFO    Tangle  Milestone confirmed (10225139): txsReferenced: 14, txsValue: 1, txsZeroValue: 13, txsConflicting: 0, collect: 1ms, total: 7ms
2023-06-20T09:55:11Z    INFO    Tangle  New confirmed milestone: 10225139, 0.67 BPS, 1.56 RBPS, 233.33% ref.rate

coo-0 (good):

2023-06-20T09:55:11Z    INFO    Glogger WhiteFlag Included Hash: 0xd632fa4d5d275a1aa38e60283bdb3b2f881cf304e6aa6ea2f0ff4d2fd3073829, Blocks: [0xd93a622f4b31e489c337c9fa656527901a39396861a98ae968e7cf54e5d52cb3 0x1f801a6c330e658996a9c2e812495ddedd980e1279dea8b66173af0e0b4496b8 0x81778629b8b793117e4f5e536724dc09f4f2342f956f5add16139bd67a43a53b 0x8c9eed94fb47cc02299b48f9bcaae178a6a8cffe3876b2224cfcd71e850711b8 0x4790efdb1a2199f233ddae9a75c71bea6bfe7803766552e3e95056c87636cdcd 0x79512b1179ce915049d17d8eb0477615113ef67d03a1eb8e9d74f7d43dcfb5ab 0xbb2e615cf8626aeb2cdc058bc88b30b1ad52ae13b263c50770eccf094f13d60d 0xd100da96a75a0b1a52280559d80949b96751db437d6b334ac9c7cb77df7b2da8 0x4ac1ad0bad6577a2e70a35a077a3526545031acd89ee6d3e6e408c78716acd36 0x4566d98ab6a7a8f4a8c5d367223ffa9462b46e97647c0ec8964386d3aa680530 0x16f1e6f6c8cb83d3772199583f76edffe796ab95c0aba89c5c5efc1d3f287f67 0x5f9550df4661861fdc485a0e4303b1625af43df0891bfbd0006ba63956f6b622 0xd2f74cc2ddebe85991fa3573ede6e52b71999a806fe8fb88f255730deec57f1d 0xde626c6a82eb4e979e82da69fcb73bd8d300924136ee7d8fbdf2904d625689a3]
2023-06-20T09:55:11Z    INFO    Glogger WhiteFlag Applied Hash: 0x0f5fb22ca84363f14d80b3ff85e45222594dfd17a1266a70be7022e94e08550e, Blocks: [0x5f9550df4661861fdc485a0e4303b1625af43df0891bfbd0006ba63956f6b622]
shufps commented 1 year ago

Max found a possible explanation (missing mutex on whiteflag calculation) and provided a fix for the issue:

https://github.com/iotaledger/hornet/commit/7fad952203ed1678054607f715224c317b6f6917

shufps commented 1 year ago

With the fix, problem seems to be solve.