search

osmosis-labs / osmosis

The AMM Laboratory
https://app.osmosis.zone
Apache License 2.0
875 stars 565 forks source link

tendermint wrong Block.Header.AppHash bug #1090

Closed p0mvn closed 1 year ago

p0mvn commented 2 years ago

Context

Report 1

Logs

 4:00AM INF indexed block height=3490885 module=txindex
 4:00AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=50 numToDial=0
 4:00AM INF Timed out dur=4685.039831 height=3490886 module=consensus round=0 step=1
 4:00AM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"467CC1EC467FF070ADBE4DBE22FC55C114F41D633D1E41E4CE372A19C2C97C50","parts":{"hash":"6C3F2E43DEDE6B49E7227E11CC6898161AEE78A3E06928DFC012AB0785B8886F","total":3}},"height":3490886,"pol_round":-1,"round":0,"signature":"mbzXIrrMQPoL4RiLaj4RdLdMj3CMFCJLqUl/CejYOh2s9/Ar40+DZVGPEN6nE7pGXJwK9sD20oYg+bCUmIMOAQ==","timestamp":"2022-03-07T09:00:07.314313098Z"}
 4:00AM INF received complete proposal block hash=467CC1EC467FF070ADBE4DBE22FC55C114F41D633D1E41E4CE372A19C2C97C50 height=3490886 module=consensus
 4:00AM INF finalizing commit of block hash=467CC1EC467FF070ADBE4DBE22FC55C114F41D633D1E41E4CE372A19C2C97C50 height=3490886 module=consensus num_txs=15 root=2FC146C364361339D8A274F72BF23CD96B9E4797261635B81F9196541506841B
 4:00AM INF burned tokens from module account amount=12309533ibc/987C17B11ABC2B20019178ACE62929FE9840202CE79498E29FE8E5CB02B7C0A4 from=transfer module=x/bank
 4:00AM INF packet sent dst_channel=channel-0 dst_port=transfer module=x/ibc/channel sequence=90381 src_channel=channel-75 src_port=transfer
 4:00AM INF IBC fungible token transfer amount=12309533 module=x/ibc-transfer receiver=stars1rdkfvdrvmxw7zlazwaqnuusm75ql60eqcyjmdp sender=osmo1rdkfvdrvmxw7zlazwaqnuusm75ql60eqyrkksz token=ibc/987C17B11ABC2B20019178ACE62929FE9840202CE79498E29FE8E5CB02B7C0A4
 4:00AM INF burned tokens from module account amount=36651579ibc/BE1BB42D4BE3C30D50B68D7C41DB4DFCE9678E8EF8C539F6E6A9345048894FCC from=transfer module=x/bank
 4:00AM INF packet sent dst_channel=channel-1 dst_port=transfer module=x/ibc/channel sequence=209719 src_channel=channel-72 src_port=transfer
 4:00AM INF IBC fungible token transfer amount=36651579 module=x/ibc-transfer receiver=terra1hx36chwr5c5tu05njjv37jme2t2vwwe42jffts sender=osmo1ememjxygxrtrdqe0xqr27tc3ms5aam9k5dc77u token=ibc/BE1BB42D4BE3C30D50B68D7C41DB4DFCE9678E8EF8C539F6E6A9345048894FCC
 4:00AM INF client state updated client-id=07-tendermint-1 height=4-9665606 module=x/ibc/client
 4:00AM INF packet received module=x/ibc/channel packet="{594255 transfer channel-141 transfer channel-0 [123 34 97 109 111 117 110 116 34 58 34 55 56 48 48 48 48 48 48 34 44 34 100 101 110 111 109 34 58 34 117 97 116 111 109 34 44 34 114 101 99 101 105 118 101 114 34 58 34 111 115 109 111 49 97 50 48 52 52 117 121 118 97 52 122 100 50 57 122 121 108 102 109 113 104 99 120 119 53 118 115 53 50 50 117 102 54 101 108 99 50 53 34 44 34 115 101 110 100 101 114 34 58 34 99 111 115 109 111 115 49 97 50 48 52 52 117 121 118 97 52 122 100 50 57 122 121 108 102 109 113 104 99 120 119 53 118 115 53 50 50 117 102 106 122 118 103 117 120 34 125] 1-3490900 0}"
 4:00AM INF minted coins from module account amount=78000000ibc/27394FB092D2ECCD56123C74F36E4C1F926001CEADA9CA97EA622B25F41E5EB2 from=transfer module=x/bank
 4:00AM INF acknowledged written module=x/ibc/channel packet="{594255 transfer channel-141 transfer channel-0 [123 34 97 109 111 117 110 116 34 58 34 55 56 48 48 48 48 48 48 34 44 34 100 101 110 111 109 34 58 34 117 97 116 111 109 34 44 34 114 101 99 101 105 118 101 114 34 58 34 111 115 109 111 49 97 50 48 52 52 117 121 118 97 52 122 100 50 57 122 121 108 102 109 113 104 99 120 119 53 118 115 53 50 50 117 102 54 101 108 99 50 53 34 44 34 115 101 110 100 101 114 34 58 34 99 111 115 109 111 115 49 97 50 48 52 52 117 121 118 97 52 122 100 50 57 122 121 108 102 109 113 104 99 120 119 53 118 115 53 50 50 117 102 106 122 118 103 117 120 34 125] 1-3490900 0}"
 4:00AM INF packet acknowledged module=x/ibc/channel packet="{528676 transfer channel-0 transfer channel-141 [123 34 97 109 111 117 110 116 34 58 34 50 48 49 57 53 52 48 54 34 44 34 100 101 110 111 109 34 58 34 116 114 97 110 115 102 101 114 47 99 104 97 110 110 101 108 45 48 47 117 97 116 111 109 34 44 34 114 101 99 101 105 118 101 114 34 58 34 99 111 115 109 111 115 49 57 113 55 53 109 116 53 103 51 120 57 107 121 57 104 104 101 48 122 108 54 116 100 54 55 103 97 116 97 102 121 54 53 122 102 109 118 119 34 44 34 115 101 110 100 101 114 34 58 34 111 115 109 111 49 57 113 55 53 109 116 53 103 51 120 57 107 121 57 104 104 101 48 122 108 54 116 100 54 55 103 97 116 97 102 121 54 117 101 54 116 54 117 34 125] 4-9665747 0}"
 4:00AM INF client state updated client-id=07-tendermint-1 height=4-9665606 module=x/ibc/client
 4:00AM INF client state updated client-id=07-tendermint-1 height=4-9665606 module=x/ibc/client
 4:00AM INF client state updated client-id=07-tendermint-1 height=4-9665606 module=x/ibc/client
 4:00AM INF minted coins from module account amount=368922023284144188095228gamm/pool/5 from=gamm module=x/bank
 4:00AM INF executed block height=3490886 module=state num_invalid_txs=2 num_valid_txs=13
 4:00AM INF commit synced commit=436F6D6D697449447B5B313731203138372038312035342031392031362031323220313030203134392031333920323720323020313433203137342037372031323920383320313830203335203136332031392031383720353720383620323036203736203139372031323420393420323033203132372037335D3A3335343434367D
 4:00AM INF committed state app_hash=ABBB513613107A64958B1B148FAE4D8153B423A313BB3956CE4CC57C5ECB7F49 height=3490886 module=state num_txs=15
 4:00AM INF indexed block height=3490886 module=txindex
 4:00AM INF Timed out dur=4783.291405 height=3490887 module=consensus round=0 step=1
 4:00AM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"1DAEBEEAC92B5B6D3FB1CE0576ED6D3D0DF221E8481E686FDA7F192993DBE3B8","parts":{"hash":"9909087200424547E0A60F42768B74BA3CF92E8E130F3F4840E9C9B609B077C3","total":1}},"height":3490887,"pol_round":-1,"round":0,"signature":"0jUmccX06Jw8DByVVHbsBb43R3Go+OiqDZRctHbnG1aS/1BesNWg38DR6fESkEN+BrKidkvdw4P3LaddcoXVBQ==","timestamp":"2022-03-07T09:00:13.285334062Z"}
 4:00AM INF received complete proposal block hash=1DAEBEEAC92B5B6D3FB1CE0576ED6D3D0DF221E8481E686FDA7F192993DBE3B8 height=3490887 module=consensus
 4:00AM INF finalizing commit of block hash=1DAEBEEAC92B5B6D3FB1CE0576ED6D3D0DF221E8481E686FDA7F192993DBE3B8 height=3490887 module=consensus num_txs=17 root=ABBB513613107A64958B1B148FAE4D8153B423A313BB3956CE4CC57C5ECB7F49
 4:00AM INF minted coins from module account amount=36199772072901574754gamm/pool/1 from=gamm module=x/bank
 4:00AM INF executed block height=3490887 module=state num_invalid_txs=6 num_valid_txs=11
 4:00AM INF commit synced commit=436F6D6D697449447B5B34342032343020313030203135382032303620313734203136332032323920323220363420313232203135203138332032313820343820313120313331203130312031393020313335203931203234203130372032323520323120313138203130362031203131302031303320313436203232355D3A3335343434377D
 4:00AM INF committed state app_hash=2CF0649ECEAEA3E516407A0FB7DA300B8365BE875B186BE115766A016E6792E1 height=3490887 module=state num_txs=17
 4:00AM INF indexed block height=3490887 module=txindex
 4:00AM INF client state updated client-id=07-tendermint-1457 height=1-2178766 module=x/ibc/client
 4:00AM INF packet acknowledged module=x/ibc/channel packet="{335723 transfer channel-42 transfer channel-0 [123 34 97 109 111 117 110 116 34 58 34 49 48 48 48 48 48 48 48 34 44 34 100 101 110 111 109 34 58 34 116 114 97 110 115 102 101 114 47 99 104 97 110 110 101 108 45 52 50 47 117 106 117 110 111 34 44 34 114 101 99 101 105 118 101 114 34 58 34 106 117 110 111 49 104 113 53 121 121 48 115 51 119 108 48 53 119 114 57 121 119 119 56 122 108 115 114 117 113 112 97 112 112 100 50 55 112 104 121 101 51 113 34 44 34 115 101 110 100 101 114 34 58 34 111 115 109 111 49 104 113 53 121 121 48 115 51 119 108 48 53 119 114 57 121 119 119 56 122 108 115 114 117 113 112 97 112 112 100 50 55 108 55 53 106 113 119 34 125] 1-2178909 0}"
 4:00AM INF packet acknowledged module=x/ibc/channel packet="{335724 transfer channel-42 transfer channel-0 [123 34 97 109 111 117 110 116 34 58 34 49 53 54 51 56 56 53 34 44 34 100 101 110 111 109 34 58 34 116 114 97 110 115 102 101 114 47 99 104 97 110 110 101 108 45 52 50 47 117 106 117 110 111 34 44 34 114 101 99 101 105 118 101 114 34 58 34 106 117 110 111 49 113 112 99 53 103 109 116 52 52 99 110 53 109 117 55 104 119 57 100 117 102 56 55 53 106 56 116 54 100 52 107 114 119 115 108 54 100 109 34 44 34 115 101 110 100 101 114 34 58 34 111 115 109 111 49 113 112 99 53 103 109 116 52 52 99 110 53 109 117 55 104 119 57 100 117 102 56 55 53 106 56 116 54 100 52 107 114 115 101 48 51 117 52 34 125] 1-2178910 0}"
 4:00AM INF Timed out dur=4815.169894 height=3490888 module=consensus round=0 step=1
 4:00AM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"247C12BE0218EFCA47D1936BC389489A6A631DBE5F8EC7787EF2129DB3301A21","parts":{"hash":"BF2AE3FFD2BFC8474397CEAB679F2E32DA716DC3F88982F7349F06A70D8ADAC0","total":3}},"height":3490888,"pol_round":-1,"round":0,"signature":"zdgzoFLF9kX4OPE99sscYZXT7CFieOR7g4++dZljLlv0HogEnQKGkKW7qk0lfwI78SwW1hB0/4glRk9z9EYsCg==","timestamp":"2022-03-07T09:00:19.425930568Z"}
 4:00AM INF received complete proposal block hash=247C12BE0218EFCA47D1936BC389489A6A631DBE5F8EC7787EF2129DB3301A21 height=3490888 module=consensus
 4:00AM INF client state updated client-id=07-tendermint-1457 height=1-2178767 module=x/ibc/client
 4:00AM INF packet acknowledged module=x/ibc/channel packet="{335725 transfer channel-42 transfer channel-0 [123 34 97 109 111 117 110 116 34 58 34 49 49 54 55 52 52 53 34 44 34 100 101 110 111 109 34 58 34 116 114 97 110 115 102 101 114 47 99 104 97 110 110 101 108 45 52 50 47 117 106 117 110 111 34 44 34 114 101 99 101 105 118 101 114 34 58 34 106 117 110 111 49 107 97 116 120 112 120 107 107 106 120 57 97 104 107 109 117 48 101 115 108 115 109 117 108 48 108 116 118 56 115 116 100 116 99 116 107 52 48 34 44 34 115 101 110 100 101 114 34 58 34 111 115 109 111 49 107 97 116 120 112 120 107 107 106 120 57 97 104 107 109 117 48 101 115 108 115 109 117 108 48 108 116 118 56 115 116 100 52 51 109 97 121 112 34 125] 1-2178912 0}"
 4:00AM INF packet acknowledged module=x/ibc/channel packet="{335726 transfer channel-42 transfer channel-0 [123 34 97 109 111 117 110 116 34 58 34 53 49 53 55 55 50 51 34 44 34 100 101 110 111 109 34 58 34 116 114 97 110 115 102 101 114 47 99 104 97 110 110 101 108 45 52 50 47 117 106 117 110 111 34 44 34 114 101 99 101 105 118 101 114 34 58 34 106 117 110 111 49 107 112 53 108 99 104 121 54 52 120 121 102 51 100 112 121 97 109 54 54 102 116 113 99 112 103 53 109 108 54 57 104 48 102 100 114 103 97 34 44 34 115 101 110 100 101 114 34 58 34 111 115 109 111 49 107 112 53 108 99 104 121 54 52 120 121 102 51 100 112 121 97 109 54 54 102 116 113 99 112 103 53 109 108 54 57 104 51 113 97 103 101 110 34 125] 1-2178912 0}"
 4:00AM INF finalizing commit of block hash=247C12BE0218EFCA47D1936BC389489A6A631DBE5F8EC7787EF2129DB3301A21 height=3490888 module=consensus num_txs=20 root=2CF0649ECEAEA3E516407A0FB7DA300B8365BE875B186BE115766A016E6792E1
 4:00AM INF client state updated client-id=07-tendermint-1457 height=1-2178766 module=x/ibc/client
 4:00AM INF packet acknowledged module=x/ibc/channel packet="{335723 transfer channel-42 transfer channel-0 [123 34 97 109 111 117 110 116 34 58 34 49 48 48 48 48 48 48 48 34 44 34 100 101 110 111 109 34 58 34 116 114 97 110 115 102 101 114 47 99 104 97 110 110 101 108 45 52 50 47 117 106 117 110 111 34 44 34 114 101 99 101 105 118 101 114 34 58 34 106 117 110 111 49 104 113 53 121 121 48 115 51 119 108 48 53 119 114 57 121 119 119 56 122 108 115 114 117 113 112 97 112 112 100 50 55 112 104 121 101 51 113 34 44 34 115 101 110 100 101 114 34 58 34 111 115 109 111 49 104 113 53 121 121 48 115 51 119 108 48 53 119 114 57 121 119 119 56 122 108 115 114 117 113 112 97 112 112 100 50 55 108 55 53 106 113 119 34 125] 1-2178909 0}"
 4:00AM INF packet acknowledged module=x/ibc/channel packet="{335724 transfer channel-42 transfer channel-0 [123 34 97 109 111 117 110 116 34 58 34 49 53 54 51 56 56 53 34 44 34 100 101 110 111 109 34 58 34 116 114 97 110 115 102 101 114 47 99 104 97 110 110 101 108 45 52 50 47 117 106 117 110 111 34 44 34 114 101 99 101 105 118 101 114 34 58 34 106 117 110 111 49 113 112 99 53 103 109 116 52 52 99 110 53 109 117 55 104 119 57 100 117 102 56 55 53 106 56 116 54 100 52 107 114 119 115 108 54 100 109 34 44 34 115 101 110 100 101 114 34 58 34 111 115 109 111 49 113 112 99 53 103 109 116 52 52 99 110 53 109 117 55 104 119 57 100 117 102 56 55 53 106 56 116 54 100 52 107 114 115 101 48 51 117 52 34 125] 1-2178910 0}"
 4:00AM INF minted coins from module account amount=28795uosmo from=superfluid module=x/bank
 4:00AM INF client state updated client-id=07-tendermint-1457 height=1-2178766 module=x/ibc/client
 4:00AM INF burned tokens from module account amount=614988ibc/27394FB092D2ECCD56123C74F36E4C1F926001CEADA9CA97EA622B25F41E5EB2 from=transfer module=x/bank
 4:00AM INF packet sent dst_channel=channel-141 dst_port=transfer module=x/ibc/channel sequence=528681 src_channel=channel-0 src_port=transfer
 4:00AM INF IBC fungible token transfer amount=614988 module=x/ibc-transfer receiver=cosmos1fadm7n8asynycdcp7uduz6y6t93qu6jjlr95nz sender=osmo1fadm7n8asynycdcp7uduz6y6t93qu6jjhcky9s token=ibc/27394FB092D2ECCD56123C74F36E4C1F926001CEADA9CA97EA622B25F41E5EB2
 4:00AM INF minted coins from module account amount=443450869341238985gamm/pool/626 from=gamm module=x/bank
 4:00AM INF client state updated client-id=07-tendermint-1666 height=1-1198292 module=x/ibc/client
 4:00AM INF packet received module=x/ibc/channel packet="{193063 transfer channel-7 transfer channel-113 [123 34 97 109 111 117 110 116 34 58 34 49 52 55 48 48 48 48 48 48 34 44 34 100 101 110 111 109 34 58 34 117 104 117 97 104 117 97 34 44 34 114 101 99 101 105 118 101 114 34 58 34 111 115 109 111 49 50 52 119 48 121 55 100 51 52 121 118 57 116 104 114 104 102 116 53 100 109 56 120 113 117 55 51 50 51 48 57 113 118 52 50 106 54 102 34 44 34 115 101 110 100 101 114 34 58 34 99 104 105 104 117 97 104 117 97 49 55 118 56 104 100 115 55 103 104 56 50 53 107 50 120 101 56 50 122 118 119 117 108 117 56 54 53 52 50 53 52 54 114 97 103 53 107 52 34 125] 1-3491884 0}"
 4:00AM INF minted coins from module account amount=147000000ibc/B9E0A1A524E98BB407D3CED8720EFEFD186002F90C1B1B7964811DD0CCC12228 from=transfer module=x/bank
 4:00AM INF acknowledged written module=x/ibc/channel packet="{193063 transfer channel-7 transfer channel-113 [123 34 97 109 111 117 110 116 34 58 34 49 52 55 48 48 48 48 48 48 34 44 34 100 101 110 111 109 34 58 34 117 104 117 97 104 117 97 34 44 34 114 101 99 101 105 118 101 114 34 58 34 111 115 109 111 49 50 52 119 48 121 55 100 51 52 121 118 57 116 104 114 104 102 116 53 100 109 56 120 113 117 55 51 50 51 48 57 113 118 52 50 106 54 102 34 44 34 115 101 110 100 101 114 34 58 34 99 104 105 104 117 97 104 117 97 49 55 118 56 104 100 115 55 103 104 56 50 53 107 50 120 101 56 50 122 118 119 117 108 117 56 54 53 52 50 53 52 54 114 97 103 53 107 52 34 125] 1-3491884 0}"
 4:00AM INF client state updated client-id=07-tendermint-1666 height=1-1198292 module=x/ibc/client
 message repeated 2 times: [ 4:00AM INF client state updated client-id=07-tendermint-1666 height=1-1198292 module=x/ibc/client]
 4:00AM INF executed block height=3490888 module=state num_invalid_txs=3 num_valid_txs=17
 4:00AM INF commit synced commit=436F6D6D697449447B5B31322031343120353720313220313638203134362031313520313439203231203234322031323020333320363220313933203634203233382038332031303420313120313536203136352031383120383020313330203235352031303820313436203430203136382032343420313833203235325D3A3335343434387D
 4:00AM INF committed state app_hash=0C8D390CA892739515F278213EC140EE53680B9CA5B55082FF6C9228A8F4B7FC height=3490888 module=state num_txs=20
 4:00AM INF indexed block height=3490888 module=txindex
 4:00AM INF Timed out dur=4648.25283 height=3490889 module=consensus round=0 step=1
 4:00AM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"28ACA34439C4F038DC723191F88A2193E742624D03E2FD723D50B66C2E004025","parts":{"hash":"1C12A6FEBAB55AF424276F826CFA4402C3836C19DB3F4B7F1324DFE8EFEA8CE9","total":8}},"height":3490889,"pol_round":-1,"round":0,"signature":"jOxegFZeSQpCc+lzlsUfNhioiN8YylhH1i6KBEgXuees2RJEbTRg5A6R2rfVkOZJ9/5xHtupl+RYRSOgiQUyAg==","timestamp":"2022-03-07T09:00:25.754686549Z"}
 4:00AM INF received complete proposal block hash=28ACA34439C4F038DC723191F88A2193E742624D03E2FD723D50B66C2E004025 height=3490889 module=consensus
 4:00AM ERR prevote step: ProposalBlock is invalid err="wrong Block.Header.AppHash.  Expected 0C8D390CA892739515F278213EC140EE53680B9CA5B55082FF6C9228A8F4B7FC, got BF09B47EB54FC09EA9456863ABE0EB3DB52FB929050D8ADD5E77A8E718E3740D" height=3490889 module=consensus round=0
 4:00AM INF client state updated client-id=07-tendermint-1457 height=1-2178768 module=x/ibc/client
 4:00AM INF packet received module=x/ibc/channel packet="{141496 transfer channel-0 transfer channel-42 [123 34 97 109 111 117 110 116 34 58 34 52 48 52 56 48 48 48 34 44 34 100 101 110 111 109 34 58 34 117 106 117 110 111 34 44 34 114 101 99 101 105 118 101 114 34 58 34 111 115 109 111 49 110 107 50 104 55 57 54 110 121 116 103 54 107 117 48 51 48 116 113 120 103 110 110 121 121 104 121 51 122 48 54 107 108 115 118 114 100 100 34 44 34 115 101 110 100 101 114 34 58 34 106 117 110 111 49 110 107 50 104 55 57 54 110 121 116 103 54 107 117 48 51 48 116 113 120 103 110 110 121 121 104 121 51 122 48 54 107 112 101 117 103 117 114 34 125] 1-3491028 0}"
 4:00AM INF minted coins from module account amount=4048000ibc/46B44899322F3CD854D2D46DEEF881958467CDD4B3B10086DA49296BBED94BED from=transfer module=x/bank
 4:00AM INF acknowledged written module=x/ibc/channel packet="{141496 transfer channel-0 transfer channel-42 [123 34 97 109 111 117 110 116 34 58 34 52 48 52 56 48 48 48 34 44 34 100 101 110 111 109 34 58 34 117 106 117 110 111 34 44 34 114 101 99 101 105 118 101 114 34 58 34 111 115 109 111 49 110 107 50 104 55 57 54 110 121 116 103 54 107 117 48 51 48 116 113 120 103 110 110 121 121 104 121 51 122 48 54 107 108 115 118 114 100 100 34 44 34 115 101 110 100 101 114 34 58 34 106 117 110 111 49 110 107 50 104 55 57 54 110 121 116 103 54 107 117 48 51 48 116 113 120 103 110 110 121 121 104 121 51 122 48 54 107 112 101 117 103 117 114 34 125] 1-3491028 0}"
 4:00AM ERR CONSENSUS FAILURE!!! err="precommit step; +2/3 prevoted for an invalid block: wrong Block.Header.AppHash.  Expected 0C8D390CA892739515F278213EC140EE53680B9CA5B55082FF6C9228A8F4B7FC, got BF09B47EB54FC09EA9456863ABE0EB3DB52FB929050D8ADD5E77A8E718E3740D" module=consensus stack="goroutine 7992376 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x65\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2()\n\tgithub.com/tendermint/tendermint@v0.34.15/consensus/state.go:726 +0x4c\npanic({0x1b0c920, 0xc20e2b16f0})\n\truntime/panic.go:1038 +0x215\ngithub.com/tendermint/tendermint/consensus.(*State).enterPrecommit(0xc0ef26aa80, 0x354449, 0x0)\n\tgithub.com/tendermint/tendermint@v0.34.15/consensus/state.go:1390 +0x171f\ngithub.com/tendermint/tendermint/consensus.(*State).addVote(0xc0ef26aa80, 0xc191f23540, {0xc0f171ad50, 0x28})\n\tgithub.com/tendermint/tendermint@v0.34.15/consensus/state.go:2104 +0x188f\ngithub.com/tendermint/tendermint/consensus.(*State).tryAddVote(0xc0ef26aa80, 0xc191f23540, {0xc0f171ad50, 0xc188c2ef00})\n\tgithub.com/tendermint/tendermint@v0.34.15/consensus/state.go:1930 +0x2c\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc0ef26aa80, {{0x2773300, 0xc27a906cc0}, {0xc0f171ad50, 0x0}})\n\tgithub.com/tendermint/tendermint@v0.34.15/consensus/state.go:838 +0x40b\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc0ef26aa80, 0x0)\n\tgithub.com/tendermint/tendermint@v0.34.15/consensus/state.go:762 +0x419\ncreated by github.com/tendermint/tendermint/consensus.(*State).OnStart\n\tgithub.com/tendermint/tendermint@v0.34.15/consensus/state.go:378 +0x12f\n"
 4:00AM INF Stopping baseWAL service impl={"Logger":{}} module=consensus wal=/home/osmosisd/.osmosisd/data/cs.wal/wal
 4:00AM INF Stopping Group service impl={"Dir":"/home/osmosisd/.osmosisd/data/cs.wal","Head":{"ID":"PGqUB6bhWPob:/home/osmosisd/.osmosisd/data/cs.wal/wal","Path":"/home/osmosisd/.osmosisd/data/cs.wal/wal"},"ID":"group:PGqUB6bhWPob:/home/osmosisd/.osmosisd/data/cs.wal/wal","Logger":{}} module=consensus wal=/home/osmosisd/.osmosisd/data/cs.wal/wal
 4:00AM INF Timed out dur=3000 height=3490889 module=consensus round=0 step=3
 4:00AM INF Saving AddrBook to file book=/home/osmosisd/.osmosisd/config/addrbook.json module=p2p size=1335
 4:00AM INF Ensure peers module=pex numDialing=0 numInPeers=0 numOutPeers=50 numToDial=0
 4:00AM ERR Connection failed @ sendRoutine conn={"Logger":{}} err="pong timeout" module=p2p peer={"id":"49b307f9e17f2ab3edb1c958f902b8857b2b4ce3","ip":"135.148.55.229","port":26656}
 4:00AM INF Stopping MConnection service impl={"Logger":{}} module=p2p peer={"id":"49b307f9e17f2ab3edb1c958f902b8857b2b4ce3","ip":"135.148.55.229","port":26656}
 4:00AM ERR Stopping peer for error err="pong timeout" module=p2p peer={"Data":{},"Logger":{}}
 4:00AM INF Stopping Peer service impl={"Data":{},"Logger":{}} module=p2p peer={"id":"49b307f9e17f2ab3edb1c958f902b8857b2b4ce3","ip":"135.148.55.229","port":26656}
 4:00AM INF Stopping gossipVotesRoutine for peer module=consensus peer={"Data":{},"Logger":{}}
 4:00AM INF Stopping gossipDataRoutine for peer module=consensus peer={"Data":{},"Logger":{}}
 4:00AM INF Stopping queryMaj23Routine for peer module=consensus peer={"Data":{},"Logger":{}}
 4:01AM ERR Connection failed @ sendRoutine conn={"Logger":{}} err="pong timeout" module=p2p peer={"id":"2048e1bc1f020fa210fb475e7a0ec0948919609f","ip":"185.217.125.64","port":26656}
 4:01AM INF Stopping MConnection service impl={"Logger":{}} module=p2p peer={"id":"2048e1bc1f020fa210fb475e7a0ec0948919609f","ip":"185.217.125.64","port":26656}

Relevant Slack Thread:

p0mvn commented 2 years ago

Found a similar issue in cosmos-sdk with some suggestions on how to debug: https://github.com/cosmos/cosmos-sdk/issues/4254#issuecomment-634225379

p0mvn commented 2 years ago

Report 2

also a replay error on 7.0.3 
panic: Failed to process committed block (3365665:39D4714BC51BBCA30DE4F15124DED6E842F23D5466E43C4989B3639C8D3C4531): wrong Block.Header.AppHash.  Expected DC55D74FA7CBC774903E8E408F9C14CEAD175A53225BD99E096C3D68272130B9, got 744885FFB2185AD5DBAA8E70A2F8CF94ADB5AD40A30828E720BC716D469461B2 (edited)

Discord link: https://discordapp.com/channels/798583171548840026/842550346865180682/949734501385379850

p0mvn commented 2 years ago

Since Report 1 indicated that this happened on a relayer, I did some digging to find that https://github.com/cosmos/relayer README says:

The relayer relies on old headers and proofs constructed at past block heights to facilitate correct [IBC](https://ibcprotocol.org/) behavior. For this reason, connected full nodes may prune old blocks once they have passed the unbonding period of the chain but not before. Not pruning at all is not necessary for a fully functional relayer, however, pruning everything will lead to many issues!

It suggests around 40k pruning-keep-recent. However, validators reported on Discord that this is only needed to replay in case of a failure and can be mitigated by running an archive node.

So not having 40k pruning-keep-recent shouldn't be a problem. I will still leave this note here just in case

p0mvn commented 2 years ago

Report 3

panic: Failed to process committed block (3595581:8F1AD89E9EB106EDA5A77C8B9E144ED9BFCEB87C17F31528AC7ACD019F6A3508): wrong Block.Header.AppHash.  Expected 6BF7972E29983CDCDC89240A3EECFFB13F6BB9B7F86155578713555C0FDEC972, got B3D6645772AE3491470866434E814AAD98DE1B67ED4BD3B0AE75A35665DB54F5
I have a main-node that works fine with the same config files and is perfectly stable on the same version... Could this be related to the fact that I'm using the problematic node for ibc-relayer ? first I got this error with the default parameter. the second time already with the settings
 pruning = "custom"

# These are applied if and only if the pruning strategy is custom.
pruning-keep-recent = "10000"
pruning-keep-every = "0"
pruning-interval = "40"

Discord link: https://discordapp.com/channels/798583171548840026/842550346865180682/953176901294366800

p0mvn commented 2 years ago

Report 4

I'm having some issues getting my backup node caught up. I've tried doing osmosisd unsafe-reset-all and downloading an updated quicksync, but I keep getting Error: error during handshake: error on replay: wrong Block.Header.AppHash.  Expected 484696EA65300CC860DBCBE7E79E6B1490F8FA9CC7516985023E56D0DF97DF91, got 12C070B6EF18ED0C6B20CA5FF5EAA86A8D520C533DBBBE05AB86DEA438F8A11B Mar 17 22:06:17  cosmovisor[2266811]: 10:06PM ERR  error="exit status 1" module=cosmovisor

Discord link: https://discordapp.com/channels/798583171548840026/842550346865180682/954245585546911794

p0mvn commented 2 years ago

Analysis of Report 1:

This is first happening during prevote, and then gets fatal consensus failure in precommit: image

It is failing because +2/3 of validators prevoted for the block. However, our node detects both in prevote and precommit that the block is invalid. Therefore, it should not have been prevoted or precommited for

From the logs:

Expected 0C8D390CA892739515F278213EC140EE53680B9CA5B55082FF6C9228A8F4B7FC, got BF09B47EB54FC09EA9456863ABE0EB3DB52FB929050D8ADD5E77A8E718E3740D"

Where

Based on the above, I can think of the following cases:

Case 1

Case 2

UPDATE 1:

I queried height 3490889 on the archive node. The last committed app hash that is being displayed:

"app_hash":"BF09B47EB54FC09EA9456863ABE0EB3DB52FB929050D8ADD5E77A8E718E3740D",`

Therefore, it is case 2. We can see in the log that the node committed incorrect state.

Also, verified previous heights (3490888, 3490887) against the logs from Report 1. Everything looks as expected. Attaching log files for each height here: 3490887.log 3490888.log 3490889.log

p0mvn commented 2 years ago

Found a similar issue in cosmos-sdk with some suggestions on how to debug: cosmos/cosmos-sdk#4254 (comment)

Since in other reports, this bug is happening during a replay, I am wondering if the problem is case 3) from the linked cosmos-sdk comment. It says that a node might get this error if it is connected to a faulty node

p0mvn commented 2 years ago

Tagging potential unreleased fixes:

IAVL:

SDK:

Osmosis: https://github.com/osmosis-labs/osmosis/pull/1154

p0mvn commented 2 years ago

The above fixes are released, waiting for user feedback

p0mvn commented 2 years ago

Separated the most important details from the latest app hash log (the original was 1.5GB): https://drive.google.com/file/d/1w-fIH-lssZTWDAn5sXuhC6so7tatNGtH/view?usp=sharing

TODOs:

p0mvn commented 2 years ago

There was another log shared with us: https://drive.google.com/file/d/1Uh_hY-mbc1UAiad6NyFGL0ghBdnffqC3/view?usp=sharing

It says wrong Block.Header.LastResultsHash. This hash comes from the hash of DeliverTx(...) result hashes. It looks like this time a similar issue occurred not during Commit().

These are the transactions that could have caused the error. https://atomscan.com/osmosis/blocks/3879705

p0mvn commented 2 years ago

Separated the most important details from the latest app hash log (the original was 1.5GB): https://drive.google.com/file/d/1w-fIH-lssZTWDAn5sXuhC6so7tatNGtH/view?usp=sharing

TODOs:

  • [x] query archive node to see which store happens to cause app hash mismatch
  • [x] inspect code for more data races

With regards to these logs, the store that committed wrong data was bank. I verified this by comparing the logs to the actual data.

The logs:

 1:09PM INF committed KVStore commit_store_hash=97499994BFCEE4E8C43E544E840E305C8494B02CC20E83B330B0A1CCFBE38C8E height=3829411 key=bank
 1:09PM INF calculated commit hash commit_hash=15FA048896B62972C6D0A4079999D9F2FF8A0700D97AD41FA1A35B16F2F0C52C height=3829411 keys=["acc","authz","bank","bech32ibc","capability","claim","distribution","epochs","evidence","gamm","gov","ibc","incentives","lockup","mem_capability","mint","params","poolincentives","slashing","staking","superfluid","transfer","txfees","upgrade","wasm"]

Actual:

commit_hash=6223A32BBAEFCCC618A304A64B7F6726CA4F4B5C68A8E9C042CE8A63A09E1C0D
bank=88AB92B3CD444CF2F2BB02885CF6C40789AFFF1E52572B1B372AED7A91474D5B

All other stores had correct commit_store_hash. Therefore, the incorrect bank store commit_store_hash caused the commit_hash to be wrong.

However, since we observed wrong Block.Header.LastResultsHash which happens during DeliverTx() while wrong Block.Header.AppHash is caused by a faulty commit, it is likely that a data race/bug is still present in IAVL and is not specific to Commit().

I also don't think that the error is specific to the bank module only. It is likely possible for it to happen in other stores. However, there is no proof of that yet. We will need more data to make a conclusion.

Currently, there is still no way to reproduce this. From the discussion with the team, we will focus on our e2e test and hope to expose this problem there. At the same time, we will wait for more logs from node operators to determine if the issue is specific to the bank module or not.

alexanderbez commented 2 years ago

However, since we observed wrong Block.Header.LastResultsHash which happens during DeliverTx() while wrong Block.Header.AppHash is caused by a faulty commit, it is likely that a data race/bug is still present in IAVL and is not specific to Commit().

If there is a data race/inconsistency in IAVL, then yes, DeliverTx mis-match makes sense since we write to KVStore state during DeliverTx. Good find!

mircea-c commented 2 years ago

Here are 3 more instances of this issue. The node has the following characteristics:

logs.zip

alexanderbez commented 2 years ago

I feel like the heavy querying could have something to do with the IAVL weirdness

p0mvn commented 2 years ago

I feel like the heavy querying could have something to do with the IAVL weirdness

I addressed some of the problems related to concurrency between commit and queries in v7.1.0. However, I do not see any more data races/issues by just inspecting the code.

In addition to e2e tests with ibc in Osmosis, I'll be adding more unit tests at all layers related to committing and querying at the same time. Hopefully, that helps to expose more data races, if any

p0mvn commented 2 years ago

Here are 3 more instances of this issue. The node has the following characteristics:

  • the database had been gradually increasing in size for about 40 days, reaching 211G, and was halting in this manner 1-2 times daily
  • the node was being queried heavily by relay software and a large number of transactions was being submitted
  • since these incidents, I've reduced the database size via statesync to < 50GB and have not had it halt in 48 hours now

logs.zip

Thanks for all this @mircea-c . I'll inspect the logs

I'm wondering if there could be anything else special about your setup?

I'm asking this because since v7.1.0, we have not heard of app hash from anyone else running a relayer.

mircea-c commented 2 years ago

@p0mvn I'm hoping we don't run some special snowflake of a node :slightly_smiling_face:

The only not as common setup is that these nodes run in docker containers. We have 13 other networks that run in this setup though, and none of them have had an issue as this one so far.

The only configuration options we change from default are pruning which is set to 5000/0/509 and p2p send_rate and recv_rate which are both set to 51200000.

You can have a look at our complete node config here: https://github.com/cephalopodequipment/config/tree/main/cosmos-sdk/0.44.x

faddat commented 2 years ago

@mircea-c are you currently able to state sync?

p0mvn commented 2 years ago

The issue was reported by mintscan running v7.1.0

Use patterns:

mircea-c commented 2 years ago

@faddat have not done statesync yet. Will attempt it soon though.

@p0mvn It also hasn't stopped happening to us on v7.2.1.

p0mvn commented 2 years ago

@mircea-c sorry to hear that. We are working on our e2e tests to expose this problem still. Unfortunately, we don't have a better way to approach this at the moment since we can't deterministically reproduce this.

However, we just pre-released v7.3.0 that might be related to app hash. Essentially, there were data races related to queries. Since you're running a relayer node with a heavy query load, I think that the fix in v7.3.0 might help. Could you give v7.3.0 a try and let us know how it goes?

mattverse commented 2 years ago

@p0mvn Is this issue still open?

p0mvn commented 2 years ago

It is much less apparent but, as far as I know, it is still present in v8. We would like to keep this open until we have the ability to deterministically reproduce this

faddat commented 1 year ago

It exists in v12 too. 

9:19PM INF service start impl=MConn{136.243.248.186:26656} module=p2p msg={} peer={"id":"c124ce0b508e8b9ed1c5b6957f362225659b5343","ip":"136.243.248.186","port":26656}
9:19PM INF Added peer module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service start impl="Peer{MConn{95.216.76.251:26656} 1c398af2208984d4e59bc41132e3eac0508abb0f out}" module=p2p msg={} peer={"id":"1c398af2208984d4e59bc41132e3eac0508abb0f","ip":"95.216.76.251","port":26656}
9:19PM INF service start impl=MConn{95.216.76.251:26656} module=p2p msg={} peer={"id":"1c398af2208984d4e59bc41132e3eac0508abb0f","ip":"95.216.76.251","port":26656}
9:19PM INF Added peer module=p2p peer={"Data":{},"Logger":{}}
9:19PM ERR Error in validation err="wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=blockchain
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"f50a2ebf44d19c4816b5455f48b9cc671906ff1c","ip":"23.88.0.170","port":15609}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:50458->23.88.0.170:15609: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"f50a2ebf44d19c4816b5455f48b9cc671906ff1c","ip":"23.88.0.170","port":15609}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"ebc272824924ea1a27ea3183dd0b9ba713494f83","ip":"185.16.39.137","port":26716}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:37436->185.16.39.137:26716: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"ebc272824924ea1a27ea3183dd0b9ba713494f83","ip":"185.16.39.137","port":26716}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM ERR Error in validation err="wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=blockchain
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"a72323512ddedf580affb0e0ba0bb32218ae8e6d","ip":"34.105.148.8","port":26656}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:51264->34.105.148.8:26656: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"a72323512ddedf580affb0e0ba0bb32218ae8e6d","ip":"34.105.148.8","port":26656}
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"33cf290cc0cfec8c59e6af86f1a5579303d21087","ip":"138.68.14.64","port":26656}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:47578->138.68.14.64:26656: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"33cf290cc0cfec8c59e6af86f1a5579303d21087","ip":"138.68.14.64","port":26656}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM ERR SendTimeout module=blockchain peer=259ab883ee76f92e82f8f14d463aaaa09d857fb9 reason="peer did not send us anything" timeout=15000
9:19PM ERR Stopping peer for error err="error with peer 259ab883ee76f92e82f8f14d463aaaa09d857fb9: peer did not send us anything" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"259ab883ee76f92e82f8f14d463aaaa09d857fb9","ip":"144.76.70.108","port":9010}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:41514->144.76.70.108:9010: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"259ab883ee76f92e82f8f14d463aaaa09d857fb9","ip":"144.76.70.108","port":9010}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM ERR Error in validation err="wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=blockchain
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"a5edb41ef3ec40d09bc59a62f4337fc572971ab2","ip":"89.149.218.47","port":26656}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:39220->89.149.218.47:26656: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"a5edb41ef3ec40d09bc59a62f4337fc572971ab2","ip":"89.149.218.47","port":26656}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"32e9d4a7413dd5393c8be004bee68dea683be839","ip":"65.21.227.95","port":2004}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:59610->65.21.227.95:2004: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"32e9d4a7413dd5393c8be004bee68dea683be839","ip":"65.21.227.95","port":2004}
9:19PM ERR Error in validation err="wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=blockchain
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"d87b23a8f9134744f2370b069531fcf62e7721c9","ip":"65.109.30.119","port":26656}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:44536->65.109.30.119:26656: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"d87b23a8f9134744f2370b069531fcf62e7721c9","ip":"65.109.30.119","port":26656}
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"8e72d0b37a9dc16ea58c0da705caa6530badd6ce","ip":"138.197.68.193","port":26656}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:54316->138.197.68.193:26656: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"8e72d0b37a9dc16ea58c0da705caa6530badd6ce","ip":"138.197.68.193","port":26656}
9:19PM ERR Error in validation err="wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=blockchain
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"77bb5fb9b6964d6e861e91c1d55cf82b67d838b5","ip":"34.86.74.3","port":26656}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:47910->34.86.74.3:26656: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"77bb5fb9b6964d6e861e91c1d55cf82b67d838b5","ip":"34.86.74.3","port":26656}
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"0757373dfc3cf682f1f3c7abf6d7241a2592a19e","ip":"135.181.141.16","port":56656}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:38610->135.181.141.16:56656: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"0757373dfc3cf682f1f3c7abf6d7241a2592a19e","ip":"135.181.141.16","port":56656}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM ERR Error in validation err="wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=blockchain
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"b69e57cd6f796ac5d6efb1a834163365c37cbfa8","ip":"78.46.69.29","port":26656}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:38134->78.46.69.29:26656: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"b69e57cd6f796ac5d6efb1a834163365c37cbfa8","ip":"78.46.69.29","port":26656}
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"9b1bfb99d9eb04af32510ed8e3eb83c59448662f","ip":"95.214.52.220","port":26656}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:53940->95.214.52.220:26656: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"9b1bfb99d9eb04af32510ed8e3eb83c59448662f","ip":"95.214.52.220","port":26656}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM ERR Error in validation err="wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=blockchain
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"a5d0842d58c0fdd4ed10a39fd9c897cd168906d2","ip":"65.21.195.98","port":26706}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:36068->65.21.195.98:26706: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"a5d0842d58c0fdd4ed10a39fd9c897cd168906d2","ip":"65.21.195.98","port":26706}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"2f5e471d41e03da1f0ae62507ec6280872e493fe","ip":"162.19.21.28","port":26656}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:38442->162.19.21.28:26656: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"2f5e471d41e03da1f0ae62507ec6280872e493fe","ip":"162.19.21.28","port":26656}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM ERR Error in validation err="wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=blockchain
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"71f2451869d7363ce5d91366143de63069641303","ip":"65.108.71.166","port":33656}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:47432->65.108.71.166:33656: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"71f2451869d7363ce5d91366143de63069641303","ip":"65.108.71.166","port":33656}
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"3fe9dad254a0544b1d4bd72add18c24258c0e2da","ip":"165.232.178.40","port":26656}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:49516->165.232.178.40:26656: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"3fe9dad254a0544b1d4bd72add18c24258c0e2da","ip":"165.232.178.40","port":26656}
9:19PM ERR Error in validation err="wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=blockchain
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"3197daa0ee5245b17a546be032ff0f6814e1d1db","ip":"148.251.191.239","port":26656}
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"3197daa0ee5245b17a546be032ff0f6814e1d1db","ip":"148.251.191.239","port":26656}
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"67072788a35b736482c8570badf3e4fd6b60845a","ip":"3.141.198.220","port":26656}
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"67072788a35b736482c8570badf3e4fd6b60845a","ip":"3.141.198.220","port":26656}
9:19PM ERR Error in validation err="wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=blockchain
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"bb618070ed06f152efae2ee9ef0129074bbdc69f","ip":"202.61.201.178","port":2000}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:47788->202.61.201.178:2000: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"bb618070ed06f152efae2ee9ef0129074bbdc69f","ip":"202.61.201.178","port":2000}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"971c324f0889de5fd528402487168d88857a3df6","ip":"66.172.36.141","port":36656}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:52722->66.172.36.141:36656: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"971c324f0889de5fd528402487168d88857a3df6","ip":"66.172.36.141","port":36656}
9:19PM ERR Error in validation err="wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=blockchain
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"76a14f45019da794209dcf65dea3a0775730b78e","ip":"165.227.187.0","port":26656}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:35114->165.227.187.0:26656: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"76a14f45019da794209dcf65dea3a0775730b78e","ip":"165.227.187.0","port":26656}
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"038644cdab5548ab7c9e57784ce324181085d94c","ip":"23.88.67.24","port":10256}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:58976->23.88.67.24:10256: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"038644cdab5548ab7c9e57784ce324181085d94c","ip":"23.88.67.24","port":10256}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM ERR Error in validation err="wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=blockchain
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"77900931e443efd95c7feeb86a13968321f24d9b","ip":"194.163.161.146","port":24650}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:56836->194.163.161.146:24650: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"77900931e443efd95c7feeb86a13968321f24d9b","ip":"194.163.161.146","port":24650}
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"d929af809b87632e83a3e01379792e085a0bdd88","ip":"38.242.232.248","port":26656}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:59702->38.242.232.248:26656: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"d929af809b87632e83a3e01379792e085a0bdd88","ip":"38.242.232.248","port":26656}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM ERR Error in validation err="wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=blockchain
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Logger":{}} module=p2p msg={} peer={"id":"f023313994a3d06c81d767a10571ba9b165c4f91","ip":"65.21.238.147","port":36656}
9:19PM ERR Stopping peer for error err="read tcp 172.16.0.2:49488->65.21.238.147:36656: use of closed network connection" module=p2p peer={"Data":{},"Logger":{}}
9:19PM INF service stop impl={"Data":{},"Logger":{}} module=p2p msg={} peer={"id":"f023313994a3d06c81d767a10571ba9b165c4f91","ip":"65.21.238.147","port":36656}
9:19PM ERR error while stopping peer error="already stopped" module=p2p
9:19PM ERR Stopping peer for error err="blockchainReactor validation error: wrong Block.Header.AppHash.  Expected DF564880C458C4370588F2C5B406EAB81BCA2E9E36BA57BAFF1B21273E948696, got 9031322B023645E45A8A94D2CE7CC8F40DF2678B0725D6BE6FFD8DAA71D11556" module=p2p peer=
p0mvn commented 1 year ago

I don't think we've been running into this issue for the same (IAVL) reasons anymore. Therefore, I'm going to close this. Please let me know if there's anything I'm unaware of