search

vegaprotocol / vega

A Go implementation of the Vega Protocol, a protocol for creating and trading derivatives on a fully decentralised network.
https://vega.xyz
GNU Affero General Public License v3.0
37 stars 19 forks source link

Consensus error observed with 0.71.6 #8552

Closed gordsport closed 1 year ago

gordsport commented 1 year ago

A validator has been running version 0.71.6 and after a few hours has had a consensus error, it affected only the validator node, not the data-node.

2023-06-20T14:26:38.080Z        ERROR   tendermint      consensus/state.go:1280 prevote step: ProposalBlock is invalid  {"err": "wrong Block.Header.LastResultsHash.  Expected 38CB38B5C322AD07534E8CE02B692E67FDE5A98136F2D6984D4BAD952BD44F87, got 8E22A42006AF871B3FD472E39C6299546318316E798AB6EF27BD036820AB0186"}
2023-06-20T14:26:38.276Z        ERROR   tendermint      consensus/state.go:732  CONSENSUS FAILURE!!!    {"err": "precommit step; +2/3 prevoted for an invalid block: wrong Block.Header.LastResultsHash.  Expected 38CB38B5C322AD07534E8CE02B692E67FDE5A98136F2D6984D4BAD952BD44F87, got 8E22A42006AF871B3FD472E39C6299546318316E798AB6EF27BD036820AB0186", "stack": "goroutine 429 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2()\n\t/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/state.go:732 +0x4c\npanic({0x2d3b940, 0xc00c72fab0})\n\t/usr/local/go/src/runtime/panic.go:884 +0x212\ngithub.com/tendermint/tendermint/consensus.(*State).enterPrecommit(0xc00a703c00, 0x3bfa30, 0x0)\n\t/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/state.go:1414 +0x1739\ngithub.com/tendermint/tendermint/consensus.(*State).addVote(0xc00a703c00, 0xc00579cdc0, {0xc006987b30, 0x28})\n\t/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/state.go:2137 +0x13d8\ngithub.com/tendermint/tendermint/consensus.(*State).tryAddVote(0xc00a703c00, 0xc00579cdc0, {0xc006987b30?, 0xc0054ef900?})\n\t/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/state.go:1963 +0x2c\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc00a703c00, {{0x3ca1f60?, 0xc009b153b0?}, {0xc006987b30?, 0x0?}})\n\t/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/state.go:861 +0x170\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine(0xc00a703c00, 0x0)\n\t/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/state.go:768 +0x3f9\ncreated by github.com/tendermint/tendermint/consensus.(*State).OnStart\n\t/go/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/state.go:379 +0x12d\n"}

The logs from the validator can be seen:

daniel1302 commented 1 year ago

Logs before consensus failure:

2023-06-20T14:26:22.061Z        INFO    tendermint      txindex/indexer_service.go:97   indexed block exents    {"height": 3930658}
2023-06-20T14:26:22.228Z        INFO    tendermint      consensus/state.go:1850 received proposal       {"proposal": "Proposal{3930659/0 (B7FEBF785B2047E2BE846F97E3C17DDAD05121C78842C5616A76622544176A85:1:176B9B4384FF, -1) FF45458BA98F @ 2023-06-20T14:26:22.056881001Z}"}
2023-06-20T14:26:22.228Z        INFO    tendermint      consensus/state.go:1917 received complete proposal block        {"height": 3930659, "hash": "B7FEBF785B2047E2BE846F97E3C17DDAD05121C78842C5616A76622544176A85"}
2023-06-20T14:26:22.881Z        INFO    tendermint      consensus/state.go:1604 finalizing commit of block      {"hash": "B7FEBF785B2047E2BE846F97E3C17DDAD05121C78842C5616A76622544176A85", "root": "C9862A7E394A6FAFE1733A659B89DB6881BE8F6BBA95B5BC0D686E9CD8D5BF92", "num_txs": 5}
2023-06-20T14:26:22.893Z        INFO    tendermint      state/execution.go:333  executed block  {"height": 3930659, "num_valid_txs": 5, "num_invalid_txs": 0}
2023-06-20T14:26:22.897Z        INFO    tendermint      state/execution.go:235  committed state {"height": 3930659, "num_txs": 5, "app_hash": "94BE2EBD16DF046AF550D43C6F60510C093BA842C15A00D7833DD4C6038A8928"}
2023-06-20T14:26:22.900Z        INFO    tendermint      consensus/ticker.go:124 Timed out       {"dur": "-19.065593ms", "height": 3930660, "round": 0, "step": "RoundStepNewHeight"}
2023-06-20T14:26:22.901Z        INFO    tendermint      txindex/indexer_service.go:97   indexed block exents    {"height": 3930659}
2023-06-20T14:26:23.174Z        INFO    tendermint      consensus/state.go:1850 received proposal       {"proposal": "Proposal{3930660/0 (25A6566369AC5BFC5AAB2708F7FDDEF4B40334B20E9A7FAB519F92FD78943694:1:5C1AA02F4325, -1) 55684A17F084 @ 2023-06-20T14:26:22.937177089Z}"}
2023-06-20T14:26:23.174Z        INFO    tendermint      consensus/state.go:1917 received complete proposal block        {"height": 3930660, "hash": "25A6566369AC5BFC5AAB2708F7FDDEF4B40334B20E9A7FAB519F92FD78943694"}
2023-06-20T14:26:23.689Z        INFO    tendermint      consensus/state.go:1604 finalizing commit of block      {"hash": "25A6566369AC5BFC5AAB2708F7FDDEF4B40334B20E9A7FAB519F92FD78943694", "root": "94BE2EBD16DF046AF550D43C6F60510C093BA842C15A00D7833DD4C6038A8928", "num_txs": 5}
2023-06-20T14:26:23.699Z        INFO    tendermint      state/execution.go:333  executed block  {"height": 3930660, "num_valid_txs": 5, "num_invalid_txs": 0}
2023-06-20T14:26:23.703Z        INFO    tendermint      state/execution.go:235  committed state {"height": 3930660, "num_txs": 5, "app_hash": "256395BFFA13614FB96E597B6ECCD420716D0D2F27802FEFA54B3AE015479E49"}
2023-06-20T14:26:23.705Z        INFO    tendermint      consensus/ticker.go:124 Timed out       {"dur": "-16.920302ms", "height": 3930661, "round": 0, "step": "RoundStepNewHeight"}
2023-06-20T14:26:23.706Z        INFO    tendermint      txindex/indexer_service.go:97   indexed block exents    {"height": 3930660}
2023-06-20T14:26:24.182Z        INFO    tendermint      consensus/state.go:1850 received proposal       {"proposal": "Proposal{3930661/0 (A81BBBFC107009CCBD9702BF638DEDA181CE44DCBA7D558C47DA3FAB066A2E6D:1:F34F90EF21BD, -1) 2C38548D5724 @ 2023-06-20T14:26:23.918896663Z}"}
2023-06-20T14:26:24.182Z        INFO    tendermint      consensus/state.go:1917 received complete proposal block        {"height": 3930661, "hash": "A81BBBFC107009CCBD9702BF638DEDA181CE44DCBA7D558C47DA3FAB066A2E6D"}
2023-06-20T14:26:24.705Z        INFO    tendermint      consensus/state.go:1604 finalizing commit of block      {"hash": "A81BBBFC107009CCBD9702BF638DEDA181CE44DCBA7D558C47DA3FAB066A2E6D", "root": "256395BFFA13614FB96E597B6ECCD420716D0D2F27802FEFA54B3AE015479E49", "num_txs": 7}
2023-06-20T14:26:24.716Z        INFO    tendermint      state/execution.go:333  executed block  {"height": 3930661, "num_valid_txs": 7, "num_invalid_txs": 0}
2023-06-20T14:26:24.720Z        INFO    tendermint      state/execution.go:235  committed state {"height": 3930661, "num_txs": 7, "app_hash": "B5AC32F61013F64F660C938334BE6493F3A9B60661A6CD8CDB9AA706E87CC7DC"}
2023-06-20T14:26:24.723Z        INFO    tendermint      consensus/ticker.go:124 Timed out       {"dur": "-18.023756ms", "height": 3930662, "round": 0, "step": "RoundStepNewHeight"}
2023-06-20T14:26:24.724Z        INFO    tendermint      txindex/indexer_service.go:97   indexed block exents    {"height": 3930661}
2023-06-20T14:26:24.922Z        INFO    tendermint      consensus/state.go:1850 received proposal       {"proposal": "Proposal{3930662/0 (84580C30D32D09BADC4F95311246233870DA39283D03DEF38705683D823340C8:1:242B13AC2F73, -1) ED5EC37E1B0A @ 2023-06-20T14:26:24.672748723Z}"}
2023-06-20T14:26:24.923Z        INFO    tendermint      consensus/state.go:1917 received complete proposal block        {"height": 3930662, "hash": "84580C30D32D09BADC4F95311246233870DA39283D03DEF38705683D823340C8"}
2023-06-20T14:26:25.348Z        INFO    tendermint      consensus/state.go:1604 finalizing commit of block      {"hash": "84580C30D32D09BADC4F95311246233870DA39283D03DEF38705683D823340C8", "root": "B5AC32F61013F64F660C938334BE6493F3A9B60661A6CD8CDB9AA706E87CC7DC", "num_txs": 6}
2023-06-20T14:26:25.357Z        INFO    tendermint      state/execution.go:333  executed block  {"height": 3930662, "num_valid_txs": 4, "num_invalid_txs": 2}
2023-06-20T14:26:25.362Z        INFO    tendermint      state/execution.go:235  committed state {"height": 3930662, "num_txs": 6, "app_hash": "C4D4FAFB77585D2D8637B264A21EC71B526697BBA6C6350AE8030447AFA15ACD"}
2023-06-20T14:26:25.364Z        INFO    tendermint      consensus/ticker.go:124 Timed out       {"dur": "-16.578439ms", "height": 3930663, "round": 0, "step": "RoundStepNewHeight"}
2023-06-20T14:26:25.365Z        INFO    tendermint      txindex/indexer_service.go:97   indexed block exents    {"height": 3930662}
2023-06-20T14:26:25.897Z        INFO    tendermint      consensus/state.go:1850 received proposal       {"proposal": "Proposal{3930663/0 (04A534E2054B7AD4EE1F3E6D43B05A7C9B965BD22BD4453D2302512F16CA7F05:1:5738A98F2A03, -1) 58C3108C9A9C @ 2023-06-20T14:26:25.507282607Z}"}
2023-06-20T14:26:25.898Z        INFO    tendermint      consensus/state.go:1917 received complete proposal block        {"height": 3930663, "hash": "04A534E2054B7AD4EE1F3E6D43B05A7C9B965BD22BD4453D2302512F16CA7F05"}
2023-06-20T14:26:26.344Z        INFO    tendermint      consensus/state.go:1604 finalizing commit of block      {"hash": "04A534E2054B7AD4EE1F3E6D43B05A7C9B965BD22BD4453D2302512F16CA7F05", "root": "C4D4FAFB77585D2D8637B264A21EC71B526697BBA6C6350AE8030447AFA15ACD", "num_txs": 8}
2023-06-20T14:26:26.354Z        INFO    tendermint      state/execution.go:333  executed block  {"height": 3930663, "num_valid_txs": 8, "num_invalid_txs": 0}
2023-06-20T14:26:26.357Z        INFO    tendermint      state/execution.go:235  committed state {"height": 3930663, "num_txs": 8, "app_hash": "58BA37C982A170A5901B33A9AB5377ADA7C726FC96ED260AFBE1FCF7CFCCAD28"}
2023-06-20T14:26:26.358Z        INFO    tendermint      consensus/ticker.go:124 Timed out       {"dur": "-14.728097ms", "height": 3930664, "round": 0, "step": "RoundStepNewHeight"}
2023-06-20T14:26:26.359Z        INFO    tendermint      txindex/indexer_service.go:97   indexed block exents    {"height": 3930663}
2023-06-20T14:26:26.633Z        INFO    tendermint      consensus/state.go:1850 received proposal       {"proposal": "Proposal{3930664/0 (194FFEA4AD3C56D751984D84C8F7E76A609768AE6BF12E12FD895AEC819554C2:1:F41D30198888, -1) 79CCB9CC180F @ 2023-06-20T14:26:26.413321114Z}"}
2023-06-20T14:26:26.633Z        INFO    tendermint      consensus/state.go:1917 received complete proposal block        {"height": 3930664, "hash": "194FFEA4AD3C56D751984D84C8F7E76A609768AE6BF12E12FD895AEC819554C2"}
2023-06-20T14:26:27.171Z        INFO    tendermint      consensus/state.go:1604 finalizing commit of block      {"hash": "194FFEA4AD3C56D751984D84C8F7E76A609768AE6BF12E12FD895AEC819554C2", "root": "58BA37C982A170A5901B33A9AB5377ADA7C726FC96ED260AFBE1FCF7CFCCAD28", "num_txs": 10}
2023-06-20T14:26:27.185Z        INFO    tendermint      state/execution.go:333  executed block  {"height": 3930664, "num_valid_txs": 8, "num_invalid_txs": 2}
2023-06-20T14:26:27.190Z        INFO    tendermint      state/execution.go:235  committed state {"height": 3930664, "num_txs": 10, "app_hash": "8C14E8AF10813ECA2D166E6D0CF0F9EEB5F71B2037408F4C984C419F6134B71A"}
2023-06-20T14:26:27.192Z        INFO    tendermint      consensus/ticker.go:124 Timed out       {"dur": "-21.119794ms", "height": 3930665, "round": 0, "step": "RoundStepNewHeight"}
2023-06-20T14:26:27.193Z        INFO    tendermint      txindex/indexer_service.go:97   indexed block exents    {"height": 3930664}
2023-06-20T14:26:27.422Z        INFO    tendermint      consensus/state.go:1850 received proposal       {"proposal": "Proposal{3930665/0 (3C02204C442948670C235AAAC98815EE2FD460EEA7514F66C4172E7825B78B26:1:93C06B280E68, -1) 14E89EE48772 @ 2023-06-20T14:26:27.164218507Z}"}
2023-06-20T14:26:27.422Z        INFO    tendermint      consensus/state.go:1917 received complete proposal block        {"height": 3930665, "hash": "3C02204C442948670C235AAAC98815EE2FD460EEA7514F66C4172E7825B78B26"}
2023-06-20T14:26:27.856Z        INFO    tendermint      consensus/state.go:1604 finalizing commit of block      {"hash": "3C02204C442948670C235AAAC98815EE2FD460EEA7514F66C4172E7825B78B26", "root": "8C14E8AF10813ECA2D166E6D0CF0F9EEB5F71B2037408F4C984C419F6134B71A", "num_txs": 3}
2023-06-20T14:26:27.867Z        INFO    tendermint      state/execution.go:333  executed block  {"height": 3930665, "num_valid_txs": 3, "num_invalid_txs": 0}
2023-06-20T14:26:27.871Z        INFO    tendermint      state/execution.go:235  committed state {"height": 3930665, "num_txs": 3, "app_hash": "A1F4C4DB34380D047A08F9A33AF86F69D814A736D8C9E1A5662A74E85D4BD715"}
2023-06-20T14:26:27.873Z        INFO    tendermint      consensus/ticker.go:124 Timed out       {"dur": "-18.846826ms", "height": 3930666, "round": 0, "step": "RoundStepNewHeight"}
2023-06-20T14:26:27.874Z        INFO    tendermint      txindex/indexer_service.go:97   indexed block exents    {"height": 3930665}
2023-06-20T14:26:28.288Z        INFO    tendermint      consensus/state.go:1850 received proposal       {"proposal": "Proposal{3930666/0 (1AF8D0A60BE744AAA0E855253B524E0106E920D83254969467E8D25BEE6794DE:1:CE2A76FC4746, -1) B7F695B0080F @ 2023-06-20T14:26:27.937386741Z}"}
2023-06-20T14:26:28.289Z        INFO    tendermint      consensus/state.go:1917 received complete proposal block        {"height": 3930666, "hash": "1AF8D0A60BE744AAA0E855253B524E0106E920D83254969467E8D25BEE6794DE"}
2023-06-20T14:26:28.709Z        INFO    tendermint      consensus/state.go:1604 finalizing commit of block      {"hash": "1AF8D0A60BE744AAA0E855253B524E0106E920D83254969467E8D25BEE6794DE", "root": "A1F4C4DB34380D047A08F9A33AF86F69D814A736D8C9E1A5662A74E85D4BD715", "num_txs": 8}
2023-06-20T14:26:28.719Z        INFO    tendermint      state/execution.go:333  executed block  {"height": 3930666, "num_valid_txs": 8, "num_invalid_txs": 0}
2023-06-20T14:26:28.723Z        INFO    tendermint      state/execution.go:235  committed state {"height": 3930666, "num_txs": 8, "app_hash": "B1B60CFDA384619AF2B38A4CE56529FE8090CDED6B3B6212F2DDF2F41FB4E6D8"}
2023-06-20T14:26:28.726Z        INFO    tendermint      consensus/ticker.go:124 Timed out       {"dur": "-16.772194ms", "height": 3930667, "round": 0, "step": "RoundStepNewHeight"}
2023-06-20T14:26:28.727Z        INFO    tendermint      txindex/indexer_service.go:97   indexed block exents    {"height": 3930666}
2023-06-20T14:26:28.730Z        INFO    tendermint      consensus/state.go:1850 received proposal       {"proposal": "Proposal{3930667/0 (E843E6FD189ACDEDBCD93DF2C5608E561B87DFF0237CF18AC71655885EF353A1:1:5EE275633587, -1) 7508AF10DF9C @ 2023-06-20T14:26:28.72802947Z}"}
2023-06-20T14:26:28.731Z        INFO    tendermint      consensus/state.go:1917 received complete proposal block        {"height": 3930667, "hash": "E843E6FD189ACDEDBCD93DF2C5608E561B87DFF0237CF18AC71655885EF353A1"}
2023-06-20T14:26:29.139Z        INFO    tendermint      pex/pex_reactor.go:477  Ensure peers    {"numOutPeers": 10, "numInPeers": 0, "numDialing": 0, "numToDial": 0}
2023-06-20T14:26:29.568Z        INFO    tendermint      consensus/state.go:1604 finalizing commit of block      {"hash": "E843E6FD189ACDEDBCD93DF2C5608E561B87DFF0237CF18AC71655885EF353A1", "root": "B1B60CFDA384619AF2B38A4CE56529FE8090CDED6B3B6212F2DDF2F41FB4E6D8", "num_txs": 2}
2023-06-20T14:26:29.577Z        INFO    tendermint      state/execution.go:333  executed block  {"height": 3930667, "num_valid_txs": 2, "num_invalid_txs": 0}
2023-06-20T14:26:29.582Z        INFO    tendermint      state/execution.go:235  committed state {"height": 3930667, "num_txs": 2, "app_hash": "CFD1C7FBE7073999DDF5493F573933DCC0C22696B60BB99289B6783C04A32DAC"}
2023-06-20T14:26:29.584Z        INFO    tendermint      consensus/ticker.go:124 Timed out       {"dur": "-18.118233ms", "height": 3930668, "round": 0, "step": "RoundStepNewHeight"}
2023-06-20T14:26:29.585Z        INFO    tendermint      txindex/indexer_service.go:97   indexed block exents    {"height": 3930667}
2023-06-20T14:26:29.940Z        INFO    tendermint      consensus/state.go:1850 received proposal       {"proposal": "Proposal{3930668/0 (35B403CBD5AA86958A3595F6F6C6B71433FEC417A6D21D18E1E3249BBC45D5B6:1:3FC1DC443048, -1) 2D5106954FB5 @ 2023-06-20T14:26:29.562767167Z}"}
2023-06-20T14:26:29.940Z        INFO    tendermint      consensus/state.go:1917 received complete proposal block        {"height": 3930668, "hash": "35B403CBD5AA86958A3595F6F6C6B71433FEC417A6D21D18E1E3249BBC45D5B6"}
2023-06-20T14:26:30.393Z        INFO    tendermint      consensus/state.go:1604 finalizing commit of block      {"hash": "35B403CBD5AA86958A3595F6F6C6B71433FEC417A6D21D18E1E3249BBC45D5B6", "root": "CFD1C7FBE7073999DDF5493F573933DCC0C22696B60BB99289B6783C04A32DAC", "num_txs": 4}
2023-06-20T14:26:30.406Z        INFO    tendermint      state/execution.go:333  executed block  {"height": 3930668, "num_valid_txs": 4, "num_invalid_txs": 0}
2023-06-20T14:26:30.410Z        INFO    tendermint      state/execution.go:235  committed state {"height": 3930668, "num_txs": 4, "app_hash": "B61C70F8D4E51390A405B015367F766E000F5B74ABD83F04D106BD095C70B6BC"}
2023-06-20T14:26:30.413Z        INFO    tendermint      consensus/ticker.go:124 Timed out       {"dur": "-21.398279ms", "height": 3930669, "round": 0, "step": "RoundStepNewHeight"}
2023-06-20T14:26:30.414Z        INFO    tendermint      txindex/indexer_service.go:97   indexed block exents    {"height": 3930668}
2023-06-20T14:26:30.680Z        INFO    tendermint      consensus/state.go:1850 received proposal       {"proposal": "Proposal{3930669/0 (7F22C7FA458C9C996DC6ACBD30AC060972F403A011E61D90F79B5DD2287367D6:1:215A2CCDA0AA, -1) 61DFDB783C3E @ 2023-06-20T14:26:30.35651045Z}"}
2023-06-20T14:26:30.681Z        INFO    tendermint      consensus/state.go:1917 received complete proposal block        {"height": 3930669, "hash": "7F22C7FA458C9C996DC6ACBD30AC060972F403A011E61D90F79B5DD2287367D6"}
2023-06-20T14:26:31.139Z        INFO    tendermint      consensus/state.go:1604 finalizing commit of block      {"hash": "7F22C7FA458C9C996DC6ACBD30AC060972F403A011E61D90F79B5DD2287367D6", "root": "B61C70F8D4E51390A405B015367F766E000F5B74ABD83F04D106BD095C70B6BC", "num_txs": 4}
2023-06-20T14:26:31.148Z        INFO    tendermint      state/execution.go:333  executed block  {"height": 3930669, "num_valid_txs": 4, "num_invalid_txs": 0}
2023-06-20T14:26:31.151Z        INFO    tendermint      state/execution.go:235  committed state {"height": 3930669, "num_txs": 4, "app_hash": "69D4438C2420339AB919B7228042FF83970F55F728B2ACEBF79E06488FCBEAE6"}
2023-06-20T14:26:31.153Z        INFO    tendermint      consensus/ticker.go:124 Timed out       {"dur": "-14.670619ms", "height": 3930670, "round": 0, "step": "RoundStepNewHeight"}
2023-06-20T14:26:31.154Z        INFO    tendermint      txindex/indexer_service.go:97   indexed block exents    {"height": 3930669}
2023-06-20T14:26:31.430Z        INFO    tendermint      consensus/state.go:1850 received proposal       {"proposal": "Proposal{3930670/0 (A88EEE64A89FED6121C918796AFBEA83E05187DCD27023AD4B44226A1F098D1D:1:4EE7E656B9A3, -1) 077427537631 @ 2023-06-20T14:26:31.221939004Z}"}
2023-06-20T14:26:31.448Z        INFO    tendermint      consensus/state.go:1917 received complete proposal block        {"height": 3930670, "hash": "A88EEE64A89FED6121C918796AFBEA83E05187DCD27023AD4B44226A1F098D1D"}
2023-06-20T14:26:31.882Z        INFO    tendermint      consensus/state.go:1604 finalizing commit of block      {"hash": "A88EEE64A89FED6121C918796AFBEA83E05187DCD27023AD4B44226A1F098D1D", "root": "69D4438C2420339AB919B7228042FF83970F55F728B2ACEBF79E06488FCBEAE6", "num_txs": 12}
2023-06-20T14:26:31.898Z        INFO    tendermint      state/execution.go:333  executed block  {"height": 3930670, "num_valid_txs": 11, "num_invalid_txs": 1}
2023-06-20T14:26:31.902Z        INFO    tendermint      state/execution.go:235  committed state {"height": 3930670, "num_txs": 12, "app_hash": "27281F83D565642024CC43A14C39D44BDE04A1D1BBDAD90EFC74BA35E27959C6"}
2023-06-20T14:26:31.905Z        INFO    tendermint      consensus/ticker.go:124 Timed out       {"dur": "-22.971848ms", "height": 3930671, "round": 0, "step": "RoundStepNewHeight"}
2023-06-20T14:26:31.906Z        INFO    tendermint      txindex/indexer_service.go:97   indexed block exents    {"height": 3930670}
2023-06-20T14:26:34.906Z        INFO    tendermint      consensus/ticker.go:124 Timed out       {"dur": "3s", "height": 3930671, "round": 0, "step": "RoundStepPropose"}
2023-06-20T14:26:36.467Z        INFO    tendermint      consensus/ticker.go:124 Timed out       {"dur": "1s", "height": 3930671, "round": 0, "step": "RoundStepPrecommitWait"}
2023-06-20T14:26:36.983Z        INFO    tendermint      consensus/state.go:1850 received proposal       {"proposal": "Proposal{3930671/1 (F9F3E31B814B4B0445EC1B896BC256A22B478E9210B142A9D9B86254B80D37BF:1:4AAD071A7790, -1) 80F88A2FD50E @ 2023-06-20T14:26:36.517870192Z}"}
2023-06-20T14:26:37.131Z        INFO    tendermint      consensus/state.go:1917 received complete proposal block        {"height": 3930671, "hash": "F9F3E31B814B4B0445EC1B896BC256A22B478E9210B142A9D9B86254B80D37BF"}
2023-06-20T14:26:37.536Z        INFO    tendermint      consensus/state.go:1604 finalizing commit of block      {"hash": "F9F3E31B814B4B0445EC1B896BC256A22B478E9210B142A9D9B86254B80D37BF", "root": "27281F83D565642024CC43A14C39D44BDE04A1D1BBDAD90EFC74BA35E27959C6", "num_txs": 25}
2023-06-20T14:26:37.557Z        INFO    tendermint      state/execution.go:333  executed block  {"height": 3930671, "num_valid_txs": 23, "num_invalid_txs": 2}
2023-06-20T14:26:37.562Z        INFO    tendermint      state/execution.go:235  committed state {"height": 3930671, "num_txs": 25, "app_hash": "784748595B11046F5F08589C5BD95D3BD531F4EF1EE0E640F93212B3F32E3993"}
2023-06-20T14:26:37.564Z        INFO    tendermint      consensus/ticker.go:124 Timed out       {"dur": "-28.352827ms", "height": 3930672, "round": 0, "step": "RoundStepNewHeight"}
2023-06-20T14:26:37.565Z        INFO    tendermint      txindex/indexer_service.go:97   indexed block exents    {"height": 3930671}
2023-06-20T14:26:38.080Z        INFO    tendermint      consensus/state.go:1850 received proposal       {"proposal": "Proposal{3930672/0 (534B3B009E871FBFF148EC35B98369D7A25930DF29719E252B0A2BF9715FE1CF:1:FC7803B23F98, -1) CB77ADC96F3F @ 2023-06-20T14:26:37.656428255Z}"}
2023-06-20T14:26:38.080Z        INFO    tendermint      consensus/state.go:1917 received complete proposal block        {"height": 3930672, "hash": "534B3B009E871FBFF148EC35B98369D7A25930DF29719E252B0A2BF9715FE1CF"}
2023-06-20T14:26:38.080Z        ERROR   tendermint      consensus/state.go:1280 prevote step: ProposalBlock is invalid  {"err": "wrong Block.Header.LastResultsHash.  Expected 38CB38B5C322AD07534E8CE02B692E67FDE5A981
gordsport commented 1 year ago

Closing as wont fix:

There was a pre-release version v0.71.6 created 3 weeks ago, It was removed it today in order to get the security advisory patch in for the protocol upgrade.

The validator has pulled the version from 3 weeks ago and it did not have the latest changes included.

Team actions to take away: