search

vegaprotocol / vega

A Go implementation of the Vega Protocol, a protocol for creating and trading derivatives on a fully decentralised network.
https://vega.xyz
GNU Affero General Public License v3.0
37 stars 19 forks source link

[Bug]: panic on Denet, n07 fails to restart with AppHash mismatch #9251

Closed fkondej closed 10 months ago

fkondej commented 10 months ago

Problem encountered

We restart a random node on Devnet every 1 hour. The problem happened during the start after stopping the node.

Observed behaviour

Panic in logs. I have backed up the node state and log file in: /home/vega/backup_20230831_113500/ directory on n07.devnet1.vega.rocks.

Expected behaviour

The node should start.

Steps to reproduce

1. Submit `xyz` transaction
2. With this payload
3. Query `abc` API
4. Observe the error

Software version

v0.72.1-13240-2e12850d (commit hash: 2e12850d)

Failing test

No response

Jenkins run

No response

Configuration used

No response

Relevant log output

Block: Block{
  Header{
    Version:        {11 1}
    ChainID:        vega-devnet1-202308302128
    Height:         49204
    Time:           2023-08-31 10:18:12.937305476 +0000 UTC
    LastBlockID:    868116409DA94381D66EBF3F7F979643A609FD7254E58EE4230FB6A17D268AF0:2:5A0E9B4A46E2
    LastCommit:     3C429EE90BA6EB327164BB374FBDDDC6C5D642ED0982221F5338848279413C30
    Data:           8AB4AF58E6984B65A92219E286E69FF90B56C63BC3C6BD29F718057570FF6918
    Validators:     8BB7ADC72719B80FBBEEBBCA6CED347C6A2D0376578F12094A6DD66169DC5BAD
    NextValidators: 8BB7ADC72719B80FBBEEBBCA6CED347C6A2D0376578F12094A6DD66169DC5BAD
    App:            2C3B0358B8628A5FF93A6EE7270FFBF12925243A4FC192C9105B99620648C1DC
    Consensus:      C8CD39A92B22597AC33565A59990B4D1E440B15621B5E932D1D5B39A8F45E9DB
    Results:        4A65E03448BA3851993A8002DE0B798584BA81BA38E9D572621914D72993608E
    Evidence:       E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
    Proposer:       EE43F6C7C98B99B9826BD6F1CECD111E9BB0E3E2
  }#038229C0A4EA897787DA2BC00CA930192C97F6E1092409D02E9F4828BA1032A8
  Data{
    EDC1B38676010458CC0C033FC71FE49368F7570667767142BBB6D0B9C46C859A (308 bytes)
    6C5491C53D89D0A20A0CC96587B65EE8D6E4302D3940B2517D07E1FE28B6CCA9 (307 bytes)
    619D244D439D5871AF515B746A6BBB9D9E33102766C102D4EF67312D449625FF (416 bytes)
    FFD814C380C4A6F4A9E09F452C8F1073FE65F0BEF1868D70AB6B7C71CFBAAB4C (415 bytes)
    0E30464629A674ADED0C0A88DA9944CF633388D0FB016D4992AA472DDCFE2469 (417 bytes)
    3E03C2161CE42C6DC7BD5B2C2D44B8DD95981C769C985A090E6835A477467C5A (412 bytes)
    4E22325779A257AFD7D5F784F96D496215169289542709B7F1901B8B9A494391 (418 bytes)
    0E6777714E38B5B8AFE4D124C59746BACE3C1DFF191067B6A9ABEF5680306D9A (417 bytes)
    BE1E2622BBAD91F38A78212118281135C7DF83A4D80061E1E79BD209D8C720E1 (426 bytes)
    46A962C81CFDE4DE372BDD2F2A871954226AB04D1156353E7C87E33234EDC605 (426 bytes)
    25B65544C32E4A61025C9DA4DA3DFC13D73DF558ADCDCDBCD6CF9DBE6005D8AA (424 bytes)
    A621CB3DE57171E1F4491DBF5826751E009EA1799B257DB55BD0BD77DDB0DB50 (426 bytes)
    A3A459A8B9687F93C07E63990AB21A9875E1B6FA19D1671FC22A39B46E0990F0 (424 bytes)
    0CF62BEE7460D4C6FE0AC051EF04DD6332A8EDDA2310D8E1DB05AAD4BFC446A3 (414 bytes)
    7410F94925C2DD12D2D086BF50C040F1C7B21A98D240F07DCA65242286379F37 (416 bytes)
    6200001E9474DFF24F61DE5EE5AE2CA8BC1C43D36538418E187CCBF4F0765E69 (413 bytes)
    82FB3A1DE53A03161C4CCB1D029AC135831F89C9AD0E2FB3C3CC252F1B1FEA92 (417 bytes)
    02D9A5FEA73129FC8DA816CC3897A5F5FE188514AAD32C3171F85146FACB0AB3 (417 bytes)
    A534CDAD9174E48527EEDC6EBB0CD26AAB713B0227EF4113723C3991196441D0 (414 bytes)
    8AC12598F220129EA5B6C8A668D28E090DC5B6ECF43DA5E93491263B25DAEFCC (419 bytes)
    ... (182 total)
  }#8AB4AF58E6984B65A92219E286E69FF90B56C63BC3C6BD29F718057570FF6918
  EvidenceData{

  }#E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
  Commit{
    Height:     49203
    Round:      1
    BlockID:    868116409DA94381D66EBF3F7F979643A609FD7254E58EE4230FB6A17D268AF0:2:5A0E9B4A46E2
    Signatures:
      CommitSig{000000000000 by 000000000000 on 1 @ 0001-01-01T00:00:00Z}
      CommitSig{46A82BEB975B by 9B08B3797CAE on 2 @ 2023-08-31T10:18:13.093601264Z}
      CommitSig{6022F4DECF74 by A8F6E168F587 on 2 @ 2023-08-31T10:18:12.847113923Z}
      CommitSig{46464F2F3E03 by CAB18606029E on 2 @ 2023-08-31T10:18:12.944661849Z}
      CommitSig{9A3E1AD1B311 by EE43F6C7C98B on 2 @ 2023-08-31T10:18:12.937305476Z}
  }#3C429EE90BA6EB327164BB374FBDDDC6C5D642ED0982221F5338848279413C30
}#038229C0A4EA897787DA2BC00CA930192C97F6E1092409D02E9F4828BA1032A8
 [recovered]
        panic: block.AppHash does not match AppHash after replay. Got 9184A084A22D3B55411412F5E3557D1CBB17E802A966DF9D224B9D345DF73E5F, expected 2C3B0358B8628A5FF93A6EE7270FFBF12925243A4FC192C9105B99620648C1DC.
Block: Block{
  Header{
    Version:        {11 1}
    ChainID:        vega-devnet1-202308302128
    Height:         49204
    Time:           2023-08-31 10:18:12.937305476 +0000 UTC
    LastBlockID:    868116409DA94381D66EBF3F7F979643A609FD7254E58EE4230FB6A17D268AF0:2:5A0E9B4A46E2
    LastCommit:     3C429EE90BA6EB327164BB374FBDDDC6C5D642ED0982221F5338848279413C30
    Data:           8AB4AF58E6984B65A92219E286E69FF90B56C63BC3C6BD29F718057570FF6918
    Validators:     8BB7ADC72719B80FBBEEBBCA6CED347C6A2D0376578F12094A6DD66169DC5BAD
    NextValidators: 8BB7ADC72719B80FBBEEBBCA6CED347C6A2D0376578F12094A6DD66169DC5BAD
    App:            2C3B0358B8628A5FF93A6EE7270FFBF12925243A4FC192C9105B99620648C1DC
    Consensus:      C8CD39A92B22597AC33565A59990B4D1E440B15621B5E932D1D5B39A8F45E9DB
    Results:        4A65E03448BA3851993A8002DE0B798584BA81BA38E9D572621914D72993608E
    Evidence:       E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
    Proposer:       EE43F6C7C98B99B9826BD6F1CECD111E9BB0E3E2
  }#038229C0A4EA897787DA2BC00CA930192C97F6E1092409D02E9F4828BA1032A8
  Data{
    EDC1B38676010458CC0C033FC71FE49368F7570667767142BBB6D0B9C46C859A (308 bytes)
    6C5491C53D89D0A20A0CC96587B65EE8D6E4302D3940B2517D07E1FE28B6CCA9 (307 bytes)
    619D244D439D5871AF515B746A6BBB9D9E33102766C102D4EF67312D449625FF (416 bytes)
    FFD814C380C4A6F4A9E09F452C8F1073FE65F0BEF1868D70AB6B7C71CFBAAB4C (415 bytes)
    0E30464629A674ADED0C0A88DA9944CF633388D0FB016D4992AA472DDCFE2469 (417 bytes)
    3E03C2161CE42C6DC7BD5B2C2D44B8DD95981C769C985A090E6835A477467C5A (412 bytes)
    4E22325779A257AFD7D5F784F96D496215169289542709B7F1901B8B9A494391 (418 bytes)
    0E6777714E38B5B8AFE4D124C59746BACE3C1DFF191067B6A9ABEF5680306D9A (417 bytes)
    BE1E2622BBAD91F38A78212118281135C7DF83A4D80061E1E79BD209D8C720E1 (426 bytes)
    46A962C81CFDE4DE372BDD2F2A871954226AB04D1156353E7C87E33234EDC605 (426 bytes)
    25B65544C32E4A61025C9DA4DA3DFC13D73DF558ADCDCDBCD6CF9DBE6005D8AA (424 bytes)
    A621CB3DE57171E1F4491DBF5826751E009EA1799B257DB55BD0BD77DDB0DB50 (426 bytes)
    A3A459A8B9687F93C07E63990AB21A9875E1B6FA19D1671FC22A39B46E0990F0 (424 bytes)
    0CF62BEE7460D4C6FE0AC051EF04DD6332A8EDDA2310D8E1DB05AAD4BFC446A3 (414 bytes)
    7410F94925C2DD12D2D086BF50C040F1C7B21A98D240F07DCA65242286379F37 (416 bytes)
    6200001E9474DFF24F61DE5EE5AE2CA8BC1C43D36538418E187CCBF4F0765E69 (413 bytes)
    82FB3A1DE53A03161C4CCB1D029AC135831F89C9AD0E2FB3C3CC252F1B1FEA92 (417 bytes)
    02D9A5FEA73129FC8DA816CC3897A5F5FE188514AAD32C3171F85146FACB0AB3 (417 bytes)
    A534CDAD9174E48527EEDC6EBB0CD26AAB713B0227EF4113723C3991196441D0 (414 bytes)
    8AC12598F220129EA5B6C8A668D28E090DC5B6ECF43DA5E93491263B25DAEFCC (419 bytes)
    ... (182 total)
  }#8AB4AF58E6984B65A92219E286E69FF90B56C63BC3C6BD29F718057570FF6918
  EvidenceData{

  }#E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
  Commit{
    Height:     49203
    Round:      1
    BlockID:    868116409DA94381D66EBF3F7F979643A609FD7254E58EE4230FB6A17D268AF0:2:5A0E9B4A46E2
    Signatures:
      CommitSig{000000000000 by 000000000000 on 1 @ 0001-01-01T00:00:00Z}
      CommitSig{46A82BEB975B by 9B08B3797CAE on 2 @ 2023-08-31T10:18:13.093601264Z}
      CommitSig{6022F4DECF74 by A8F6E168F587 on 2 @ 2023-08-31T10:18:12.847113923Z}
      CommitSig{46464F2F3E03 by CAB18606029E on 2 @ 2023-08-31T10:18:12.944661849Z}
      CommitSig{9A3E1AD1B311 by EE43F6C7C98B on 2 @ 2023-08-31T10:18:12.937305476Z}
  }#3C429EE90BA6EB327164BB374FBDDDC6C5D642ED0982221F5338848279413C30
}#038229C0A4EA897787DA2BC00CA930192C97F6E1092409D02E9F4828BA1032A8
goroutine 150 [running]:
code.vegaprotocol.io/vega/cmd/vega/commands/node.(*Command).Run.func1.1()
        /jenkins/workspace/private/Deployments/Publish-vega-dev-releases/vega/cmd/vega/commands/node/node.go:154 +0x59
panic({0x2f6eae0, 0xc00f93a920})
        /usr/local/go/src/runtime/panic.go:884 +0x213
github.com/tendermint/tendermint/consensus.assertAppHashEqualsOneFromBlock({0xc00f9ec080, 0x20, 0x20}, 0xc00e7083c0)
        /jenkins/GOPATH/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/replay.go:515 +0x10d
github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlocks(_, {{{0xb, 0x1}, {0xc012d61660, 0x7}}, {0xc010a134a0, 0x19}, 0x1, 0xc151, {{0xc010a134c0, ...}, ...}, ...}, ...)
        /jenkins/GOPATH/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/replay.go:468 +0x1fd
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x1}, {0xc012d61660, 0x7}}, {0xc010a134a0, 0x19}, 0x1, 0xc151, {{0xc010a134c0, ...}, ...}, ...}, ...)
        /jenkins/GOPATH/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/replay.go:393 +0x479
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc00ed18d80, {0x3f52990, 0xc010a51a00})
        /jenkins/GOPATH/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/replay.go:268 +0x3d4
github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x1}, {0xc012d61660, 0x7}}, {0xc010a134a0, 0x19}, 0x1, 0xc151, ...}, ...)
        /jenkins/GOPATH/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/node/node.go:329 +0x1b8
github.com/tendermint/tendermint/node.NewNode(0xc00cb1a3c0, {0x3f21d10, 0xc0104595e0}, 0xc012d4c7f0, {0x3f05ca0, 0xc012d2cea0}, 0x0?, 0x0?, 0xc012d4c830, {0x3f27330, ...}, ...)
        /jenkins/GOPATH/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/node/node.go:779 +0x597
code.vegaprotocol.io/vega/core/blockchain/abci.NewTmNode({{0x0}, 0x1, 0x1, 0x0, 0x0, {0xc00137d0d0, 0xa}, {{0x0}, {0xc000319b60, 0x15}}, ...}, ...)
        /jenkins/workspace/private/Deployments/Publish-vega-dev-releases/vega/core/blockchain/abci/tm_node.go:83 +0x725
code.vegaprotocol.io/vega/cmd/vega/commands/node.(*Command).startABCI(0xc001404480, 0xc00b642758?, {0x3f4c3e0, 0xc01056dca0}, {0x7ffc4484be78, 0x1a}, {0x0?, 0x3894d70?}, {0x0, 0x0})
        /jenkins/workspace/private/Deployments/Publish-vega-dev-releases/vega/cmd/vega/commands/node/node.go:426 +0x1b6
code.vegaprotocol.io/vega/cmd/vega/commands/node.(*Command).startBlockchain(0xc001404480, 0xc00023fa40?, {0x7ffc4484be78?, 0x301fda0?}, {0x0, 0x0}, {0x0, 0x0})
        /jenkins/workspace/private/Deployments/Publish-vega-dev-releases/vega/cmd/vega/commands/node/node.go:313 +0x3ab
code.vegaprotocol.io/vega/cmd/vega/commands/node.(*Command).Run.func1()
        /jenkins/workspace/private/Deployments/Publish-vega-dev-releases/vega/cmd/vega/commands/node/node.go:157 +0x99
created by code.vegaprotocol.io/vega/cmd/vega/commands/node.(*Command).Run
        /jenkins/workspace/private/Deployments/Publish-vega-dev-releases/vega/cmd/vega/commands/node/node.go:148 +0x74f
wwestgarth commented 10 months ago

duplicate of https://github.com/vegaprotocol/vega/issues/9252

Same market has the same pegged orders being restored in the wrong order.