search

vegaprotocol / vega

A Go implementation of the Vega Protocol, a protocol for creating and trading derivatives on a fully decentralised network.
https://vega.xyz
GNU Affero General Public License v3.0
36 stars 22 forks source link

panic: block.AppHash does not match AppHash after replay #8856

Closed vega-paul closed 1 year ago

vega-paul commented 1 year ago

Snapshot panic ttps://jenkins.ops.vega.xyz/blue/organizations/jenkins/common%2Fsnapshot-soak-tests/detail/snapshot-soak-tests/8989/pipeline


goroutine 709 [running]:
go.uber.org/zap/zapcore.CheckWriteAction.OnWrite(0x1?, 0x1?, {0x0?, 0x0?, 0xc004bc2620?})
    /jenkins/GOPATH/pkg/mod/go.uber.org/zap@v1.24.0/zapcore/entry.go:198 +0x65
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc0052c7380, {0xc00339d680, 0x1, 0x1})
    /jenkins/GOPATH/pkg/mod/go.uber.org/zap@v1.24.0/zapcore/entry.go:264 +0x3ec
go.uber.org/zap.(*Logger).Panic(0xc0018def00?, {0x50d02c0?, 0x0?}, {0xc00339d680, 0x1, 0x1})
    /jenkins/GOPATH/pkg/mod/go.uber.org/zap@v1.24.0/logger.go:258 +0x59
code.vegaprotocol.io/vega/core/snapshot.(*Engine).snapshotNow.func2()
    /jenkins/workspace/common/system-tests-wrapper/vega/core/snapshot/engine.go:932 +0x14b
created by code.vegaprotocol.io/vega/core/snapshot.(*Engine).snapshotNow
    /jenkins/workspace/common/system-tests-wrapper/vega/core/snapshot/engine.go:942 +0xe05
panic: block.AppHash does not match AppHash after replay. Got E2F910CAAE56AD1D286AD2772C9805BFFBB0A0CB7D5BD58B2279FEE2F1B46DFF, expected E385DB3C357E106C6E3150E773A0C8E7D3C9E0E338FA109F30BDCEE34E887E47.

Block: Block{
  Header{
    Version:        {11 1}
    ChainID:        testnet-001
    Height:         3148
    Time:           2023-07-19 01:45:30.000783045 +0000 UTC
    LastBlockID:    381C98976A4393DFA74A64BED8F712780BD4B60E26434E70C88919BD8E1933DB:1:ED92E55E6C05
    LastCommit:     F1E4BE08D670C51B3A81FD531AE18738A6FA66A345526D75709DA806A8A2C22B
    Data:           E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
    Validators:     140F29DD3035CCB8B7D62D79D845005FB9826763A80CD28355EFBBBA77A6F638
    NextValidators: 140F29DD3035CCB8B7D62D79D845005FB9826763A80CD28355EFBBBA77A6F638
    App:            E385DB3C357E106C6E3150E773A0C8E7D3C9E0E338FA109F30BDCEE34E887E47
    Consensus:      C8CD39A92B22597AC33565A59990B4D1E440B15621B5E932D1D5B39A8F45E9DB
    Results:        E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
    Evidence:       E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
    Proposer:       8E6EAE64859D5BCB2E3A6296C3BC22B176014E6A
  }#EA53DA723CB0FAE5D9C0A351F6659E46466604C4BF0E889C7A01B682467E1D8D
  Data{

  }#E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
  EvidenceData{

  }#E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
  Commit{
    Height:     3147
    Round:      0
    BlockID:    381C98976A4393DFA74A64BED8F712780BD4B60E26434E70C88919BD8E1933DB:1:ED92E55E6C05
    Signatures:
      CommitSig{BBFC768922FF by 31DD484E006B on 2 @ 2023-07-19T01:45:30.098706324Z}
      CommitSig{CEE921E6C51C by 8E6EAE64859D on 2 @ 2023-07-19T01:45:30.000783045Z}
  }#F1E4BE08D670C51B3A81FD531AE18738A6FA66A345526D75709DA806A8A2C22B
}#EA53DA723CB0FAE5D9C0A351F6659E46466604C4BF0E889C7A01B682467E1D8D
 [recovered]
    panic: block.AppHash does not match AppHash after replay. Got E2F910CAAE56AD1D286AD2772C9805BFFBB0A0CB7D5BD58B2279FEE2F1B46DFF, expected E385DB3C357E106C6E3150E773A0C8E7D3C9E0E338FA109F30BDCEE34E887E47.

Block: Block{
  Header{
    Version:        {11 1}
    ChainID:        testnet-001
    Height:         3148
    Time:           2023-07-19 01:45:30.000783045 +0000 UTC
    LastBlockID:    381C98976A4393DFA74A64BED8F712780BD4B60E26434E70C88919BD8E1933DB:1:ED92E55E6C05
    LastCommit:     F1E4BE08D670C51B3A81FD531AE18738A6FA66A345526D75709DA806A8A2C22B
    Data:           E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
    Validators:     140F29DD3035CCB8B7D62D79D845005FB9826763A80CD28355EFBBBA77A6F638
    NextValidators: 140F29DD3035CCB8B7D62D79D845005FB9826763A80CD28355EFBBBA77A6F638
    App:            E385DB3C357E106C6E3150E773A0C8E7D3C9E0E338FA109F30BDCEE34E887E47
    Consensus:      C8CD39A92B22597AC33565A59990B4D1E440B15621B5E932D1D5B39A8F45E9DB
    Results:        E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
    Evidence:       E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
    Proposer:       8E6EAE64859D5BCB2E3A6296C3BC22B176014E6A
  }#EA53DA723CB0FAE5D9C0A351F6659E46466604C4BF0E889C7A01B682467E1D8D
  Data{

  }#E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
  EvidenceData{

  }#E3B0C44298FC1C149AFBF4C8996FB92427AE41E4649B934CA495991B7852B855
  Commit{
    Height:     3147
    Round:      0
    BlockID:    381C98976A4393DFA74A64BED8F712780BD4B60E26434E70C88919BD8E1933DB:1:ED92E55E6C05
    Signatures:
      CommitSig{BBFC768922FF by 31DD484E006B on 2 @ 2023-07-19T01:45:30.098706324Z}
      CommitSig{CEE921E6C51C by 8E6EAE64859D on 2 @ 2023-07-19T01:45:30.000783045Z}
  }#F1E4BE08D670C51B3A81FD531AE18738A6FA66A345526D75709DA806A8A2C22B
}#EA53DA723CB0FAE5D9C0A351F6659E46466604C4BF0E889C7A01B682467E1D8D

goroutine 160 [running]:
code.vegaprotocol.io/vega/cmd/vega/commands/node.(*Command).Run.func1.1()
    /jenkins/workspace/common/system-tests-wrapper/vega/cmd/vega/commands/node/node.go:137 +0x59
panic({0x470c700, 0xc005c6c450})
    /usr/local/go/src/runtime/panic.go:884 +0x212
github.com/tendermint/tendermint/consensus.assertAppHashEqualsOneFromBlock({0xc005f08ee0, 0x20, 0x20}, 0xc003bb43c0)
    /jenkins/GOPATH/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/replay.go:515 +0x10d
github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlocks(_, {{{0xb, 0x1}, {0xc001574bf0, 0x7}}, {0xc001574c00, 0xb}, 0x1, 0x2b6b, {{0xc00186cac0, ...}, ...}, ...}, ...)
    /jenkins/GOPATH/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/replay.go:468 +0x1fd
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x1}, {0xc001574bf0, 0x7}}, {0xc001574c00, 0xb}, 0x1, 0x2b6b, {{0xc00186cac0, ...}, ...}, ...}, ...)
    /jenkins/GOPATH/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/replay.go:393 +0x479
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc001786d90, {0x585e3c0, 0xc0002a8dd0})
    /jenkins/GOPATH/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/consensus/replay.go:268 +0x3d4
github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x1}, {0xc001574bf0, 0x7}}, {0xc001574c00, 0xb}, 0x1, 0x2b6b, ...}, ...)
    /jenkins/GOPATH/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/node/node.go:329 +0x1b8
github.com/tendermint/tendermint/node.NewNode(0xc000003b80, {0x5826e50, 0xc000f132c0}, 0xc0014feb90, {0x5809780, 0xc0014b45a0}, 0x0?, 0x0?, 0xc0014febf0, {0x582d490, ...}, ...)
    /jenkins/GOPATH/pkg/mod/github.com/vegaprotocol/cometbft@v0.34.28-0.20230322133204-3d8588de736e/node/node.go:779 +0x597
code.vegaprotocol.io/vega/core/blockchain/abci.NewTmNode({{0x0}, 0x1, 0x1, 0x0, 0x0, {0xc00189d250, 0xa}, {{0x0}, {0xc000527a70, 0x15}}, ...}, ...)
    /jenkins/workspace/common/system-tests-wrapper/vega/core/blockchain/abci/tm_node.go:83 +0x705
code.vegaprotocol.io/vega/cmd/vega/commands/node.(*Command).startABCI(0xc0018c7800, 0x0?, {0x5857cc0, 0xc001254020}, {0x7ffdefa25493, 0x10}, {0x0?, 0x0?}, {0x0, 0x0})
    /jenkins/workspace/common/system-tests-wrapper/vega/cmd/vega/commands/node/node.go:412 +0x1b6
code.vegaprotocol.io/vega/cmd/vega/commands/node.(*Command).startBlockchain(0xc0018c7800, 0x0?, {0x7ffdefa25493?, 0x0?}, {0x0, 0x0}, {0x0, 0x0})
    /jenkins/workspace/common/system-tests-wrapper/vega/cmd/vega/commands/node/node.go:299 +0x3ab
code.vegaprotocol.io/vega/cmd/vega/commands/node.(*Command).Run.func1()
    /jenkins/workspace/common/system-tests-wrapper/vega/cmd/vega/commands/node/node.go:140 +0x90
created by code.vegaprotocol.io/vega/cmd/vega/commands/node.(*Command).Run
    /jenkins/workspace/common/system-tests-wrapper/vega/cmd/vega/commands/node/node.go:131 +0x66f
panic: failed to save snapshot to disk
gordsport commented 1 year ago

@vega-paul - please can you let me know if this issue is still presenting?

ze97286 commented 1 year ago

@gordsport it's most likely the same issue as currently in fairground, so I'd say yes.

ze97286 commented 1 year ago

ah, I didn't read, it's snapshot, so ignore me. there's another ticket for the failure in system test.

gordsport commented 1 year ago

Closing as not seeiin in current soak tests - if this comes back we will need o add a zip of the logs