search

zeta-chain / node

ZetaChain’s blockchain node and an observer validator client
https://zetachain.com
MIT License
157 stars 97 forks source link

Data race warning in `zetacore` during execution of smoke tests #1486

Open lumtis opened 7 months ago

lumtis commented 7 months ago

This warning in smoke tests in the node needs to be investigated as the underlying issue could trigger a consensus failure

This appears to happen when certain txs are sent to the node during smoke test execution

==================
zetacore0  | WARNING: DATA RACE
zetacore0  | Write at 0x00c00318e2d0 by goroutine 213:
zetacore0  |   github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).BeginBlock()
zetacore0  |       /go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.46.13/baseapp/abci.go:177 +0x770
zetacore0  |   github.com/zeta-chain/zetacore/app.(*App).BeginBlock()
zetacore0  |       <autogenerated>:1 +0x64
zetacore0  |   github.com/tendermint/tendermint/abci/client.(*localClient).BeginBlockSync()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/abci/client/local_client.go:280 +0x108
zetacore0  |   github.com/tendermint/tendermint/proxy.(*appConnConsensus).BeginBlockSync()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/proxy/app_conn.go:81 +0x80
zetacore0  |   github.com/tendermint/tendermint/state.execBlockOnProxyApp()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/state/execution.go:307 +0x4c0
zetacore0  |   github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/state/execution.go:140 +0x158
zetacore0  |   github.com/tendermint/tendermint/consensus.(*State).finalizeCommit()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/consensus/state.go:1661 +0xe50
zetacore0  |   github.com/tendermint/tendermint/consensus.(*State).tryFinalizeCommit()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/consensus/state.go:1570 +0x3d0
zetacore0  |   github.com/tendermint/tendermint/consensus.(*State).enterCommit.func1()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/consensus/state.go:1505 +0xd8
zetacore0  |   runtime.deferreturn()
zetacore0  |       /usr/local/go/src/runtime/panic.go:476 +0x30
zetacore0  |   github.com/tendermint/tendermint/consensus.(*State).addVote()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/consensus/state.go:2164 +0x2200
zetacore0  |   github.com/tendermint/tendermint/consensus.(*State).tryAddVote()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/consensus/state.go:1962 +0x44
zetacore0  |   github.com/tendermint/tendermint/consensus.(*State).handleMsg()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/consensus/state.go:861 +0x4d4
zetacore0  |   github.com/tendermint/tendermint/consensus.(*State).receiveRoutine()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/consensus/state.go:768 +0x3f8
zetacore0  |   github.com/tendermint/tendermint/consensus.(*State).receiveRoutine()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/consensus/state.go:768 +0x3f8
zetacore0  |   github.com/tendermint/tendermint/consensus.(*State).receiveRoutine()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/consensus/state.go:788 +0x4a0
zetacore0  |   github.com/tendermint/tendermint/consensus.(*State).receiveRoutine()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/consensus/state.go:788 +0x4a0
zetacore0  |   github.com/tendermint/tendermint/consensus.(*State).receiveRoutine()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/consensus/state.go:788 +0x4a0
zetacore0  |   github.com/tendermint/tendermint/consensus.(*State).receiveRoutine()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/consensus/state.go:788 +0x4a0
zetacore0  |   github.com/tendermint/tendermint/consensus.(*State).receiveRoutine()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/consensus/state.go:788 +0x4a0
zetacore0  |   github.com/tendermint/tendermint/consensus.(*State).receiveRoutine()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/consensus/state.go:788 +0x4a0
zetacore0  |   github.com/tendermint/tendermint/consensus.(*State).receiveRoutine()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/consensus/state.go:788 +0x4a0
zetacore0  |   github.com/tendermint/tendermint/consensus.(*State).receiveRoutine()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/consensus/state.go:788 +0x4a0
zetacore0  |   github.com/tendermint/tendermint/consensus.(*State).receiveRoutine()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/consensus/state.go:788 +0x4a0
zetacore0  |   github.com/tendermint/tendermint/consensus.(*State).OnStart.func1()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/consensus/state.go:379 +0x38
zetacore0  |
zetacore0  | Previous read at 0x00c00318e2d0 by goroutine 935:
zetacore0  |   github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).createQueryContext()
zetacore0  |       /go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.46.13/baseapp/abci.go:653 +0x3f0
zetacore0  |   github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func1()
zetacore0  |       /go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.46.13/baseapp/grpcserver.go:50 +0x164
zetacore0  |   github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1()
zetacore0  |       /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x6c
zetacore0  |   github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1()
zetacore0  |       /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc0
zetacore0  |   github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1()
zetacore0  |       /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x6c
zetacore0  |   github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1()
zetacore0  |       /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xf0
zetacore0  |   github.com/zeta-chain/zetacore/x/crosschain/types._Query_LastZetaHeight_Handler()
zetacore0  |       /go/delivery/zeta-node/x/crosschain/types/query.pb.go:4511 +0x180
zetacore0  |   github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RegisterGRPCServer.func2()
zetacore0  |       /go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.46.13/baseapp/grpcserver.go:82 +0x100
zetacore0  |   google.golang.org/grpc.(*Server).processUnaryRPC()
zetacore0  |       /go/pkg/mod/google.golang.org/grpc@v1.55.0/server.go:1337 +0x1168
zetacore0  |   google.golang.org/grpc.(*Server).handleStream()
zetacore0  |       /go/pkg/mod/google.golang.org/grpc@v1.55.0/server.go:1714 +0xbb4
zetacore0  |   google.golang.org/grpc.(*Server).serveStreams.func1.1()
zetacore0  |       /go/pkg/mod/google.golang.org/grpc@v1.55.0/server.go:959 +0xa4
zetacore0  |
zetacore0  | Goroutine 213 (running) created at:
zetacore0  |   github.com/tendermint/tendermint/consensus.(*State).OnStart()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/consensus/state.go:379 +0x184
zetacore0  |   github.com/tendermint/tendermint/libs/service.(*BaseService).Start()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/libs/service/service.go:144 +0x4b8
zetacore0  |   github.com/tendermint/tendermint/consensus.(*Reactor).SwitchToConsensus()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/consensus/reactor.go:134 +0x168
zetacore0  |   github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/blockchain/v0/reactor.go:324 +0x13c4
zetacore0  |   github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart.func1()
zetacore0  |       /go/pkg/mod/github.com/cometbft/cometbft@v0.34.28/blockchain/v0/reactor.go:112 +0x38
zetacore0  |
zetacore0  | Goroutine 935 (finished) created at:
zetacore0  |   google.golang.org/grpc.(*Server).serveStreams.func1()
zetacore0  |       /go/pkg/mod/google.golang.org/grpc@v1.55.0/server.go:957 +0x264
zetacore0  |   google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
zetacore0  |       /go/pkg/mod/google.golang.org/grpc@v1.55.0/internal/transport/http2_server.go:631 +0x39b8
zetacore0  |   google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
zetacore0  |       /go/pkg/mod/google.golang.org/grpc@v1.55.0/internal/transport/http2_server.go:673 +0x194
zetacore0  |   google.golang.org/grpc.(*Server).serveStreams()
zetacore0  |       /go/pkg/mod/google.golang.org/grpc@v1.55.0/server.go:946 +0x1e4
zetacore0  |   google.golang.org/grpc.(*Server).handleRawConn.func1()
zetacore0  |       /go/pkg/mod/google.golang.org/grpc@v1.55.0/server.go:889 +0x54
CharlieMc0 commented 6 months ago

Added a note that when this is addresses we also need to ensure smoke tests always run with -race enabled and we begin failing the pipeline when a race condition is found.

lumtis commented 5 months ago

All the race warning are happening in the querier, which would not cause consensus failure.