search

hypersign-protocol / hid-node

A permissionless blockchain network to manage digital identity and access rights
https://hypersign.id
Apache License 2.0
223 stars 23 forks source link

Consensus module error in EC2 (DevNet) #122

Closed arnabghose997 closed 2 years ago

arnabghose997 commented 2 years ago

While running the 3-node network on EC2, following error occured:

4:04AM INF executed block height=426299 module=state num_invalid_txs=0 num_valid_txs=0                                                                                          
4:04AM INF commit synced commit=436F6D6D697449447B5B313335203131392031333420313138203131302031363320313633203238203633203137332039302031303420323134203538203136203130302038332$
31383420313431203138312033312031323720323236203831203234322036203434203133322031363620313339203738203232355D3A36383133427D                                                      
4:04AM INF committed state app_hash=877786766EA3A31C3FAD5A68D63A106453B88DB51F7FE251F2062C84A68B4EE1 height=426299 module=state num_txs=0                                       
4:04AM INF indexed block height=426299 module=txindex                                                                                                                           
4:04AM INF Timed out dur=4982.375264 height=426300 module=consensus round=0 step=1                                                                                              
4:04AM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"4FC8D58C42A18C686BA071D504D495FE1CAA7B59CE87CCE7DB2CB0F1F31ABC33","parts":{"hash":"4C54EA$
5AC50CA69115A8914B00D777D27AE10D0DFBA2E2C446A81528154CBBF","total":1}},"height":426300,"pol_round":-1,"round":0,"signature":"hYZFBd/3IsFLoDlpmpJbJBvFdu0qf/RZf29Uz7jQaSAk/Atn6l$
1OxIWI7DKn2V7Eukuf83SRrUAwmESB1sMCA==","timestamp":"2022-04-06T04:04:16.259725519Z"}                                                                                            
4:04AM INF received complete proposal block hash=4FC8D58C42A18C686BA071D504D495FE1CAA7B59CE87CCE7DB2CB0F1F31ABC33 height=426300 module=consensus                                
4:04AM INF finalizing commit of block hash=4FC8D58C42A18C686BA071D504D495FE1CAA7B59CE87CCE7DB2CB0F1F31ABC33 height=426300 module=consensus num_txs=0 root=877786766EA3A31C3FAD5$
68D63A106453B88DB51F7FE251F2062C84A68B4EE1                                                                                                                                      
4:04AM INF minted coins from module account amount=2215uhid from=mint module=x/bank                                                                                             
4:04AM INF executed block height=426300 module=state num_invalid_txs=0 num_valid_txs=0                                                                                          
4:04AM ERR CONSENSUS FAILURE!!! err="cannot delete latest saved version (8)" module=consensus stack="goroutine 212 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runti$
e/debug/stack.go:24 +0x65\ngithub.com/tendermint/tendermint/consensus.(*State).receiveRoutine.func2()\n\t/home/ubuntu/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/cons$
nsus/state.go:726 +0x4c\npanic({0x188f140, 0xc00b831a88})\n\t/usr/local/go/src/runtime/panic.go:1038 +0x215\ngithub.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).pruneStores($
xc0010f4000)\n\t/home/ubuntu/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.5/store/rootmulti/store.go:424 +0x1b6\ngithub.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).Commit($
xc0010f4000)\n\t/home/ubuntu/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.5/store/rootmulti/store.go:398 +0x175\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc001$
1f1e0)\n\t/home/ubuntu/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.5/baseapp/abci.go:307 +0x231\ngithub.com/tendermint/tendermint/abci/client.(*localClient).CommitSync(0xc00$
c1bd40)\n\t/home/ubuntu/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/abci/client/local_client.go:264 +0xb6\ngithub.com/tendermint/tendermint/proxy.(*appConnConsensus).$
ommitSync(0x0)\n\t/home/ubuntu/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/proxy/app_conn.go:93 +0x22\ngithub.com/tendermint/tendermint/state.(*BlockExecutor).Commit($
, {{{0xb, 0x0}, {0xc0022d04e0, 0x7}}, {0xc0022d04e7, 0x7}, 0x1, 0x6813c, {{0xc00b78c6a0, ...}, ...}, ...}, ...)\n\t/home/ubuntu/go/pkg/mod/github.com/tendermint/tendermint@v0.$
4.14/state/execution.go:228 +0x269\ngithub.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc0022d04e0, 0x7}}, {0xc0022d04e7, 0x7}, 0x1, 0x6813c$
 {{0xc00b78c6a0, ...}, ...}, ...}, ...)\n\t/home/ubuntu/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/state/execution.go:180 +0x6ee\ngithub.com/tendermint/tendermint/co$
sensus.(*State).finalizeCommit(0xc0000bf880, 0x6813c)\n\t/home/ubuntu/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/state.go:1635 +0x9fd\ngithub.com/tendermin$
/tendermint/consensus.(*State).tryFinalizeCommit(0xc0000bf880, 0x6813c)\n\t/home/ubuntu/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/state.go:1546 +0x305\ngit
hub.com/tendermint/tendermint/consensus.(*State).enterCommit.func1()\n\t/home/ubuntu/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/state.go:1481 +0x87\ngithub.
com/tendermint/tendermint/consensus.(*State).enterCommit(0xc0000bf880, 0x6813c, 0x0)\n\t/home/ubuntu/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/state.go:151
9 +0xc06\ngithub.com/tendermint/tendermint/consensus.(*State).addVote(0xc0000bf880, 0xc00b73cfa0, {0xc002a16540, 0x28})\n\t/home/ubuntu/go/pkg/mod/github.com/tendermint/tenderm
int@v0.34.14/consensus/state.go:2132 +0xb6e\ngithub.com/tendermint/tendermint/consensus.(*State).tryAddVote(0xc0000bf880, 0xc00b73cfa0, {0xc002a16540, 0xc00b54c100})\n\t/home/u
buntu/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/state.go:1930 +0x2c\ngithub.com/tendermint/tendermint/consensus.(*State).handleMsg(0xc0000bf880, {{0x1f8762
0, 0xc00b4419d8}, {0xc002a16540, 0x0}})\n\t/home/ubuntu/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/state.go:838 +0x40b\ngithub.com/tendermint/tendermint/con
sensus.(*State).receiveRoutine(0xc0000bf880, 0x0)\n\t/home/ubuntu/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/state.go:762 +0x419\ncreated by github.com/tend
ermint/tendermint/consensus.(*State).OnStart\n\t/home/ubuntu/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/state.go:378 +0x12f\n"
4:04AM INF Stopping baseWAL service impl={"Logger":{}} module=consensus wal=/home/ubuntu/.hid-node/node1/data/cs.wal/wal
4:04AM INF Stopping Group service impl={"Dir":"/home/ubuntu/.hid-node/node1/data/cs.wal","Head":{"ID":"UjLsKFNhezNM:/home/ubuntu/.hid-node/node1/data/cs.wal/wal","Path":"/home/
ubuntu/.hid-node/node1/data/cs.wal/wal"},"ID":"group:UjLsKFNhezNM:/home/ubuntu/.hid-node/node1/data/cs.wal/wal","Logger":{}} module=consensus wal=/home/ubuntu/.hid-node/node1/d
ata/cs.wal/wal

Following is the error after performing the restart of hid-node:

panic: cannot delete latest saved version (8)

goroutine 1 [running]:
github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).pruneStores(0xc0010a8d20)
    /home/ubuntu/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.5/store/rootmulti/store.go:424 +0x1b6
github.com/cosmos/cosmos-sdk/store/rootmulti.(*Store).Commit(0xc0010a8d20)
    /home/ubuntu/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.5/store/rootmulti/store.go:398 +0x175
github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).Commit(0xc001071380)
    /home/ubuntu/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.5/baseapp/abci.go:307 +0x231
github.com/tendermint/tendermint/abci/client.(*localClient).CommitSync(0xc000f6a180)
    /home/ubuntu/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/abci/client/local_client.go:264 +0xb6
github.com/tendermint/tendermint/proxy.(*appConnConsensus).CommitSync(0x2e5c4f8)
    /home/ubuntu/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/proxy/app_conn.go:93 +0x22
github.com/tendermint/tendermint/state.(*BlockExecutor).Commit(_, {{{0xb, 0x0}, {0xc00435ce28, 0x7}}, {0xc00435ce50, 0x7}, 0x1, 0x6813c, {{0xc004243f60, ...}, ...}, ...}, ...)
    /home/ubuntu/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/state/execution.go:228 +0x269
github.com/tendermint/tendermint/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc00435ce28, 0x7}}, {0xc00435ce50, 0x7}, 0x1, 0x6813c, {{0xc004243f60, ...}, ...}, ...}, ...)
    /home/ubuntu/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/state/execution.go:180 +0x6ee
github.com/tendermint/tendermint/consensus.(*Handshaker).replayBlock(_, {{{0xb, 0x0}, {0xc00435ce28, 0x7}}, {0xc00435ce50, 0x7}, 0x1, 0x6813b, {{0xc0042429c0, ...}, ...}, ...}, ...)
    /home/ubuntu/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/replay.go:503 +0x23c
github.com/tendermint/tendermint/consensus.(*Handshaker).ReplayBlocks(_, {{{0xb, 0x0}, {0xc00435ce28, 0x7}}, {0xc00435ce50, 0x7}, 0x1, 0x6813b, {{0xc0042429c0, ...}, ...}, ...}, ...)
    /home/ubuntu/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/replay.go:416 +0x7ae
github.com/tendermint/tendermint/consensus.(*Handshaker).Handshake(0xc00440fdd0, {0x1ff2fa0, 0xc00106f2b0})
    /home/ubuntu/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/consensus/replay.go:268 +0x3c8
github.com/tendermint/tendermint/node.doHandshake({_, _}, {{{0xb, 0x0}, {0xc00435ce28, 0x7}}, {0xc00435ce50, 0x7}, 0x1, 0x6813b, ...}, ...)
    /home/ubuntu/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/node/node.go:325 +0x1b8
github.com/tendermint/tendermint/node.NewNode(0xc000f83cc0, {0x1fa5328, 0xc0010a9d60}, 0xc001112c80, {0x1f877a0, 0xc003ec16f8}, 0x0, 0x0, 0xc001112e40, {0x1fc7a20, ...}, ...)
    /home/ubuntu/go/pkg/mod/github.com/tendermint/tendermint@v0.34.14/node/node.go:733 +0x578
github.com/cosmos/cosmos-sdk/server.startInProcess(_, {{0x0, 0x0, 0x0}, {0x2002ed8, 0xc0001f9920}, {0x0, 0x0}, {0x1fcfc90, 0xc000f87690}, ...}, ...)
    /home/ubuntu/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.5/server/start.go:263 +0x7c5
github.com/cosmos/cosmos-sdk/server.StartCmd.func2(0xc000fea500, {0xc001057600, 0x0, 0x1})
    /home/ubuntu/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.5/server/start.go:129 +0x193
github.com/spf13/cobra.(*Command).execute(0xc000fea500, {0xc0010575f0, 0x1, 0x1})
    /home/ubuntu/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:856 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0xc000d15400)
    /home/ubuntu/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).Execute(...)
    /home/ubuntu/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:902
github.com/spf13/cobra.(*Command).ExecuteContext(...)
    /home/ubuntu/go/pkg/mod/github.com/spf13/cobra@v1.2.1/command.go:895
github.com/cosmos/cosmos-sdk/server/cmd.Execute(0x1d9a3d8, {0xc000c61380, 0x16})
    /home/ubuntu/go/pkg/mod/github.com/cosmos/cosmos-sdk@v0.44.5/server/cmd/execute.go:36 +0x1eb
main.main()
    /home/ubuntu/hm/hid-node/cmd/hid-noded/main.go:21 +0x7d
arnabghose997 commented 2 years ago

The consensus module failed because there fundamental shift of KV store structure. Initially the variables used in DID proto structure was simply a string. In the following iterations, it was changed from string to individual fields hence causing exception at store level