search

ainblockchain / ain-blockchain

Official Javascript implementation of AI Network Blockchain.
https://www.ainetwork.ai/
MIT License
67 stars 27 forks source link

[BUG] lastBlock doesn't have enough votes in restart with --keep-data, --keep-code #891

Open platfowner opened 2 years ago

platfowner commented 2 years ago

commands:

bash deploy_blockchain_genesis_gcp.sh dev <Username> 0 --private-key --keep-data --keep-code

logs:

{"name":"testnet-dev","hostname":"dev-node-0-taiwan","pid":11499,"level":50,"msg":"[CONSENSUS] [tryPropose] Error while creating a proposal: Error: [getValidLastVotes] lastBlock doesn't have enough votes\n
    at BlockPool.getValidLastVotes (/home/ain-blockchain/block-pool/index.js:579:11)\n
    at Consensus.createProposal (/home/ain-blockchain/consensus/index.js:428:36)\n
    at Consensus.tryPropose (/home/ain-blockchain/consensus/index.js:965:18)\n
    at Timeout._onTimeout (/home/ain-blockchain/consensus/index.js:192:16)\n
    at listOnTimeout (node:internal/timers:557:17)\n
    at processTimers (node:internal/timers:500:7)","time":"2022-01-07T19:05:18.916Z","v":0}
platfowner commented 2 years ago

when the problematic last block (with its snapshot file) was deleted manually, the second trial was successful.

platfowner commented 2 years ago

Reproduced since the last PR merge: blockchain_snapshot

with logs of node-4:

{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] [tryPropose] Error
 while creating a proposal: Error: [getValidLastVotes] lastBlock doesn't have enough votes: 20\n    at BlockPool.getVa
lidLastVotes (/home/ain-blockchain/block-pool/index.js:579:11)\n    at Consensus.createProposal (/home/ain-blockchain/
consensus/index.js:428:36)\n    at Consensus.tryPropose (/home/ain-blockchain/consensus/index.js:972:18)\n    at Timeo
ut._onTimeout (/home/ain-blockchain/consensus/index.js:192:16)\n    at processTicksAndRejections (node:internal/proces
s/task_queues:96:5)","time":"2022-01-08T08:49:17.191Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 18 with hash 0xaaee887356b75152ab5d9726471d24231c5d3c463d7dffaa5be98f880559e7fa","time":"2022-01-08T08:49:1
9.231Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] [handleConsensusMe
ssage] Cannot process the vote or it's invalid: {\"tx_body\":{\"operation\":{\"type\":\"SET_VALUE\",\"ref\":\"/consens
us/number/19/0xb388db23fa7daa4b9f7e9e899cd9e981b9bcec7c3d9ef76ba3677feea7b8a606/vote/0x00ADEc28B6a845a085e03591bE7550d
d68673C1C\",\"value\":{\"block_hash\":\"0xb388db23fa7daa4b9f7e9e899cd9e981b9bcec7c3d9ef76ba3677feea7b8a606\",\"stake\"
:10000000,\"is_against\":false,\"vote_nonce\":1641631759096}},\"nonce\":-1,\"gas_price\":0,\"timestamp\":1641631759096
},\"signature\":\"0xc6728e3a02f3c0bea88a827241fb63c7e90bb594ec0cfd795a42acb95f5a307856c46ea7195ebc657810bbe268a5687945
556a9cb25c4a308e80dbf067f350153bd1b03322c4a01f7f90ec5fe89e1cd2c05b1f7928e33737421e9cd84ae8528b1c\",\"hash\":\"0xc6728e
3a02f3c0bea88a827241fb63c7e90bb594ec0cfd795a42acb95f5a3078\",\"address\":\"0x00ADEc28B6a845a085e03591bE7550dd68673C1C\
"}","time":"2022-01-08T08:49:19.233Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 18 with hash 0xaaee887356b75152ab5d9726471d24231c5d3c463d7dffaa5be98f880559e7fa","time":"2022-01-08T08:49:1
9.271Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] [handleConsensusMe
ssage] Cannot process the vote or it's invalid: {\"tx_body\":{\"operation\":{\"type\":\"SET_VALUE\",\"ref\":\"/consens
us/number/19/0xb388db23fa7daa4b9f7e9e899cd9e981b9bcec7c3d9ef76ba3677feea7b8a606/vote/0x01A0980d2D4e418c7F27e1ef539d01A
5b5E93204\",\"value\":{\"block_hash\":\"0xb388db23fa7daa4b9f7e9e899cd9e981b9bcec7c3d9ef76ba3677feea7b8a606\",\"stake\"
:10000000,\"is_against\":false,\"vote_nonce\":1641631759195}},\"nonce\":-1,\"gas_price\":0,\"timestamp\":1641631759195
},\"signature\":\"0x64f2f9929c415b40beefa982ee71d1d837d490bb23f862018eebbb4ec7903c359c569793c3786e855480573307be4871fe
8be371fdcb1b82fb8eb9975a7babfb553221f0470ec0c5add00e9625051f598aaa8a64b38b2b56ac11f215b1df80841c\",\"hash\":\"0x64f2f9
929c415b40beefa982ee71d1d837d490bb23f862018eebbb4ec7903c35\",\"address\":\"0x01A0980d2D4e418c7F27e1ef539d01A5b5E93204\
"}","time":"2022-01-08T08:49:19.273Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] [handleConsensusMe
ssage] Cannot process the vote or it's invalid: {\"tx_body\":{\"operation\":{\"type\":\"SET_VALUE\",\"ref\":\"/consens
us/number/19/0xb388db23fa7daa4b9f7e9e899cd9e981b9bcec7c3d9ef76ba3677feea7b8a606/vote/0x00ADEc28B6a845a085e03591bE7550d
d68673C1C\",\"value\":{\"block_hash\":\"0xb388db23fa7daa4b9f7e9e899cd9e981b9bcec7c3d9ef76ba3677feea7b8a606\",\"stake\"
:10000000,\"is_against\":false,\"vote_nonce\":1641631759096}},\"nonce\":-1,\"gas_price\":0,\"timestamp\":1641631759096
},\"signature\":\"0xc6728e3a02f3c0bea88a827241fb63c7e90bb594ec0cfd795a42acb95f5a307856c46ea7195ebc657810bbe268a5687945
556a9cb25c4a308e80dbf067f350153bd1b03322c4a01f7f90ec5fe89e1cd2c05b1f7928e33737421e9cd84ae8528b1c\",\"hash\":\"0xc6728e
3a02f3c0bea88a827241fb63c7e90bb594ec0cfd795a42acb95f5a3078\",\"address\":\"0x00ADEc28B6a845a085e03591bE7550dd68673C1C\
"}","time":"2022-01-08T08:49:19.274Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] Block's last_votes
 don't correctly notarize its previous block of number 20 with hash 0xd4a25cf3509e68b52aad74ec9b099f83505969925cbbe9b3
229ca63365d8f008:\n{\n  \"proposal\": {\n    \"tx_body\": {\n      \"operation\": {\n        \"type\": \"SET\",\n     
   \"op_list\": [\n          {\n            \"type\": \"SET_VALUE\",\n            \"ref\": \"/consensus/number/20/prop
ose\",\n            \"value\": {\n              \"number\": 20,\n              \"epoch\": 31828,\n              \"vali
dators\": {\n                \"0x00ADEc28B6a845a085e03591bE7550dd68673C1C\": {\n                  \"stake\": 10000000,
\n                  \"proposal_right\": true\n                },\n                \"0x01A0980d2D4e418c7F27e1ef539d01A5
b5E93204\": {\n                  \"stake\": 10000000,\n                  \"proposal_right\": true\n                },\
n                \"0x02A2A1DF4f630d760c82BE07F18e5065d103Fa00\": {\n                  \"stake\": 10000000,\n          
        \"proposal_right\": true\n                },\n                \"0x03AAb7b6f16A92A1dfe018Fe34ee420eb098B98A\": 
{\n                  \"stake\": 10000000,\n                  \"proposal_right\": true\n                },\n           
     \"0x04A456C92A880cd59D7145C457475515a6f6E0f2\": {\n                  \"stake\": 10000000,\n                  \"pr
oposal_right\": true\n                }\n              },\n              \"total_at_stake\": 50000000,\n              
\"proposer\": \"0x00ADEc28B6a845a085e03591bE7550dd68673C1C\",\n              \"block_hash\": \"0xd4a25cf3509e68b52aad7
4ec9b099f83505969925cbbe9b3229ca63365d8f008\",\n              \"last_hash\": \"0xb388db23fa7daa4b9f7e9e899cd9e981b9bce
c7c3d9ef76ba3677feea7b8a606\",\n              \"timestamp\": 1641631779048,\n              \"gas_cost_total\": 0\n    
        }\n          }\n        ]\n      },\n      \"nonce\": -1,\n      \"gas_price\": 0,\n      \"timestamp\": 16416
31779068\n    },\n    \"signature\": \"0x9d8690fd45f938906470008fd5cced6f35f81a1d4856857e6ac8fd53ec54617f0fc750dde73e4
424c73963c021a984f1f0c1d6a38e3d4fcdaeea385f4e9bc5c948322ab69aee8ac199ab93e2366ca0ae6393e42e1fc8676e200b8072828684131b\
",\n    \"hash\": \"0x9d8690fd45f938906470008fd5cced6f35f81a1d4856857e6ac8fd53ec54617f\",\n    \"address\": \"0x00ADEc
28B6a845a085e03591bE7550dd68673C1C\"\n  },\n  \"votes\": [\n    {\n      \"tx_body\": {\n        \"operation\": {\n   
       \"type\": \"SET_VALUE\",\n          \"ref\": \"/consensus/number/20/0xd4a25cf3509e68b52aad74ec9b099f83505969925
cbbe9b3229ca63365d8f008/vote/0x02A2A1DF4f630d760c82BE07F18e5065d103Fa00\",\n          \"value\": {\n            \"bloc
...

"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[P2P_SERVER] Error: ReferenceE
rror: filePath is not defined ReferenceError: filePath is not defined\n    at Function.readH2nFile (/home/ain-blockcha
in/common/file-util.js:250:59)\n    at Blockchain.getBlockByHash (/home/ain-blockchain/blockchain/index.js:94:18)\n   
 at Consensus.getSnapDb (/home/ain-blockchain/consensus/index.js:1137:34)\n    at Consensus.checkVoteTx (/home/ain-blo
ckchain/consensus/index.js:931:27)\n    at Consensus.handleConsensusMessage (/home/ain-blockchain/consensus/index.js:3
38:17)\n    at WebSocket.<anonymous> (/home/ain-blockchain/p2p/server.js:555:30)\n    at WebSocket.emit (node:events:3
90:28)\n    at Receiver.receiverOnMessage (/home/ain-blockchain/node_modules/ws/lib/websocket.js:1008:20)\n    at Rece
iver.emit (node:events:390:28)\n    at Receiver.dataMessage (/home/ain-blockchain/node_modules/ws/lib/receiver.js:517:
14)","time":"2022-01-08T08:49:41.905Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[P2P_SERVER] Error: ReferenceE
rror: filePath is not defined ReferenceError: filePath is not defined\n    at Function.readH2nFile (/home/ain-blockcha
in/common/file-util.js:250:59)\n    at Blockchain.getBlockByHash (/home/ain-blockchain/blockchain/index.js:94:18)\n   
 at Consensus.getSnapDb (/home/ain-blockchain/consensus/index.js:1137:34)\n    at Consensus.checkVoteTx (/home/ain-blo
ckchain/consensus/index.js:931:27)\n    at Consensus.handleConsensusMessage (/home/ain-blockchain/consensus/index.js:3
38:17)\n    at WebSocket.<anonymous> (/home/ain-blockchain/p2p/server.js:555:30)\n    at WebSocket.emit (node:events:3
90:28)\n    at Receiver.receiverOnMessage (/home/ain-blockchain/node_modules/ws/lib/websocket.js:1008:20)\n    at Rece
iver.emit (node:events:390:28)\n    at Receiver.dataMessage (/home/ain-blockchain/node_modules/ws/lib/receiver.js:517:
14)","time":"2022-01-08T08:49:41.906Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[P2P_SERVER] Error: ReferenceE
rror: filePath is not defined ReferenceError: filePath is not defined\n    at Function.readH2nFile (/home/ain-blockcha
in/common/file-util.js:250:59)\n    at Blockchain.getBlockByHash (/home/ain-blockchain/blockchain/index.js:94:18)\n   
 at Consensus.getSnapDb (/home/ain-blockchain/consensus/index.js:1137:34)\n    at Consensus.checkVoteTx (/home/ain-blo
ckchain/consensus/index.js:931:27)\n    at Consensus.handleConsensusMessage (/home/ain-blockchain/consensus/index.js:3
38:17)\n    at WebSocket.<anonymous> (/home/ain-blockchain/p2p/server.js:555:30)\n    at WebSocket.emit (node:events:3
90:28)\n    at Receiver.receiverOnMessage (/home/ain-blockchain/node_modules/ws/lib/websocket.js:1008:20)\n    at Rece
iver.emit (node:events:390:28)\n    at Receiver.dataMessage (/home/ain-blockchain/node_modules/ws/lib/receiver.js:517:
14)","time":"2022-01-08T08:49:41.909Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[P2P_SERVER] Error: ReferenceE
rror: filePath is not defined ReferenceError: filePath is not defined\n    at Function.readH2nFile (/home/ain-blockcha
in/common/file-util.js:250:59)\n    at Blockchain.getBlockByHash (/home/ain-blockchain/blockchain/index.js:94:18)\n   
 at Consensus.getSnapDb (/home/ain-blockchain/consensus/index.js:1137:34)\n    at Consensus.checkVoteTx (/home/ain-blo
ckchain/consensus/index.js:931:27)\n    at Consensus.handleConsensusMessage (/home/ain-blockchain/consensus/index.js:3
38:17)\n    at WebSocket.<anonymous> (/home/ain-blockchain/p2p/server.js:555:30)\n    at WebSocket.emit (node:events:3
90:28)\n    at Receiver.receiverOnMessage (/home/ain-blockchain/node_modules/ws/lib/websocket.js:1008:20)\n    at Rece
iver.emit (node:events:390:28)\n    at Receiver.dataMessage (/home/ain-blockchain/node_modules/ws/lib/receiver.js:517:
14)","time":"2022-01-08T08:49:41.910Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 21 with hash 0x5d99fe564a6b3a8de53c1a883cfef031b49aff0a01cd8cfab0382f740d949f1d","time":"2022-01-08T08:50:3
9.682Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 21 with hash 0x5d99fe564a6b3a8de53c1a883cfef031b49aff0a01cd8cfab0382f740d949f1d","time":"2022-01-08T08:50:3
9.821Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 21 with hash 0x5d99fe564a6b3a8de53c1a883cfef031b49aff0a01cd8cfab0382f740d949f1d","time":"2022-01-08T08:50:3
9.931Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 21 with hash 0x5d99fe564a6b3a8de53c1a883cfef031b49aff0a01cd8cfab0382f740d949f1d","time":"2022-01-08T08:50:4
0.071Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 21 with hash 0x5d99fe564a6b3a8de53c1a883cfef031b49aff0a01cd8cfab0382f740d949f1d","time":"2022-01-08T08:50:4
0.331Z","v":0} 
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 21 with hash 0x5d99fe564a6b3a8de53c1a883cfef031b49aff0a01cd8cfab0382f740d949f1d","time":"2022-01-08T08:50:4
0.588Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 21 with hash 0x5d99fe564a6b3a8de53c1a883cfef031b49aff0a01cd8cfab0382f740d949f1d","time":"2022-01-08T08:50:4
0.728Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 21 with hash 0x5d99fe564a6b3a8de53c1a883cfef031b49aff0a01cd8cfab0382f740d949f1d","time":"2022-01-08T08:50:4
0.837Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 21 with hash 0x5d99fe564a6b3a8de53c1a883cfef031b49aff0a01cd8cfab0382f740d949f1d","time":"2022-01-08T08:50:4
0.944Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 21 with hash 0x5d99fe564a6b3a8de53c1a883cfef031b49aff0a01cd8cfab0382f740d949f1d","time":"2022-01-08T08:50:4
1.083Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 21 with hash 0x5d99fe564a6b3a8de53c1a883cfef031b49aff0a01cd8cfab0382f740d949f1d","time":"2022-01-08T08:50:4
1.339Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 21 with hash 0x5d99fe564a6b3a8de53c1a883cfef031b49aff0a01cd8cfab0382f740d949f1d","time":"2022-01-08T08:50:4
1.478Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 21 with hash 0x5d99fe564a6b3a8de53c1a883cfef031b49aff0a01cd8cfab0382f740d949f1d","time":"2022-01-08T08:50:4
1.736Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 21 with hash 0x5d99fe564a6b3a8de53c1a883cfef031b49aff0a01cd8cfab0382f740d949f1d","time":"2022-01-08T08:50:4
1.993Z","v":0}
...

{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 40 with hash 0xf9956a68ee20ec575ec578260dbe6fa33f33fc2c619d5cc1025ddeb66b33d9c9","time":"2022-01-08T08:58:3
9.847Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 41 with hash 0xb62c0f958f7c73dcfb93dbccf6d02796fb73e89f97211407943a021348081838","time":"2022-01-08T08:58:3
9.848Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 40 with hash 0xf9956a68ee20ec575ec578260dbe6fa33f33fc2c619d5cc1025ddeb66b33d9c9","time":"2022-01-08T08:58:4
0.063Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 41 with hash 0xb62c0f958f7c73dcfb93dbccf6d02796fb73e89f97211407943a021348081838","time":"2022-01-08T08:58:4
0.064Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 40 with hash 0xf9956a68ee20ec575ec578260dbe6fa33f33fc2c619d5cc1025ddeb66b33d9c9","time":"2022-01-08T08:58:4
0.278Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 41 with hash 0xb62c0f958f7c73dcfb93dbccf6d02796fb73e89f97211407943a021348081838","time":"2022-01-08T08:58:4
0.279Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 40 with hash 0xf9956a68ee20ec575ec578260dbe6fa33f33fc2c619d5cc1025ddeb66b33d9c9","time":"2022-01-08T08:58:4
0.494Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 41 with hash 0xb62c0f958f7c73dcfb93dbccf6d02796fb73e89f97211407943a021348081838","time":"2022-01-08T08:58:4
0.495Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 40 with hash 0xf9956a68ee20ec575ec578260dbe6fa33f33fc2c619d5cc1025ddeb66b33d9c9","time":"2022-01-08T08:58:4
1.265Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 41 with hash 0xb62c0f958f7c73dcfb93dbccf6d02796fb73e89f97211407943a021348081838","time":"2022-01-08T08:58:4
1.265Z","v":0}
{"name":"testnet-dev","hostname":"dev-node-4-netherlands","pid":21261,"level":50,"msg":"[CONSENSUS] No notarized block
 at number 40 with hash 0xf9956a68ee20ec575ec578260dbe6fa33f33fc2c619d5cc1025ddeb66b33d9c9","time":"2022-01-08T08:58:4
1.863Z","v":0}
...
platfowner commented 2 years ago

Just confirmed that the conflict of the last block and the snapshot can be fixed by removing them manually: last_block_manusl_fix

platfowner commented 2 years ago

It seems it happens only with genesis deploy (not with incremental deploy) so far. And shutdown API for genesis deploy is coming soon..

Related issue: