Consensys / quorum

A permissioned implementation of Ethereum supporting data privacy
https://www.goquorum.com/
GNU Lesser General Public License v3.0
4.68k stars 1.29k forks source link

Quorum nodes unable to sync with BAD Block error after validators are shutdown and restored. #1587

Open rupesh-pithva opened 1 year ago

rupesh-pithva commented 1 year ago

Setup Details: We are running GoQuorum with the high availability setup over AKS.

Quorum Version: 22.7.4 Number of Parties: 2 (say Party A and Party B) Quorum: 3 pods for both parties Tessera: 1 node with 2 replicas (3 pods for both parties) Validator: 4 pods with Party A, none with Party B Other details: IBFT consensus protocol

Scenario: All 4 validators of PartyA are down forcefully. 5 Transactions are submitted on PartyA. All the transactions are in pending state in transaction queue. After 1 hour, the validators on PartyA are restored.

Issue: Quorum transaction nodes are unable to sync blocks further with BAD Block error on all transaction PODs with below error. INFO [12-15|12:44:05.288] Skip duplicated bad blocknumber=825,158 hash=89cb9a..84fd3b ERROR[12-15|12:44:05.288] ########## BAD BLOCK ######### Chain config: {ChainID: 1000 Homestead: 0 DAO: DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 IsQuorum: true Constantinople: 0 TransactionSizeLimit: 128 MaxCodeSize: 0 Petersburg: 0 Istanbul: 0, Muir Glacier: , Berlin: Catalyst: YOLO v3: PrivacyEnhancements: 0 PrivacyPrecompile: EnableGasPriceBlock: Engine: istan bul}

Number: 825158 Hash: 0x89cb9abbb9adad15f2f317db4db844484298bb2815311da4a5656700db84fd3b

Error: could not apply tx 2 [0x027b82b570c5442ea943dfefc169a025d98379d9a968eb908533e94ec08b93eb]: gas limit reached ##############################

The block # 825158 on validator is having all 5 transactions which were in pending queue. Since the gas limit for block is 700000000 (each transaction is submitted with 350000000 gas), we expect only 2 transactions in a block(this is what happens in sunny day scenario when validators are up and running). Can additional transactions in block cause gas limit reached error and processing is stuck because of that? Block Data from validator { difficulty: "0x1", extraData: "0xd983010a03846765746889676f312e31362e3135856c696e7578000000000000f90164f854944946579522783b7b0cb3643fef1241ae99e2a33f9465bb63d8d1445e13b2359baebdc445e8d76e7ec9948e29bd29a d6c2bcec89246983b68c4790afe64ae949d4be5c6eae5ee3b3394ece3fe8ef5b9710bde40b841813bb0b1c93ba261faf4fb129ef5c3692f8925c85d5e35d9881205b7ab57829663b155a415a9cf9f0f956ed045a46e093e5f280431e3 8ac2662a3cd66a8a2f9901f8c9b841626c558084b5f02f0be4fb04e4769098f6a98d26dd32d2a16772b6dbf5d978162f3859468b200317e3595ba1090f3180d242bcc84596280babab28a829aa457e01b841c64c89b1680f84a9d65fc b82cc5e506ee779cb9f2ae55e4921efaf13e900d52936db05fd7e32fa44a9a182c3b48f45f398fcd5c1aefeb4cf8de87cd4b2efef2500b8416dae1fde506ca96d8dc735104d05fd4cde863f5a82f0639a5e372666a1a1c0642a2b6889 6fdb7934637b2be576afa6c703f03bc88c45592e82ff5f270a76e35a00", gasLimit: "0x29b92700", gasUsed: "0x0", hash: "0x89cb9abbb9adad15f2f317db4db844484298bb2815311da4a5656700db84fd3b", logsBloom: "0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000", miner: "0x0000000000000000000000000000000000000000", mixHash: "0x63746963616c2062797a616e74696e65206661756c7420746f6c6572616e6365", nonce: "0x0000000000000000", number: "0xc9746", parentHash: "0xf8a98ec4c2ed50a44adb0ce453dee399bd8fe767932c72f79094df37f86f0626", receiptsRoot: "0xa0004892f3c1af17ec3582d2c44c272f3dfb6211ef63491cad74814114d66ea8", sha3Uncles: "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347", size: "0x6c9", stateRoot: "0x8ced0a226907ab111de2f676dbc9c8fdcc3636ab2cef4caf2efcb1773097d882", timestamp: "0x639b109d", totalDifficulty: "0xc9747", transactions: ["0x6a741068cf181a3bf57374ffad874a35244ea3e426960094490c4cb0ec0b3ab0", "0xd987514580509c2515642165fdf6a6043c26359ae949a968dd0f17a06767464f", "0x027b82b570c5442ea943dfefc 169a025d98379d9a968eb908533e94ec08b93eb", "0xe4d8fa6539cba54763370d74fd66db362a3d0d44e07ec44e5d6c78da783b81d8", "0x473d1d334d55e2b50414dec7ebbe65dab9d4f3abfc6b6e18f897cc868c4fc66d"], transactionsRoot: "0x80fa789362499a14598d5658996597edf66404780676dae4cb85509bd8b5d081", uncles: [] }

Note:- The gasLimit set on genesis file is 3758096384.

The genesis file ` { "alloc": {}, "coinbase": "0x0000000000000000000000000000000000000000", "config": { "homesteadBlock": 0, "byzantiumBlock": 0, "constantinopleBlock": 0, "petersburgBlock": 0, "istanbulBlock": 0, "eip150Block": 0, "eip150Hash": "0x0000000000000000000000000000000000000000000000000000000000000000", "eip155Block": 0, "eip158Block": 0, "maxCodeSizeConfig": [ { "block": 0, "size": 128 } ], "txnSizeLimit": 128, "chainId": 1000, "isQuorum": true, "istanbul": { "epoch": 30000, "policy": 0, "ceil2Nby3Block": 0 } },

"difficulty": "0x1", "extraData": "0x0000000000000000000000000000000000000000000000000000000000000000f89af85494bfb4dc5445976bb5060508517d279b27616501b39494b5a886f3931f25c415855bfe8ee807267551db94fb807f7bc4b6ccd23f5430983e9274aa4106e33094c8ec42597df155de1bd26f95516f41857ce6dde1b8410000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000c0", "gasLimit": "0xE0000000", "mixHash": "0x63746963616c2062797a616e74696e65206661756c7420746f6c6572616e6365", "nonce": "0x0", "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000", "timestamp": "0x00" }

`

yohanelly95 commented 2 months ago

facing a similar issue with bad block @rupesh-pithva have you been able to resolve this?

Skip duplicated bad block                number=654 hash=90e389..2cbc30
ERROR[08-19|04:02:28.017]
########## BAD BLOCK #########
Chain config: {ChainID: 221337 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 IsQuorum: true Constantinople: 0 TransactionSizeLimit: 64 MaxCodeSize: 0 Petersburg: 0 Istanbul: 0,
Muir Glacier: 0, Berlin: 0  Catalyst: <nil> YOLO v3: <nil> PrivacyEnhancements: <nil> PrivacyPrecompile: <nil> EnableGasPriceBlock: 10 Engine: unknown}

Number: 654
Hash: 0x90e389472c73cffd98e8e31d60cf7d9435d6e83741d55e429cfdf118f32cbc30

Error: missing trie node 00f80973ba25c01f5ead672ee998280533441107cec0c69e5da84beed8f2da6a (path )
##############################