golashr commented 5 years ago

Hi, We have Geth+Tessera nodes deployed on distributed machines. So we are extending the quorum code. We are facing one issue after doing upstream synch with commit details given below.

System information

System information Geth version: 1.8.18-stable Quorum version:quorum-v2.2.3 Tessera version: quorumengineering/tessera:0.9.1 OS & Version: Linux/OSX Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-47-generic x86_64) Branch - master Commit -'53105b16400284551b5eefc491b7d7092a7b43c1'

The trie states are missing intermittently but pretty consistently.

The below is the log from the geth console

Welcome to the Geth JavaScript console!

instance: Geth/validator-128.167.133.211/v1.8.18-stable-68efe47c(quorum-v2.2.3)/linux-amd64/go1.11.10
coinbase: 0x2fc6991e16308104cf15130ea92a87066bc4dfb1
at block: 200 (Tue, 25 Jun 2019 21:55:46 AEST)
 datadir: /eth
 modules: admin:1.0 debug:1.0 eth:1.0 istanbul:1.0 net:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0

> eth.getBalance(eth.coinbase)
3.56811
> eth.getBalance(eth.coinbase)
3.56811
> eth.getBalance(eth.coinbase)
3.56811
> eth.getBalance(eth.coinbase)
Error: missing trie node d5daa6769555b347a312cbac59f896427afab5ce0adc7949944ad4110c08e44d (path )
    at web3.js:3143:20
    at web3.js:6347:15
    at web3.js:5081:36
    at <anonymous>:1:1

> eth.getBalance(eth.coinbase)
3.56811

Below is the log from the application where we are trying to submit public transaction while deploying contract using nodejs/web3 tech stack.

➜  testsmartcontracts git:(master) node index.js protocol=http hostname=128.167.133.211 port=8545 readkeyconfig=true **testgreeter**
There are 17 ethereum accounts & private keys in the privatekey file
gasPrice  0x3b9aca00
FromAccount 0x2fc6991e16308104cf15130ea92a87066bc4dfb1 has balance of 356811923176489970264571594.360650378095686655 xlg
nonceToUse  27
Greeter deployedAddress  0x3a96c8984Fb2ef956Dc4B2AA84Aa388493E7c733
(node:48741) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: Returned error: missing trie node ff06b53404d85acba066a634fad6f71baf58987eec8a1b55166c9da66f7da12e (path )
(node:48741) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Help appreciated.

jbhurat commented 5 years ago

Hi @golashr, I tried to reproduce the issue, but so far no luck. Can you provide more details on your setup;

How many nodes/validators are part of the network
What is the block period
Is the network running locally or on cloud
Are you using docker
Also, please share your genesis json

golashr commented 5 years ago

@jbhurat Thanks for the quick revert! How many nodes/validators are part of the network Ledgerium Blockchain will be public blockchain which will support private transactions using Quorum with Istanbul BFT as base. In the current test environment, there are 4 nodes.

What is the block period We have kept it 5 seconds. Default value is 1.

Is the network running locally or on cloud In the current test environment, all nodes are local.

Are you using docker Yes. These are docker containers

Also, please share your genesis json

{  
   "config":{  
      "chainId":2018,
      "byzantiumBlock":1,
      "eip150Block":2,
      "eip150Hash":"0x0000000000000000000000000000000000000000000000000000000000000000",
      "eip155Block":0,
      "eip158Block":3,
      "istanbul":{  
         "epoch":30000,
         "policy":0
      },
      "isQuorum":true
   },
   "nonce":"0x0",
   "timestamp":"0x5b7fa5cb",
   "extraData":"0x0000000000000000000000000000000000000000000000000000000000000000f89af854941a67eea756b9c074219dbbd1a68b7a691941264594cb680d316b281aebb68e73e813b7a53ec93c8f0594c1836760988668eea5275cd562755ec0b82cef929486dc44a5ae35a09a62585f2e3de29f14dce2ff2cb8410000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000c0",
   "gasLimit":"0x1FFFFFFFFFFC20",
   "difficulty":"0x1",
   "mixHash":"0x63746963616c2062797a616e74696e65206661756c7420746f6c6572616e6365",
   "coinbase":"0x0000000000000000000000000000000000000000",
   "alloc":{  
      "1a67eea756b9c074219dbbd1a68b7a6919412645":{  
         "balance":"0xfffffffffffffffffffffffffffffffffffff"
      },
      "cb680d316b281aebb68e73e813b7a53ec93c8f05":{  
         "balance":"0xfffffffffffffffffffffffffffffffffffff"
      },
      "c1836760988668eea5275cd562755ec0b82cef92":{  
         "balance":"0xfffffffffffffffffffffffffffffffffffff"
      },
      "86dc44a5ae35a09a62585f2e3de29f14dce2ff2c":{  
         "balance":"0xfffffffffffffffffffffffffffffffffffff"
      }
   },
   "number":"0x0",
   "gasUsed":"0x0",
   "parentHash":"0x0000000000000000000000000000000000000000000000000000000000000000"
}

Could block period be a problem?

golashr commented 5 years ago

ok, so I tried with block period as 1 (default value) too. The same problem existed.

jbhurat commented 5 years ago

Thanks for info @golashr. I tried the below to reproduce the issue, but still no luck

Start the network with 3 nodes and a block period of 1 sec
Executed around hundred transactions
Added a new node after 1000 blocks
Ran eth.getBalance(eth.coinbase) in succession to reproduce the issue, but no luck

Did you do anything different?

golashr commented 5 years ago

The only difference that we have from the code is, https://github.com/jpmorganchase/quorum/blob/master/consensus/istanbul/backend/engine.go#L383 we are adding some balance to the miner by calling

state.AddBalance(author, minerblockReward)

Mind you, this code was working well before merging the last commit so I raised the ticket.

jbhurat commented 5 years ago

I am able to reproduce the issue after adding the above code. From the logs and correlating it back to the code, it looks like the issue is happening because the state is not in a consistent state. Also, Quorum v2.2.3 is based on Geth 1.8.18 and there are lots of changes around how state works.

I think you would want to enable miner block rewards with the above change. If yes, can you see if this PR addresses your needs https://github.com/jpmorganchase/quorum/pull/624

golashr commented 5 years ago

624 changes are related to gas price. Not sure if that will help. Would you like to suggest specific code change? I am not sure if I understood "enable miner block rewards with the above change". Can you help understand it? thanks. BTW, we indeed have non-zero gas price and it is working as expected.

jbhurat commented 5 years ago

state.AddBalance(author, minerblockReward)
From the above code change, I thought that you are trying to enable miner block rewards. Also, #624 is to enable gas price and as you are using non-zero gas price it might help.

jbhurat commented 5 years ago

Hi @golashr, I believe this was working on quorum version v2.2.1 and earlier. Quorum v2.2.1 is based on Geth 1.8.12 and Quorum v2.2.3 is based on Geth 1.8.18. The code around how state is handled is completely rewritten between these two versions. Also, looks like the changes you are trying to make changes to the state object and hence the error you are getting.

Ideally we only support issues with Quorum and not on changes being made on top of it. Please reach out on slack and we can bounce ideas around how to proceed from here

golashr commented 5 years ago

@jbhurat I guess, we need some investigation on it still. As mentioned above, adding AddBalance() only makes the state inconsistent. Else, the state remains stable. Geth 1.8.18 still has AddBalance() code in place as expected. https://github.com/ethereum/go-ethereum/blob/master/consensus/ethash/consensus.go#L614

Let me take the discussion on slack.

Consensys / quorum

Intermittent Error: missing trie node XXX (path ) #752

System information