error storing block: could not transform hex string to big int:

[deantheiceman]

I'm connecting mine to a private ethereum network, but seem to be getting this error in my logs, any ideas?

time="2019-12-04T03:50:35Z" level=error msg="[core] error storing block: could not transform hex string to big int: " block=4497684
time="2019-12-04T03:50:35Z" level=info msg="[core] processing block" block=4497683

[deantheiceman]

time="2019-12-04T03:52:20Z" level=debug msg="[core] validating block" block=4497697
time="2019-12-04T03:52:20Z" level=debug msg="[core] block is valid" block=4497697
time="2019-12-04T03:52:20Z" level=debug msg="[core] storing block into the database" block=4497697
time="2019-12-04T03:52:20Z" level=error msg="[data] could not transform hex string to big int: "
time="2019-12-04T03:52:20Z" level=debug msg="[data] done storing block" duration="82.151µs"
time="2019-12-04T03:52:20Z" level=error msg="[core] error storing block: could not transform hex string to big int: " block=4497697
time="2019-12-04T03:52:20Z" level=info msg="[core] processing block" block=4497696
time="2019-12-04T03:52:20Z" level=debug msg="[scraper] getting block" block=4497696

With debug mode on

[lacasian]

Hello!

It looks like it is trying to convert an empty string to big int. We have to find out which property is missing and handle it accordingly.

Could you share what kind of ethereum client are you using? Could you execute the following command and post the output?

curl http://your-node-here -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["0x44A114", true],"id":1}'

[deantheiceman]

Hello @kwix, thanks for the reply!

Here is the output from your command, I'm using https://github.com/vechain/web3-gear which appears to have some missing methods as described by the typical ethereum client.

{"result": {"number": "0x44a114", "hash": "0x0044a114619f8927b640e126e1af0d57ecfc03dd1b9d5b68c297f87f6db59236", "size": "0xf3", "parentHash": "0x0044a113a09c355e6e5b8e67911d3ed13ee828ce661089320a3b3004491a4dce", "timestamp": "0x5de0ae22", "gasLimit": "0x1000000", "beneficiary": "0xb4094c25f86d628fdd571afc4077f0d0196afb48", "gasUsed": "0x0", "totalDifficulty": "0x1126bb3", "transactionsRoot": "0x45b0cfc220ceec5b7c1c62c4d4193d38e4eba48e8815729ce75f9c0ab0e4c1c0", "txsFeatures": 1, "stateRoot": "0x076c2b4df092378385eed346709d829ac8df61b1d1ee2f37fe7aa51e6571aa43", "receiptsRoot": "0x45b0cfc220ceec5b7c1c62c4d4193d38e4eba48e8815729ce75f9c0ab0e4c1c0", "miner": "0x231f3c552e085b04ce33b1d0c00d433b8c6fabe7", "isTrunk": true, "transactions": []}, "id": 1, "jsonrpc": "2.0"}

[lacasian]

Hey @deantheiceman! Thanks for the output. I've had a look and it looks like it is missing multiple fields that Memento is looking for:

difficulty 
extraData
logsBloom
mixHash
nonce
sha3Uncles
transactionsRoot

The field that generates the error in your case is difficulty. I'm afraid that even if we would modify Memento to allow missing difficulty in order to fix this particular error, we would encounter other issues down the road generated by other missing properties. We will add this to our roadmap as part of a broader solution.

[deantheiceman]

Thanks.

The client is written in python, I may be able to edit it to return the data points needed. Could you help me understand what results I should return.

Is there any other methods other than eth_getBlockByNumber which is used?

[lacasian]

Memento uses 3 RPC calls internally:

• eth_getBlockByNumber
• eth_getTransactionReceipt
• eth_getUncleByBlockHashAndIndex - this can be disabled via feature flags if the method is not implemented (ganache, for example)

For eth_getBlockByNumber I already shared the fields that are missing in my previous comment. This RPC also contains transactions which I couldn't inspect since there were none in the example you shared. The name of the properties I shared is the exact same with the responses you would get from the most known ethereum clients - geth or parity, for example. I'm not sure how I could help further with this. If you have other questions, feel free to reach out.

If you could provide me with some more examples of raw RPC responses from the methods I shared, I will give you the list of properties needed. For eth_getBlockByNumber please include one that has one or more transactions.

[deantheiceman]

Thanks for the help.

Here is a getBlockbyNumber request with a transaction:

{
  "result": {
    "number": "0x44c192",
    "hash": "0x0044c192ac307b04353193611094b575c642a1f76d36c704c1b2c6dcb401d26a",
    "size": "0x19c",
    "parentHash": "0x0044c191bc6c52b017ff365f8ff305d060d543aba17e1643dee3296daf1b5040",
    "timestamp": "0x5de872c4",
    "gasLimit": "0xf2f0d4",
    "beneficiary": "0xeb0c565f69557481c6c7fa347cae273128a0996e",
    "gasUsed": "0x9088",
    "totalDifficulty": "0x19fa1536",
    "transactionsRoot": "0xa9261b07281ff08a391fb36006561ff00866a26623cb58ab2b72a642962426d0",
    "txsFeatures": 1,
    "stateRoot": "0x201dc731df6fd49ffaf757222b97c82cd590bfec3fcf3e83c0f51da36aa42429",
    "receiptsRoot": "0x156cd2c11591e4a29eabc5aac9a6f9cdb3ebd548041b3bae40e35ec883b2443c",
    "miner": "0x8eaefdf7d25c001e7e59363c33d7f5ad47970086",
    "isTrunk": true,
    "transactions": [{
      "hash": "0xabd323770c59ee9b19f96b6fa39eb087584578bd7019fb298d64e4de07db4502",
      "nonce": "0xd37783d9273a6043",
      "blockHash": "0x0044c192ac307b04353193611094b575c642a1f76d36c704c1b2c6dcb401d26a",
      "blockNumber": "0x44c192",
      "transactionIndex": "0x0",
      "from": "0xa4adafaef9ec07bc4dc6de146934c7119341ee25",
      "to": "0x00bbaa077dbcc4dd1a47a64b53f8b654efc971d6",
      "value": "0xd3c34d1e805e03980000",
      "gas": "0xd6d8",
      "gasPrice": "0x1",
      "input": "0x"
    }]
  },
  "id": 1,
  "jsonrpc": "2.0"
}

[deantheiceman]

Here's getTransactionReceipt

curl http://68.183.235.56:8545 -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_getTransactionReceipt","params":["0xabd323770c59ee9b19f96b6fa39eb087584578bd7019fb298d64e4de07db4502"],"id":1}'

{
  "result": {
    "status": "0x1",
    "transactionHash": "0xabd323770c59ee9b19f96b6fa39eb087584578bd7019fb298d64e4de07db4502",
    "transactionIndex": "0x0",
    "blockNumber": "0x44c192",
    "blockHash": "0x0044c192ac307b04353193611094b575c642a1f76d36c704c1b2c6dcb401d26a",
    "cumulativeGasUsed": "0x9088",
    "gasUsed": "0x9088",
    "contractAddress": null,
    "logs": []
  },
  "id": 1,
  "jsonrpc": "2.0"
}

[deantheiceman]

The mapping for it is here getTransactionReceipt is here https://github.com/vechain/web3-gear/blob/5391fa8c4d0e66b2b051a5eca77ef41e8652325d/gear/utils/compat.py#L55

And the real output is here:

{
  "gasUsed": 21000,
  "gasPayer": "0xdb4027477b2a8fe4c83c6dafe7f86678bb1b8a8d",
  "paid": "0x1236efcbcbb340000",
  "reward": "0x576e189f04f60000",
  "reverted": false,
  "outputs": [
    {
      "contractAddress": null,
      "events": [
        {
          "address": "0x7567d83b7b8d80addcb281a71d54fc7b3364ffed",
          "topics": [
            "0x4de71f2d588aa8a1ea00fe8312d92966da424d9939a511fc0be81e65fad52af8"
          ],
          "data": "0x4de71f2d588aa8a1ea00fe8312d92966da424d9939a511fc0be81e65fad52af8"
        }
      ],
      "transfers": [
        {
          "sender": "0xdb4027477b2a8fe4c83c6dafe7f86678bb1b8a8d",
          "recipient": "0x5034aa590125b64023a0262112b98d72e3c8e40e",
          "amount": "0x47fdb3c3f456c0000"
        }
      ]
    }
  ],
  "meta": {
    "blockID": "0x0004f6cc88bb4626a92907718e82f255b8fa511453a78e8797eb8cea3393b215",
    "blockNumber": 325324,
    "blockTimestamp": 1533267900,
    "txID": "0x284bba50ef777889ff1a367ed0b38d5e5626714477c40de38d71cedd6f9fa477",
    "txOrigin": "0xdb4027477b2a8fe4c83c6dafe7f86678bb1b8a8d"
  }
}

I'll just need to update the mapping for which memento expects I guess

[lacasian]

The data on transactions looks good.

The only thing missing on receipts is logsBloom.

Could you also look for a transaction receipt that includes one or more logs?

[lacasian]

By looking at the mapping, the logs from getTransactionReceipt should also be fine.

[deantheiceman]

Let me try to find a transaction receipt which includes more then one logs.

What is logsBloom is it possible I can just return a placeholder data if it's not provided by my network?

[deantheiceman]

So in summary so far we identified:

eth_getBlockByNumber:

difficulty  <-- only totalDifficulty is returned  
extraData <-- not provided, can we leave it null/return a placeholder?
logsBloom <-- not provided, can we leave it null/return a placeholder?
mixHash <-- not provided, can we leave it null/return a placeholder?
nonce <-- not provided, can we leave it null/return a placeholder?
sha3Uncles <-- disable
transactionsRoot <-- not provided, can we leave it null/return a placeholder?

getTransactionReceipt:

logsBloom <-- not provided, can we leave it null/return a placeholder?

[lacasian]

The difficulty property must be returned, which will fix the error you've encountered. After that, you can test and see if anything else breaks.

Hint: the difficulty is the difference between totalDifficulty of this block and the parent block.

[deantheiceman]

Thanks, I'm trying it now will report back with my success

[deantheiceman]

Awesome, blocks are importing now, thanks a lot for the help!

[deantheiceman]

Good news, blocks are importing. Bad news, for some reason it seems to fill up all my postgresql connections almost immediately. Especially when backdating

[lacasian]

Interesting. Thanks for sharing that. Memento is using the default sql package which has an internal pool of connections and decides how it opens and reuses the connections by itself. To be hones, it is surprising to me that it uses more than one connection since it is only doing one thing at a time.

[deantheiceman]

Actually seems like the error was because it failed to insert it was opening a new connection and it would just keep opening more connections.

In the end I found out my issue was related to nonce being returned as an extremely large number, so seems like by setting it to a ByteArray I was able to get it to insert

Thanks!