getTransactionByHash and getTransactionReceipt returns null for valid transaction hash

[NZT48]

Description

RPC calls eth_getTransactionByHash and eth_getTransactionReceipt returns null result for valid transactions.

• This is happening roughly on half of our nodes or more across all networks and we can't seem to find the reason.
• They are all running the same code (runtime and client), on the same hardware and issue exists on different cloud providers.
• Nodes that have this issues are running more than 3 months. At start they didn't have this issue, it started happening at undefined moment.
• By inspecting /frontier/db directory and /frontier/db/LOG file we didn't managed to identify any suspicious difference between working and non-working nodes.

Just restarting nodes does not affect this issue. We are suspecting that it has to be something with storage, because copying database (db/full directory) from working node to the node with issue and replacing his db/full directory, mitigates this issue.

This issue affects web3.js and metamask:

• Web3.js gets stuck while waiting for transaction receipt while transaction was successfully executed
• Metamask marks successful transactions as dropped

Steps to Reproduce

It cannot be reproduced in local environment because it is happening only on nodes that are running more than 3 months. We used following command to check:

curl http://localhost:9933 -H "Content-Type:application/json;charset=utf-8" -d \
'{
"jsonrpc":"2.0",
"id":1,
"method":"eth_getTransactionByHash",
"params": ["0x555a49ce69eedc22393a03586f8b9853eb0dda9385bbd299356bfa4256e79cd0"]
}'

Expected vs. Actual Behavior

Result that we get:

{"jsonrpc":"2.0","result":null,"id":1}

Expected result:

{"jsonrpc":"2.0","result":{"hash":"0x555a49ce69eedc22393a03586f8b9853eb0dda9385bbd299356bfa4256e79cd0","nonce":"0x26","blockHash":"0xa4ee773ab4b6a302fa68691086405a41767b1218df077340429ccddab6d1e8cb","blockNumber":"0x8a7d4","transactionIndex":"0x0","from":"0x051387e48e5d668f812cb91b708b0f38364ec5c1","to":"0xffffffff00000000000000000000000000000001","value":"0x0","gasPrice":"0x8","gas":"0xb0fc","input":"0xa9059cbb00000000000000000000000093edf89d9f9f55339fdef1e1be541959f2acea140000000000000000000000000000000000000000000000de6c4fbe0ba50fa000","creates":null,"raw":"0xf8a6260882b0fc94ffffffff0000000000000000000000000000000180b844a9059cbb00000000000000000000000093edf89d9f9f55339fdef1e1be541959f2acea140000000000000000000000000000000000000000000000de6c4fbe0ba50fa000821019a04c5684f58de15d29aeb0d3fae3d47c80c0e7dbe03ed328c806c5cb6946497acea02b0b8258f00cd39403b632a225371c8725760b6923a5b9fa67439569a4621c03","publicKey":"0x6c83f74f603117ff9c8fbee6e4ee18b36d7ef394c207d3ed7d66e856ba7e6ccf8355578b7d9e69267f64bb1643b8ef4a5906eb65b28a712e91df5a4d61ecd8ab","chainId":"0x7fb","standardV":"0x0","v":"0x1019","r":"0x4c5684f58de15d29aeb0d3fae3d47c80c0e7dbe03ed328c806c5cb6946497ace","s":"0x2b0b8258f00cd39403b632a225371c8725760b6923a5b9fa67439569a4621c03","accessList":null,"type":"0x0"},"id":1}

Environment

• Operating system: Ubuntu
• Parachain github repo: https://github.com/OriginTrail/origintrail-parachain
• Frontier repo fork: https://github.com/OriginTrail/frontier/tree/polkadot-v0.9.27

Logs, Errors or Screenshots

There were no errors in node logs.

Additional Information

We would like to ask for additional guide how to investigate this issue and to check with other teams if they have experienced it.

• Is there some node flag to get debug logs for frontier, please let us know and we will provide those logs?
• Is there a way to check if Frontier DB is corrupted, or should we look more at some frontier db files? We checked, issue is in substrate storage (ethereum pallet storage)

[boundless-forest]

Can you fetch the corresponding tx-related information from pallet-ethereum storage? If not, the tx is not included in the chain.

[NZT48]

@AsceticBear the transaction is included in the chain because it is available on the other half of nodes which work correctly and change in transfer funds is visible on all nodes.

Can you give me more instructions on how to fetch the corresponding tx-related information from pallet-ethereum storage?

[boundless-forest]

the transaction is included in the chain because it is available on the other half of nodes which work correctly and change in transfer funds is visible on all nodes.

This situation indicates that the transaction was not rejected because of the validation. Another possibility is that the PALLET_ETHEREUM_SCHEMA of the problematic node is incorrect, which leads to a wrong decode result at the client.

[NZT48]

@AsceticBear is there a way how we can check those two assumptions?

[boundless-forest]

Hmm, Check out the PALLET_ETHEREUM_SCHEMA storage item in your chain. Nothing magic.

[tgmichel]

Please confirm:

• You introduced frontier on your Sept 12 release.
• You are using runtime api (not storage overrides).
• You are using ethereum runtime api version 4.
• All your nodes are using the same cli args with wasm execution, pruning archive, etc..
• All your nodes, including the erroring ones, are on your >= 1.0.4 client.

If all the above is true then the only possible way of not getting ethereum related data through rpc is some of your nodes didn't properly synced frontier DB.

[NZT48]

Hey @tgmichel, sorry for late reply.

• Yes, we introduced frontier on Sept 12 release
• I am not sure about storage overrides, maybe this helps
• Yes, we are using ethereum runtime api version 4
• All nodes have wasm execution flag, some of them have pruning archive, some not (issue exists on both)
• All nodes are on version 1.0.4 or above

I think it is not related to Frontier DB, because copying frontier/db directory from working node does not fix the issue. It was the replacement of db directory that fixes the issue.

All our nodes are constantly checked for this issue, and we saw that for a new node (that is started with copied and synced DB from working node) it took around 11 days for this issue to start happening. We couldn't find anything suspicious in logs for that period.