Receipts for certain tx's are null after the v19.1.1 upgrade on mainnet

[SMihai97]

Describe the Bug After the v19.1.1 upgrade yesterday for certain tx's the method eth_getTransactionReceipt return null.

To Reproduce Example request: curl localhost:8545 -H "Content-Type: application/json" -d '{"jsonrpc":"2.0","method":"eth_getTransactionReceipt","params":["0x6c70f22272d06db357b2dca0f0ec47d662dbfcb78a23b546e85843efd5d57248"],"id":1}' returns {"jsonrpc":"2.0","id":1,"result":null}

Expected Behavior For that certain request to give this result: "id":1,"jsonrpc":"2.0","result":{"blockHash":"0x50a346108a7a0a71c1696190e9d3992688427b43ac73321a362c30f50a83a30f","blockNumber":"0x47a87f","contractAddress":null,"cumulativeGasUsed":"0x208dc","from":"0x735b14bb79463307aacbed86daf3322b1e6226ab","gasUsed":"0x650c","logs":[{"address":"0x91d18e54daf4f677cb28167158d6dd21f6ab3921","topics":["0x49f492222906ac486c3c1401fa545626df1f0c0e5a77a05597ea2ed66af9850d"],"data":"0x00000000000000000000000000000000000000000000000000000000000000380000000000000000000000000000000000000000000000000000000047868c00","blockNumber":"0x47a87f","transactionHash":"0x6c70f22272d06db357b2dca0f0ec47d662dbfcb78a23b546e85843efd5d57248","transactionIndex":"0x0","blockHash":"0x50a346108a7a0a71c1696190e9d3992688427b43ac73321a362c30f50a83a30f","logIndex":"0x0","removed":false}],"logsBloom":"0x00000000000000000000000000000008000000000000000000000000000000000000000008000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000002000000020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","status":"0x1","to":"0x91d18e54daf4f677cb28167158d6dd21f6ab3921","transactionHash":"0x6c70f22272d06db357b2dca0f0ec47d662dbfcb78a23b546e85843efd5d57248","transactionIndex":"0x22b8","type":"0x58"}}

Environment (please complete the following information):

• OS - > Ubuntu 22.04
• Version used -> zetacored 19.1.1

There seems to be a lot of blocks after the upgrade that have this issue with the tx's like 4696191, 4696192, 4696193, 4696195, 4696196 and others.

[skosito]

@SMihai97 thanks for reporting this, could you maybe share which rpc url you are using or if you know if the node has some pruning strategy? tried to reproduce with internal mainnet rpc and it appears to be working on my end, and also is working with some public rpcs i quickly googled, eg:

curl -X POST https://zeta-chain.drpc.org     -H "Content-Type: application/json" --data '{"method":"eth_getTransactionReceipt","params":["0x6c70f22272d06db357b2dca0f0ec47d662dbfcb78a23b546e85843efd5d57248"], "id":1, "jsonrpc":"2.0"}'
{"id":1,"jsonrpc":"2.0","result":{"blockHash":"0x50a346108a7a0a71c1696190e9d3992688427b43ac73321a362c30f50a83a30f","blockNumber":"0x47a87f","contractAddress":null,"cumulativeGasUsed":"0x208dc","from":"0x735b14bb79463307aacbed86daf3322b1e6226ab","gasUsed":"0x650c","logs":[{"address":"0x91d18e54daf4f677cb28167158d6dd21f6ab3921","topics":["0x49f492222906ac486c3c1401fa545626df1f0c0e5a77a05597ea2ed66af9850d"],"data":"0x00000000000000000000000000000000000000000000000000000000000000380000000000000000000000000000000000000000000000000000000047868c00","blockNumber":"0x47a87f","transactionHash":"0x6c70f22272d06db357b2dca0f0ec47d662dbfcb78a23b546e85843efd5d57248","transactionIndex":"0x0","blockHash":"0x50a346108a7a0a71c1696190e9d3992688427b43ac73321a362c30f50a83a30f","logIndex":"0x0","removed":false}],"logsBloom":"0x00000000000000000000000000000008000000000000000000000000000000000000000008000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000100000000000000000000000000000000000000002000000020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","status":"0x1","to":"0x91d18e54daf4f677cb28167158d6dd21f6ab3921","transactionHash":"0x6c70f22272d06db357b2dca0f0ec47d662dbfcb78a23b546e85843efd5d57248","transactionIndex":"0x22b8","type":"0x58"}}  

[SMihai97]

Hey @skosito the node has no pruning. I've also tried several providers and I also got a response but on consecutive requests I got some nulls as well, so maybe they have some affected nodes as well.

[skosito]

Hey @skosito the node has no pruning. I've also tried several providers and I also got a response but on consecutive requests I got some nulls as well, so maybe they have some affected nodes as well.

If you can send some more details that could help with debugging this that would be great. Eg. examples of requests that failed, rpc providers, verifying which binary is run on the nodes and anything that could help.

[SMihai97]

I'll have to make a list of more examples, but the general problem seems to be missing receipts for certain tx's after the v19.1.1 upgrade. For example for the tx that I included in the issue 0x6c70f22272d06db357b2dca0f0ec47d662dbfcb78a23b546e85843efd5d57248 the receipt is null across multiple nodes. All nodes running on the v19.1.1 zetacored binary

[skosito]

I'll have to make a list of more examples, but the general problem seems to be missing receipts for certain tx's after the v19.1.1 upgrade. For example for the tx that I included in the issue 0x6c70f22272d06db357b2dca0f0ec47d662dbfcb78a23b546e85843efd5d57248 the receipt is null across multiple nodes. All nodes running on the v19.1.1 zetacored binary

can you share rpc urls of those nodes?

[SMihai97]

yeah, here is the url: https://zetachain-mainnet.public.blastapi.io

[SMihai97]

We have deployed another node using the snapshot provided by the zetachain team and we are facing the same issue, here are the config files: https://gist.github.com/SMihai97/d85d90dcaec1bf4ed5fb632dd63ada99 -> app.toml https://gist.github.com/SMihai97/4a99f0e019cf13bd737d35e96a419760 -> config.toml

[CryptoFewka]

Two questions @SMihai97,

• Is there any kind of proxy in between the caller and the node?
• On the docs page listing snapshots, did you use the full node snapshot or did you instead use the archive node snapshot?

[SMihai97]

Yes, there is a proxy between the caller and the node, but the problem persists even if the call is made directly on the node.

Yes, the archive snapshot was used for redeploy.

[skosito]

Closing this one since the cause was identified: enable-indexer flag in json-rpc section in config.toml should be set to false. This will default to tendermint indexer being used in rpc methods. If this flag is set to true, custom indexer is used - we should follow up to see if we somehow disable this completely, but that can be separate issue.

Thanks for reporting @SMihai97.