search

cosmos / gaia

Cosmos Hub
https://hub.cosmos.network
Apache License 2.0
471 stars 690 forks source link

Loss transactions in full node database #2889

Open tdnguyenND opened 9 months ago

tdnguyenND commented 9 months ago

Problem

Some transactions are missing (not from latest block)

Closing criteria

Deploy full node successfully with no missing transactions

Problem details

When I deploy a full node cosmos, I found that some transactions are missing from my database (i.e 02940919FFC8E9A2B153654EA104AA041B603719C2A50A73F8A28BED1EAF109D - which I found exists in mintscan. For data analyzing purpose, I expect my database to have all the neccessary transaction. So the question is how can I download the missing transaction?

How to reproduce this problems?

  1. Install gaiad & cosmovisor. I'm using version 13.0.2 image
  2. Download archieved data source from quicksync. At the moment I downloaded, it was 8.5TB, sadly I don't remember the latest block when downloading it
  3. Unzip downloaded file and put in into the correct database folder
  4. Start gaiad app
  5. Run search for transaction 
    curl http://localhost:26657/tx?hash=0xAA9AC07139812CFF0BBC01B3B0CDA89474B2470E8A72B0750B4AE94B3B4CF2FD

    Expected result: Return transaction detail. Actual result: {"jsonrpc":"2.0","id":-1,"error":{"code":-32603,"message":"Internal error","data":"tx (AA9AC07139812CFF0BBC01B3B0CDA89474B2470E8A72B0750B4AE94B3B4CF2FD) not found"}}

In fact, all 3 transactions from block 7412749 are missed. Although they're all searchable at mintscan, I can't find it from any other explorer platform (maybe mintscan just simply decode from block data, i guess?)

mmulji-ic commented 9 months ago

Thanks @tdnguyenND for reporting, I'll check with the Comet team why this has occurred, they're supporting the db and p2p layers. I'm assuming that you have transactions before and after this transaction. Also how did you stumble across this transaction that was missing?

andynog commented 9 months ago

Hi @tdnguyenND, I'm Andy from the CometBFT team, in order to help @mmulji-ic troubleshoot this, could you please let us know how you're querying for this transaction using your full node ?

tdnguyenND commented 9 months ago

@andynog I have provide more detail in problem description. Hope it help

andynog commented 9 months ago

Thanks @tdnguyenND, what do you get when you query the block ?

curl http://localhost:26657/block?height=7412749

tdnguyenND commented 9 months ago

The block is still fine with all txns included 7412749.json

mmulji-ic commented 8 months ago

@andynog any further thoughts?

andynog commented 7 months ago

https://www.mintscan.io/cosmos/block/7412749 doesn't have the tx hash you mentioned 0xAA9AC07139812CFF0BBC01B3B0CDA89474B2470E8A72B0750B4AE94B3B4CF2FD

tdnguyenND commented 6 months ago

https://www.mintscan.io/cosmos/block/7412749 doesn't have the tx hash you mentioned 0xAA9AC07139812CFF0BBC01B3B0CDA89474B2470E8A72B0750B4AE94B3B4CF2FD

Maybe I was wrongly referring the command, but I'm sure that some transaction were somehow removed from quicksync's archieved file - i.e 02940919FFC8E9A2B153654EA104AA041B603719C2A50A73F8A28BED1EAF109D (I have a list of transactions that are missed)

Unfortunately this happened quite long and I decided to ignore them, my fullnode no longer exists so I can't recreate the scenario.

a26nine commented 5 months ago

I am seeing this as well. For example, 8CFC7A4E2A389A9F7D89C03F458A9A25F763ACF40159B55E57586C1E625875D6 exisats on https://www.mintscan.io/cosmos/tx/8CFC7A4E2A389A9F7D89C03F458A9A25F763ACF40159B55E57586C1E625875D6?height=15213800 , but the node gives : {"jsonrpc":"2.0","id":-1,"error":{"code":-32603,"message":"Internal error","data":"tx (8CFC7A4E2A389A9F7D89C03F458A9A25F763ACF40159B55E57586C1E625875D6) not found"}}