search

terra-money / fcd-classic

Terra ETL + RestFul API Server
https://fcd.terra.dev/swagger
MIT License
63 stars 42 forks source link

FCD failing after restart #146

Closed muratso closed 2 years ago

muratso commented 2 years ago

Hey folks, I'm using the branch bombay (since it contains some variables that I needed) and I could sync the whole columbus-5 just fine. However, after a restart I'm getting the following error

12-06 17:16 [ERROR]: Cannot destructure property 'tx_response' of '(intermediate value)' as it is undefined.
TypeError: Cannot destructure property 'tx_response' of '(intermediate value)' as it is undefined.
    at Object.getTx (/app/src/lib/lcd.ts:55:11)
    at async generateLcdTransactionToTxEntity (/app/src/collector/block/tx.ts:166:14)

Any ideas on how to fix/workaround this?

hanjukim commented 2 years ago

I think your node is not responding required transaction. https://github.com/terra-money/fcd/blob/4d190416e719f3659947879aa4684701e08fabe2/src/lib/lcd.ts#L55

Check the node endpoint /cosmos/tx/v1beta1/txs/{hash}

muratso commented 2 years ago

@hanjukim this was blocking us... so the easiest solution was to simply drop the DB and let it sync from scratch again. We're once again synced... but I'm afraid it will die again if I simply restart it.

As dropping the db did the trick... I don't think this error was anyhow related to the node itself.

hanjukim commented 2 years ago

I think your node data is corrupted. I had a similar experience with one of my node that it didn't indexed transactions in specific block.

muratso commented 2 years ago

@hanjukim I tried to restore from a full archive snapshot and end up getting the same error :(. This time there weren't any restarts or anything. The FCD was progressing and eventually started throwing those errors

hanjukim commented 2 years ago

@muratso Try to query {LCD_ENDPOINT}/cosmos/tx/v1beta1/txs/{hash} manually. Data must be there to continue without having missing data.

muratso commented 2 years ago

Indeed there were missing TXs from the node. But the thing is, the node was restored from an archival snapshot. This means these TXs are probably missing from the snapshot itself. One way or another, the new FCD dump did the trick as it had indexed all the missing txs.