search

baking-bad / tzkt

😼 Awesome Tezos blockchain indexer and API
https://tzkt.io
MIT License
183 stars 35 forks source link

Indexer got stuck in reverting status #149

Closed puppetninja closed 1 year ago

puppetninja commented 1 year ago

Hi,

Somehow our mainnet tzkt got stuck in reverting status and become out of sync...

puppetninja commented 1 year ago

Screenshot from 2023-04-04 18-29-14

puppetninja commented 1 year ago

Once we restart the indexer, it is not recovering also Screenshot from 2023-04-04 18-48-19

puppetninja commented 1 year ago

we have only one rolling node to sync with

puppetninja commented 1 year ago

after rebooting seems it is back...I would assume tzkt is kinda self healing, please feel free to close it if that is the case, sorry for the inconvience

Groxan commented 1 year ago

Great! Yeah, reverting is a more complicated process and in some specific cases may take longer time. So, basically, if there are no errors in the logs, then the indexer is not stuck but likely waiting for the DB, executing some slow query.

I also saw your message about debug log level. Have you managed to enable it? Setting Logging__LogLevel__Default actually should work. Note, those are double underscores.

puppetninja commented 1 year ago

Hi @Groxan yeah, setting log level via env var actually works, I think my concern was that tzkt becomes out of sync when the above symptoms happens. I will close it for now

puppetninja commented 1 year ago

Hi @Groxan unfortunately I am seeing this issue again, I bumped db timeout from 600 to 3000 and still hitting this error

Screenshot from 2023-05-30 19-27-47

like you said if there are no errors in the logs, but I do see that sync is stucked with fatal log level, any pointer would be appreciated...

Groxan commented 1 year ago

The record in the logs fail: Invalid head. Reverting... means that there was chain reorg, and the latest block is no longer valid and will be reverted. Then you see debug: Revert block - this means that the indexer is in process of reverting the block. So, as you can see, it's not stuck with fatal.

Try to check Postgres' logs to see what happens there and on which SQL query the DB is stuck.

puppetninja commented 1 year ago

Thanks @Groxan much appreciate it !