Fix LevelDB panic when shutting down the node

lightstreams-network / lightchain

Fast proof-of-authority blockchain based on go-ethereum and tendermint

https://lightstreams.network

GNU General Public License v3.0

55 stars 16 forks source link

Fix LevelDB panic when shutting down the node #210

Open EnchanterIO opened 4 years ago

EnchanterIO commented 4 years ago

How to reproduce

Boot up a new no-synced node
Before node is fully sync force a shut down (CTRL+C)

Steps to resolve:

Shutting down the node should close all goroutines gracefully

ggarri commented 4 years ago

Current status After the first research over this issue I could narrow the issue to the scenario where a node being shut down triggers tendermint dbs to close down and it appears to happen that the tendermint stateDb is closed and Tendermint node try to persist a block right after that happens. This is a racing condition and it does not happen on every shut down.

Next steps Try to validate the potential solution of stopping Tendermint ABCI reactor before DBs are closed

EnchanterIO commented 4 years ago

Seems like the 1s timeout should be enough for persisting the changes to disk. If it fails we increase it or implement a proper solution. I am bumping the version and will test it for few days on my node. If everything goes fine we merge it by the end of the month 👍