search

lbryio / lbry-sdk

The LBRY SDK for building decentralized, censorship resistant, monetized, digital content apps.
https://lbry.com
MIT License
7.19k stars 483 forks source link

recover gracefully from reorg failure #1497

Closed tzarebczan closed 3 years ago

tzarebczan commented 6 years ago
## The Issue Although re-org failures should not occur after https://github.com/lbryio/torba/pull/14, we should make sure that the daemon can automatically recover from a re-org event. Currently, you'd see the below error and would need to delete your headers file + restart manually. `IndexError: Blockchain reorganization dropped 100 headers. This is highly unusual. Will not continue to attempt reorganizing. Please, delete the ledger synchronization directory inside your wallet directory (folder: 'lbc_mainnet') and restart the program to synchronize from scratch.` ## Solution Simple solution: Tell the user to wipe their headers and sync from scratch. Better solution: Do that for them. Additionally, add a more recent checkpoint so its faster to do the sync.
alyssaoc commented 5 years ago

@tzarebczan Is this still happening?

tzarebczan commented 4 years ago

A user recently ran into this and could not start up LBRY. Headers file: https://mega.nz/#!nbYXzSCR!YYe1ujYcn8nPpiRxM_Wf_eFRoaJl5SQ1b-jUDlcKU-8

It should at least be able to recover on its own by resyncing/deleting headers. 

2019-11-24 17:41:10,839 ERROR    lbry.extras.daemon.Component:62: Error setting up wallet
Traceback (most recent call last):
  File "lbry/extras/daemon/Component.py", line 55, in _setup
  File "lbry/extras/daemon/Components.py", line 260, in start
  File "torba/client/basemanager.py", line 62, in start
  File "lbry/wallet/ledger.py", line 106, in start
  File "torba/client/baseledger.py", line 288, in start
  File "torba/client/baseledger.py", line 294, in join_network
  File "torba/client/baseledger.py", line 367, in update_headers
IndexError: Blockchain reorganization dropped 100 headers. This is highly unusual. Will not continue to attempt reorganizing. Please, delete the ledger synchronization directory inside your wallet directory (folder: 'lbc_mainnet') and restart the program to synchronize from scratch.
2019-11-24 10:01:00,173 INFO     lbry.extras.daemon.Components:180: remote height: 668566, local height: 671645
2019-11-24 10:01:00,367 INFO     lbry.extras.daemon.Components:220: Header checkpoint verified.
2019-11-24 10:01:00,876 INFO     lbry.extras.daemon.Components:258: Starting torba wallet
2019-11-24 10:01:01,130 INFO     lbry.extras.daemon.Components:336: start the dht
2019-11-24 10:01:01,131 INFO     lbry.extras.daemon.Components:360: Started the dht
2019-11-24 10:01:01,131 INFO     torba.client.basedatabase:235: connecting to database: /home/michael/.local/share/lbry/lbryum/lbc_mainnet/blockchain.db
2019-11-24 10:01:01,132 INFO     lbry.dht.node:119: DHT node listening on UDP 0.0.0.0:4444
2019-11-24 10:01:01,234 INFO     torba.client.basenetwork:314: connected to spv7.lbry.com:50001
2019-11-24 10:01:01,234 INFO     torba.client.basenetwork:175: Switching to SPV wallet server: spv7.lbry.com:50001
2019-11-24 10:01:01,235 INFO     torba.client.baseledger:292: Subscribing and updating accounts.
2019-11-24 10:01:01,256 INFO     torba.client.basenetwork:179: Subscribed to headers: spv7.lbry.com:50001
2019-11-24 10:01:01,258 WARNING  torba.client.baseledger:349: Blockchain Reorganization: attempting rewind to height 671644 from starting height 671645
2019-11-24 10:01:01,282 WARNING  torba.client.baseledger:349: Blockchain Reorganization: attempting rewind to height 671643 from starting height 671645
2019-11-24 10:01:01,308 WARNING  torba.client.baseledger:349: Blockchain Reorganization: attempting rewind to height 671642 from starting height 671645
2019-11-24 10:01:01,333 WARNING  torba.client.baseledger:349: Blockchain Reorganization: attempting rewind to height 671641 from starting height 671645