search

kyuupichan / electrumx

Alternative implementation of spesmilo/electrum-server
Other
724 stars 733 forks source link

Crash with error "IOError(b'IO error: hist/198381.ldb: Invalid argument')" #1033

Closed pedr0-fr closed 3 years ago

pedr0-fr commented 4 years ago

I usually run a electrumx v1.13.0 server connected to bitcoin core v0.20 without any issues. Today it crashed with this message in the log:

Sep 17 20:34:17 lenovoW541 electrumx_server[26249]: INFO:ElectrumX:[770] TCP 127.0.0.1:54804, 1 total
Sep 17 20:34:23 lenovoW541 electrumx_server[26249]: INFO:Prefetcher:cancelled; prefetcher stopping
Sep 17 20:34:23 lenovoW541 electrumx_server[26249]: INFO:BlockProcessor:flushing to DB for a clean shutdown...
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]: INFO:DB:flush #3,135 took 0.2s.  Height 648,789 txs: 569,294,041 (-2,385)
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]: INFO:SessionManager:closing down server for rpc://localhost:8000
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]: INFO:SessionManager:closing down server for tcp://localhost:50001
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]: ERROR:root:task crashed: <Task finished coro=<Controller.serve() done, defined at /media/pedro/Seagate_4T/bitcoin/electrumx/electrumx/server/controller.py:81> exception=IOError(b'IO error: hist/198381.ldb: Invalid argument',)>
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]: Traceback (most recent call last):
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:   File "/usr/local/lib/python3.6/dist-packages/aiorpcX-0.18.4-py3.6.egg/aiorpcx/util.py", line 309, in check_task
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:     task.result()
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:   File "/media/pedro/Seagate_4T/bitcoin/electrumx/electrumx/server/controller.py", line 134, in serve
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:     await group.spawn(wait_for_catchup())
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:   File "/usr/local/lib/python3.6/dist-packages/aiorpcX-0.18.4-py3.6.egg/aiorpcx/curio.py", line 242, in __aexit__
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:     await self.join()
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:   File "/usr/local/lib/python3.6/dist-packages/aiorpcX-0.18.4-py3.6.egg/aiorpcx/curio.py", line 211, in join
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:     raise task.exception()
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:   File "/media/pedro/Seagate_4T/bitcoin/electrumx/electrumx/server/block_processor.py", line 665, in fetch_and_process_blocks
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:     await group.spawn(self._process_prefetched_blocks())
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:   File "/usr/local/lib/python3.6/dist-packages/aiorpcX-0.18.4-py3.6.egg/aiorpcx/curio.py", line 242, in __aexit__
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:     await self.join()
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:   File "/usr/local/lib/python3.6/dist-packages/aiorpcX-0.18.4-py3.6.egg/aiorpcx/curio.py", line 211, in join
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:     raise task.exception()
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:   File "/media/pedro/Seagate_4T/bitcoin/electrumx/electrumx/server/block_processor.py", line 626, in _process_prefetched_blocks
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:     await self.check_and_advance_blocks(blocks)
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:   File "/media/pedro/Seagate_4T/bitcoin/electrumx/electrumx/server/block_processor.py", line 226, in check_and_advance_blocks
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:     await self.reorg_chain()
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:   File "/media/pedro/Seagate_4T/bitcoin/electrumx/electrumx/server/block_processor.py", line 269, in reorg_chain
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:     await self.run_in_thread_with_lock(flush_backup)
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:   File "/media/pedro/Seagate_4T/bitcoin/electrumx/electrumx/server/block_processor.py", line 198, in run_in_thread_with_lock
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:     return await asyncio.shield(run_in_thread_locked())
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:   File "/media/pedro/Seagate_4T/bitcoin/electrumx/electrumx/server/block_processor.py", line 197, in run_in_thread_locked
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:     return await run_in_thread(func, *args)
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:   File "/usr/local/lib/python3.6/dist-packages/aiorpcX-0.18.4-py3.6.egg/aiorpcx/curio.py", line 68, in run_in_thread
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:     return await get_event_loop().run_in_executor(None, func, *args)
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:   File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:     result = self.fn(*self.args, **self.kwargs)
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:   File "/media/pedro/Seagate_4T/bitcoin/electrumx/electrumx/server/block_processor.py", line 261, in flush_backup
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:     self.db.flush_backup(self.flush_data(), self.touched)
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:   File "/media/pedro/Seagate_4T/bitcoin/electrumx/electrumx/server/db.py", line 343, in flush_backup
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:     self.history.backup(touched, flush_data.tx_count)
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:   File "/media/pedro/Seagate_4T/bitcoin/electrumx/electrumx/server/history.py", line 161, in backup
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:     for key, hist in self.db.iterator(prefix=hashX, reverse=True):
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:   File "plyvel/_plyvel.pyx", line 831, in plyvel._plyvel.Iterator.__next__
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:   File "plyvel/_plyvel.pyx", line 938, in plyvel._plyvel.Iterator.real_prev
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]:   File "plyvel/_plyvel.pyx", line 88, in plyvel._plyvel.raise_for_status
Sep 17 20:34:24 lenovoW541 electrumx_server[26249]: plyvel._plyvel.IOError: b'IO error: hist/198381.ldb: Invalid argument'
kyuupichan commented 3 years ago

No idea; if this repeats perhaps your DB is corrupt