search

keep-network / tbtc

Trustlessly tokenized Bitcoin on Ethereum ;)
https://tbtc.network
MIT License
214 stars 45 forks source link

Compact history job for ElectrumX, statefulset services host for rpc set to 0.0.0.0 #821

Closed pdyraga closed 3 years ago

pdyraga commented 3 years ago

ElectrumX instances can enter a state where their index is too large and requires out-of-band compacting, at which point the ElectrumX server will crash and fail to restart. The job added here can be run once and will compact the history.

It needs to be pointed to the correct volume via the persistentVolumeClaim, and is designed to run only once to completion---successful or not. To run again later, the job can be deleted from the GCP UI or using kubectl, and then re-applied.

Additionally, changed RPC host in statefulset services to 0.0.0.0. Electrum could not bind to port 8000 when we were using rpc://localhost:8000 but it binds just well if we use rpc://0.0.0.0:8000. See https://github.com/lukechilds/docker-electrumx/issues/25#issuecomment-492957782

The problem is revealed by the electrumx server with a log message: 

Traceback (most recent call last):
  File "/electrumx/electrumx_server", line 35, in main
    asyncio.run(controller.run())
  File "/usr/local/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "uvloop/loop.pyx", line 1501, in uvloop.loop.Loop.run_until_complete
  File "/electrumx/electrumx/lib/server_base.py", line 129, in run
    await server_task
  File "/electrumx/electrumx/lib/server_base.py", line 102, in serve
    await self.serve(shutdown_event)
  File "/electrumx/electrumx/server/controller.py", line 134, in serve
    await group.spawn(wait_for_catchup())
  File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.7-py3.7.egg/aiorpcx/curio.py", line 255, in __aexit__
    await self.join()
  File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.7-py3.7.egg/aiorpcx/curio.py", line 213, in join
    raise task.exception()
  File "/electrumx/electrumx/server/block_processor.py", line 702, in fetch_and_process_blocks
    await group.spawn(self._process_prefetched_blocks())
  File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.7-py3.7.egg/aiorpcx/curio.py", line 255, in __aexit__
    await self.join()
  File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.7-py3.7.egg/aiorpcx/curio.py", line 213, in join
    raise task.exception()
  File "/electrumx/electrumx/server/block_processor.py", line 663, in _process_prefetched_blocks
    await self.check_and_advance_blocks(blocks)
  File "/electrumx/electrumx/server/block_processor.py", line 230, in check_and_advance_blocks
    await self._maybe_flush()
  File "/electrumx/electrumx/server/block_processor.py", line 368, in _maybe_flush
    await self.flush(flush_arg)
  File "/electrumx/electrumx/server/block_processor.py", line 358, in flush
    await self.run_in_thread_with_lock(flush)
  File "/electrumx/electrumx/server/block_processor.py", line 212, in run_in_thread_with_lock
    return await asyncio.shield(run_in_thread_locked())
  File "/electrumx/electrumx/server/block_processor.py", line 211, in run_in_thread_locked
    return await run_in_thread(func, *args)
  File "/usr/local/lib/python3.7/site-packages/aiorpcX-0.18.7-py3.7.egg/aiorpcx/curio.py", line 68, in run_in_thread
    return await get_event_loop().run_in_executor(None, func, *args)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/electrumx/electrumx/server/block_processor.py", line 357, in flush
    self.estimate_txs_remaining)
  File "/electrumx/electrumx/server/db.py", line 244, in flush_dbs
    self.flush_history()
  File "/electrumx/electrumx/server/db.py", line 313, in flush_history
    self.history.flush()
  File "/electrumx/electrumx/server/history.py", line 160, in flush
    flush_id = pack_be_uint16(self.flush_count)
struct.error: 'H' format requires 0 <= number <= 65535