search

kyuupichan / electrumx

Alternative implementation of spesmilo/electrum-server
Other
720 stars 732 forks source link

server.block_processor.ChainError: UTXO 000... / 0 not found in "h" table #323

Closed shsmith closed 6 years ago

shsmith commented 6 years ago

I am attempting to update a BitcoinCash node to the version 6 database format. The rebuild fails hard with this traceback:

INFO:BlockProcessor:our height: 234,946 daemon: 506,077 UTXOs 827MB hist 152MB
ERROR:Controller:Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/electrumx-1.2.1-py3.6.egg/server/controller.py", line 177, in on_future_done
    future.result()
  File "/usr/local/lib/python3.6/dist-packages/electrumx-1.2.1-py3.6.egg/server/block_processor.py", line 206, in main_loop
    await task()
  File "/usr/local/lib/python3.6/dist-packages/electrumx-1.2.1-py3.6.egg/server/block_processor.py", line 250, in check_and_advance_blocks
    await self.controller.run_in_executor(self.advance_blocks, blocks)
  File "/usr/local/lib/python3.6/dist-packages/electrumx-1.2.1-py3.6.egg/server/controller.py", line 157, in run_in_executor
    return await self.loop.run_in_executor(None, func, *args)
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/dist-packages/electrumx-1.2.1-py3.6.egg/server/block_processor.py", line 502, in advance_blocks
    undo_info = self.advance_txs(block.transactions)
  File "/usr/local/lib/python3.6/dist-packages/electrumx-1.2.1-py3.6.egg/server/block_processor.py", line 543, in advance_txs
    cache_value = spend_utxo(txin.prev_hash, txin.prev_idx)
  File "/usr/local/lib/python3.6/dist-packages/electrumx-1.2.1-py3.6.egg/server/block_processor.py", line 728, in spend_utxo
    .format(hash_to_str(tx_hash), tx_idx))
server.block_processor.ChainError: UTXO 0000000000000000000000000000000000000000000000000000000000000000 / 0 not found in "h" table

At this point the output stops and CPU load is 0. After several minutes of waiting I pressed ^C and got this output:

^CWARNING:Controller:received SIGINT signal, initiating shutdown
INFO:Controller:shutting down
INFO:Controller:closing down RPC listening servers
INFO:BlockProcessor:flushing state to DB for a clean shutdown...
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/electrumx-1.2.1-py3.6.egg/EGG-INFO/scripts/electrumx_server.py", line 25, in main
  File "/usr/local/lib/python3.6/dist-packages/electrumx-1.2.1-py3.6.egg/lib/server_base.py", line 127, in run
    loop.run_until_complete(self._wait_for_shutdown_event())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 467, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.6/dist-packages/electrumx-1.2.1-py3.6.egg/lib/server_base.py", line 81, in _wait_for_shutdown_event
    await self.shutdown()
  File "/usr/local/lib/python3.6/dist-packages/electrumx-1.2.1-py3.6.egg/server/controller.py", line 110, in shutdown
    self.bp.shutdown(self.executor)
  File "/usr/local/lib/python3.6/dist-packages/electrumx-1.2.1-py3.6.egg/server/block_processor.py", line 215, in shutdown
    self.flush(True)
  File "/usr/local/lib/python3.6/dist-packages/electrumx-1.2.1-py3.6.egg/server/block_processor.py", line 368, in flush
    self.fs_flush()
  File "/usr/local/lib/python3.6/dist-packages/electrumx-1.2.1-py3.6.egg/server/block_processor.py", line 426, in fs_flush
    self.fs_update(self.fs_height, self.headers, self.tx_hashes)
  File "/usr/local/lib/python3.6/dist-packages/electrumx-1.2.1-py3.6.egg/server/db.py", line 196, in fs_update
    assert len(block_tx_hashes) == blocks_done
AssertionError
CRITICAL:root:ElectrumX server terminated abnormally
shsmith@abc2:~$

If I start electrumx again it starts at height 202,490 but fails again with the same tracebacks.

I tried deleting the new db to completely start over but it gets stuck at the same place on the next try.

Things were working ok with the previous database version and ElectrumX 1.1.1

kyuupichan commented 6 years ago

That's weird. Is this reproducible on a different machine? Is this ElectrumX 1.2 as tagged (there are many commits since then)

shsmith commented 6 years ago

This is happening with git HEAD tagged 1.2.1. I had similar issues back when 1.2 came out, which is why abc1.hsmiths.com is still running 1.1.x. I am attempting to do the rebuild in a clone of that environment.

I suspect maybe it is the bitcoin database causing this. I am reindexing to see if that makes a difference.

Is there a foundry for the ABC blockchain and electrumx DB somewhere? Building these from zero is a barrier to spinning up new nodes.

kyuupichan commented 6 years ago

Ah yes JWU42 had that issue - the bitcoind DB was corrupt and he couldn't sync. He resync-ed bitcoind and all was fine.

kyuupichan commented 6 years ago

I'm not sure if there's a way to determine bitcoind "lying"; the base assumption of ElectrumX is that the bitcoin node is reliable and friendly. Its data is largely assumed good.

kyuupichan commented 6 years ago

I don't think this kind of chain-sync logic has changed for a long time in e-x. Most changes have been to networking or other layers on top that should not affect the DB.

kyuupichan commented 6 years ago

Note your hash was all zeroes. IMO this is more likely a DB driver bug or an O/S bug than a bitcoind or e-x bug. But let us know how a resync of bitcoind helps with everything else the same.

shsmith commented 6 years ago

I let electrumX sync while bitcoind was reindexing and it made it past height 234,946. Seems likely it was indeed bitcoind index corruption.