MemoryError (OOM) in advance block

[jackrobison]

Traceback (most recent call last):
  File "/home/lbry/scribe/blockchain/service.py", line 1625, in process_blocks_and_mempool_forever
    raise err
  File "/home/lbry/scribe/blockchain/service.py", line 1620, in process_blocks_and_mempool_forever
    await self.check_and_advance_blocks(blocks)
  File "/home/lbry/scribe/blockchain/service.py", line 212, in check_and_advance_blocks
    txo_count = await self.run_in_thread_with_lock(self.advance_block, block)
  File "/home/lbry/scribe/blockchain/service.py", line 140, in run_in_thread_with_lock
    return await asyncio.shield(run_in_thread_locked())
  File "/home/lbry/scribe/blockchain/service.py", line 139, in run_in_thread_locked
    return await asyncio.get_event_loop().run_in_executor(self._executor, func, *args)
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/lbry/scribe/blockchain/service.py", line 1441, in advance_block
    self.db.prefix_db.commit(self.height, self.tip)
  File "/home/lbry/scribe/db/interface.py", line 209, in commit
    undo_ops = self._op_stack.get_undo_ops()
  File "/home/lbry/scribe/db/revertable.py", line 163, in get_undo_ops
    return b''.join(op.invert().pack() for op in reversed(self))
MemoryError

This error can be recovered from by restarting scribe. The server that encountered this also runs lbrycrd and has limited available memory.

[jackrobison]

scribe_1               | 2022-05-06 11:06:34,399 INFO scribe.db.db:799: loading tx hashes
scribe_1               | 2022-05-06 11:07:41,185 ERROR scribe.service.BlockchainProcessorService:92: unexpected fatal error: 
scribe_1               | Traceback (most recent call last):
scribe_1               |   File "/home/lbry/scribe/service.py", line 87, in _run
scribe_1               |     await self.start()
scribe_1               |   File "/home/lbry/scribe/service.py", line 68, in start
scribe_1               |     await self.db.initialize_caches()
scribe_1               |   File "/home/lbry/scribe/db/db.py", line 863, in initialize_caches
scribe_1               |     await self._read_tx_hashes()
scribe_1               |   File "/home/lbry/scribe/db/db.py", line 804, in _read_tx_hashes
scribe_1               |     self.tx_num_mapping = {
scribe_1               |   File "/home/lbry/scribe/db/db.py", line 805, in <dictcomp>
scribe_1               |     tx_hash: tx_num for tx_num, tx_hash in enumerate(self.total_transactions)
scribe_1               | MemoryError
scribe_1               | 2022-05-06 11:07:41,629 INFO scribe.service.BlockchainProcessorService:76: stopping

[jackrobison]

I think with https://github.com/lbryio/hub/pull/65 merged we (hopefully) have seen the last of OOM crashes