Electrumx stuck processing something..? Eats all the memory and gets killed by OOM killer

[Overtorment]

I have 2 electrumx servers going down today, with similar pattern.

# ./run-electrum.sh
INFO:electrumx:ElectrumX server starting
WARNING:Env:lowered maximum sessions from 100,000 to 98,650 because your open file limit is 99,000
INFO:electrumx:logging level: INFO
INFO:Controller:Python version: 3.8.10 (default, Jun 22 2022, 20:18:18)  [GCC 9.4.0]
INFO:Controller:software version: ElectrumX 1.16.0
INFO:Controller:aiorpcX version: 0.18.5
INFO:Controller:supported protocol versions: 1.4-1.4.2
INFO:Controller:event loop policy: None
INFO:Controller:reorg limit is 200 blocks
INFO:Daemon:daemon #1 at 127.0.0.1:8332/ (current)
INFO:DB:switching current directory to /home/overtorment/electrum_datadir/
INFO:DB:using leveldb for DB backend
./run-electrum.sh: line 20: 1698456 Killed                  ./electrumx/electrumx_server

Will try to gather more info and post here.
Is it happening for anyone else..?

[Overtorment]

when i gave it big enough swap:

INFO:BlockProcessor:our height: 764,158 daemon: 764,158 UTXOs 16MB hist 16MB
INFO:BlockProcessor:processed 7 blocks size 10.77 MB in 125.9s
INFO:BlockProcessor:caught up to height 764158
INFO:DB:flushed filesystem data in 0.01s
INFO:Prefetcher:cancelled; prefetcher stopping
INFO:Controller:shutting down
INFO:Controller:shutdown complete
ERROR:electrumx:ElectrumX server terminated abnormally
Traceback (most recent call last):
  File "./electrumx/electrumx_server", line 35, in main
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/home/overtorment/electrumx/electrumx/lib/server_base.py", line 129, in run
    await server_task
  File "/home/overtorment/electrumx/electrumx/lib/server_base.py", line 102, in serve
    await self.serve(shutdown_event)
  File "/home/overtorment/electrumx/electrumx/server/controller.py", line 134, in serve
    await group.spawn(wait_for_catchup())
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/curio.py", line 242, in __aexit__
    await self.join()
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/curio.py", line 211, in join
    raise task.exception()
  File "/home/overtorment/electrumx/electrumx/server/block_processor.py", line 702, in fetch_and_process_blocks
    await group.spawn(self._process_prefetched_blocks())
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/curio.py", line 242, in __aexit__
    await self.join()
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/curio.py", line 211, in join
    raise task.exception()
  File "/home/overtorment/electrumx/electrumx/server/block_processor.py", line 654, in _process_prefetched_blocks
    await self._first_caught_up() 
  File "/home/overtorment/electrumx/electrumx/server/block_processor.py", line 670, in _first_caught_up
    await self.flush(True)
  File "/home/overtorment/electrumx/electrumx/server/block_processor.py", line 358, in flush
    await self.run_in_thread_with_lock(flush)
  File "/home/overtorment/electrumx/electrumx/server/block_processor.py", line 212, in run_in_thread_with_lock
    return await asyncio.shield(run_in_thread_locked())
  File "/home/overtorment/electrumx/electrumx/server/block_processor.py", line 211, in run_in_thread_locked
    return await run_in_thread(func, *args)
  File "/usr/local/lib/python3.8/dist-packages/aiorpcx/curio.py", line 68, in run_in_thread
    return await get_event_loop().run_in_executor(None, func, *args)
  File "/usr/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/overtorment/electrumx/electrumx/server/block_processor.py", line 356, in flush
    self.db.flush_dbs(self.flush_data(), flush_utxos,
  File "/home/overtorment/electrumx/electrumx/server/db.py", line 244, in flush_dbs
    self.flush_history()
  File "/home/overtorment/electrumx/electrumx/server/db.py", line 313, in flush_history
    self.history.flush()
  File "/home/overtorment/electrumx/electrumx/server/history.py", line 160, in flush
    flush_id = pack_be_uint16(self.flush_count)
struct.error: 'H' format requires 0 <= number <= 65535

[Overtorment]

nevermind, i think it just needs compaction