search

near / queryapi

Near Indexing as a Service
18 stars 3 forks source link

Redis memory flooded on initial start up #852

Open morgsmccauley opened 4 months ago

morgsmccauley commented 4 months ago

This seemed to happen after restarting Runner after a few hours downtime. Most likely related to the expirable streamer message cache written by Block Streamer: All Redis Streams cap out when Runner is down, upon restart, Redis Stream is drained, therefore the Block Streams start back up, causing multiple cache writes. Needs to investigated further:

image
morgsmccauley commented 3 months ago

This appears to happen after the block streams get backed up for an extended period of time.

This doesn't result in system failure, only degraded performance. The confiugredmaxmemory-policy of volatile-lru will evict all expirable keys, which are the cause of this memory spike.