search

Zannick / logic-graph

Tools for video game logic representation and analysis, particularly routing and beatability checks for speedruns and randomizers.
MIT License
3 stars 0 forks source link

Dump the heap to a file / restore from disk #52

Closed Zannick closed 1 year ago

Zannick commented 1 year ago

The heap can get very large over time; we should dump some or most of the states in the heap to disk to keep memory usage manageable. Ideally we can serialize with serde, and compress with zstd or similar.

Zannick commented 1 year ago

Success! But unfortunately at the cost of around 80s per million states backed up, which is significant. Even if we make that asynchronous, we're likely to need to back up again before the previous one finishes.

Similarly, restoring later may find swaths of the backed up data are duplicates or expired on max_time, but it still takes a long time to read and decompress it just to find that out... so also a good candidate for a background operation. Especially as we serialize a giant vector into a single file and read it back into a giant vector as well... which is running me out of memory all the same as having too many things in the heap. So that also needs improvement, whether it's serializing individual entries or outputting chunks so less needs to be allocated at once on loading from disk.

It would be great if this heap could be like an LRU and bump off elements as it gets too large, then we could move those into another struct or an mmap'ed file.

Zannick commented 1 year ago

I suppose we should be using a database instead of rolling our own persistence layer. 😅