search

yoiqsram / convenience-store-simulator

Agent-based modeling to simulate convenience store transaction data
0 stars 0 forks source link

Scalability Issue: Memory leak when dumping simulator session #2

Open yoiqsram opened 3 weeks ago

yoiqsram commented 3 weeks ago

Simulator could take up to 8 GB of RAM when dumping the checkpoint session when testing with market population size of 1 Mio. Another approach has to be made to save the checkpoint. Instead of session, we could serialize the data from the environment and agents to an unstructured data type. The easiest option is dump to JSON file or maybe to a NoSQL database using TinyDB for the proof of concept.

However, continuous update will increase the computation cost which is bad for the main purpose of this project to simulate data for long period. We will still keep the checkpoint; thus, we also need to resolve the database to roll back to the same checkpoint as the simulator.

yoiqsram commented 3 weeks ago

I think NoSQL database may be a better choice. If we want use TinyDB, we could split the file based on store. Currently we only run by single thread. If we split the data to different files, we could load on different thread, even different processes.