RBMHTechnology / eventuate

Global-scale event sourcing and event collaboration with causal consistency (This project is in maintenance mode. Only critical bugs will be fixed, but there is no more feature development.).
http://rbmhtechnology.github.io/eventuate/
Apache License 2.0
708 stars 99 forks source link

Stream snapshots to disk with constant memory usage #265

Closed jrudolph closed 8 years ago

jrudolph commented 8 years ago

FilesystemSnapshotStore, currently writes snapshots to disk using akka's Serialization.serialize and Serializer.toBinary methods. Both methods return a strict byte array which requires that the whole snapshot needs to be buffered in memory before it can be written to disk.

This means that you need to provision a significant amount of additional memory to your Java process to be able to write snapshots all the time (or otherwise, cleverly orchestrate snapshot creation to make sure that enough memory is available at those times). In the worst case of only one EventsourcedActor that contains all the state it would mean that you need to have a heap double the size of your working set to be able to create snapshots.

krasserm commented 8 years ago

Covered by #144