State Management

[jspenger]

The Portals runtime currently only supports in-memory state, and does not support checkpointing/recovering state. We should fix this, and implement a suitable persistent state backend. RocksDB is a good choice for this, as it is widely used by other stream processing systems.

The tasks for this issue are the following:

• [ ] Create a state backend, which can checkpoint and recover.
• [ ] Test the state backend.
• [x] #97
• [ ] Make runtime checkpoint state at regular intervals.
• [ ] Recover from previous state after a restart.
• [x] #100
• [ ] #146

Notes:

• The new state backend should implement the StateBackend trait, a good example of this is the MapStateBackendImpl.
• I have previously had issues with RocksDB on the new M1 Macbook, this may again be an issue, just something to be aware of.
• Adding checkpointing and recovery, and enabling the state backend, should be made available through some config settings in the runtime.
• The new state backend should be put in a new sub-project, as to keep the core pure Scala (so it compiles with ScalaJS). This can be done as a last step, and we should probably discuss together how to do it best, once we have all the pieces; still, I would start of working directly form the core directory.

[jspenger]

Due to some of the issues that have appeared during this issue, such as serialization similar, we will isolate this new work to a branch rocksdb-experiment. I suggest we continue working on that branch, it is ok for now to break any of the tests and examples on the branch. Before merging the new changes to main, however, we will do a larger review of the changes to cleanup the code and make sure that it works, see #164. Also, we should make sure to squash all commits for that.