The plan here is from the docs. Problem is that we're eagerly evaluating state at every point which is not particularly efficient (TBD, however). This also helps us solve this in a clean way: https://github.com/DAGWorks-Inc/burr/issues/28.
We plan the ability to manage state in a few ways:
commit -- an internal tool to commit/compile a series of changes so that we have the latest state evaluated
persist -- a user-facing API to persist state to a database. This will be pluggable by the user, and we will have a few built-in options (e.g. a simple in-memory store, a file store, a database store, etc...)
hydrate -- a static method to hydrate state from a database. This will be pluggable by the user, and we will have a few built-in options that mirror those in persist options.
Currently state is immutable, but it utilizes an inefficient copy mechanism. This is out of expedience -- we don't anticipate this will
be painful for the time being, but plan to build a more efficient functional paradigm. We will likely have:
Each state object be a node in a linked list, with a pointer to the previous state. It carries a diff of the changes from the previous state.
An ability to checkpoint (allowing for state garbage collection), and store state in memory/kill out the pointers.
We will also consider having the ability to have a state solely backed by redis (and not memory), but we are still thinking through the API.
The plan here is from the docs. Problem is that we're eagerly evaluating state at every point which is not particularly efficient (TBD, however). This also helps us solve this in a clean way: https://github.com/DAGWorks-Inc/burr/issues/28.
We plan the ability to manage state in a few ways:
commit
-- an internal tool to commit/compile a series of changes so that we have the latest state evaluatedpersist
-- a user-facing API to persist state to a database. This will be pluggable by the user, and we will have a few built-in options (e.g. a simple in-memory store, a file store, a database store, etc...)hydrate
-- a static method to hydrate state from a database. This will be pluggable by the user, and we will have a few built-in options that mirror those inpersist
options.Currently state is immutable, but it utilizes an inefficient copy mechanism. This is out of expedience -- we don't anticipate this will be painful for the time being, but plan to build a more efficient functional paradigm. We will likely have:
checkpoint
(allowing for state garbage collection), and store state in memory/kill out the pointers.We will also consider having the ability to have a state solely backed by redis (and not memory), but we are still thinking through the API.