rixed / ramen

A stream processing language and compiler for small-scale monitoring
Other
14 stars 4 forks source link

Replay past data when starting a new worker to initialize function state #1449

Open rixed opened 2 years ago

rixed commented 2 years ago

A replay request will replay further back in history than the requested range to make sure the first point of the requested range will account for the recent past of that function. Similarly when starting a new worker, we ideally want to start from where the function would be if it had started earlier.

In practice, it means that when starting a worker supervisor should also start past replayers (but not for the started worker itself), and the started worker would first act as a replayer before transitioning to normal front time worker, with its internal state.

Alternatively, we could have a normal replay up to the front time, and the snapshotted internal state of the last worker be transfered to the new worker.