search

seznam / euphoria

Euphoria is an open source Java API for creating unified big-data processing flows. It provides an engine independent programming model which can express both batch and stream transformations.
Apache License 2.0
82 stars 11 forks source link

Local executor checkpointing #263

Open dmvk opened 6 years ago

dmvk commented 6 years ago

For streaming apps, we should implement simple checkpointing for LocalExecutor. Eg. when reading from kafka, we want to resume from last commited offset, so we can restart the app.

je-ik commented 6 years ago

Implementation note: we should periodically persist state to local disk and add parameter to continue processing from given checkpointId.