tremor-rs / tremor-runtime

Main Tremor Project Rust Codebase
https://www.tremor.rs
Apache License 2.0
842 stars 128 forks source link

Migrate stateful operators to use pipeline state mechanism #71

Open anupdhml opened 4 years ago

anupdhml commented 4 years ago

Describe the problem you are trying to solve

Operators that maintain state can be migrated to use the pipeline state mechanism (eg: for the batch or backpressure operator) -- this would be a necessity when we want to support pipeline migration to a different tremor node, when we have clustering for tremor.

Describe the solution you'd like

Use the pipeline state mechanism for these operators:

Come up with a solution for these operators, that have internal state not easily expressible as Value (our primitive for the operator state). One solution is to serialize the stateful data structures into (and deserialize out) of the pipeline state, when we do need to store them somewhere central (eg: to enable pipeline migrations as part of clustering effort).

Notes

Pulled from https://rfcs.tremor.rs/0002-pipeline-state-mechanism/#future-possibilities to track as an issue.

Also see https://rfcs.tremor.rs/0002-pipeline-state-mechanism/#unresolved-questions

anupdhml commented 4 years ago

For reference, generic::counter operator is already using the state mechanism: https://github.com/wayfair-tremor/tremor-runtime/blob/v0.7.3/tremor-pipeline/src/op/generic/counter.rs

Implementation PR it was part of: https://github.com/wayfair-tremor/tremor-runtime/pull/59