cda-group / arcon

State-first Streaming Applications in Rust
https://cda-group.github.io/arcon/
Apache License 2.0
175 stars 17 forks source link

Operator Kind #288

Open Max-Meldrum opened 2 years ago

Max-Meldrum commented 2 years ago

The runtime will need some way to distinguish Operators. An example is knowing that a sequence of Operators is of type Non-Blocking and thus can be fused/chained. A straightforward approach would be something like the following:

pub enum OperatorKind {
    /// Map, Filter, Flatmap, ...
    NonBlocking,
    /// Rolling aggregations (sum, max, min)
    NonBlockingPartialAggregator,
    /// Streaming Windows
    Blocking,
}

pub trait Operator: Send + Sized {
    const KIND: OperatorKind;
}
segeljakt commented 2 years ago

I think this is a good argument for having an explicit key_by operator #284. If it's implicit then we can't know if an operator changes the key of incoming events before executing the pipeline, so we can't know if it's safe to fuse it.