cda-group / arcon

State-first Streaming Applications in Rust
https://cda-group.github.io/arcon/
Apache License 2.0
175 stars 17 forks source link

Refactor ChannelStrategy/OperatorContext #248

Closed Max-Meldrum closed 3 years ago

Max-Meldrum commented 3 years ago

The current implementation passes along a reference of ComponentDefinition across multiple functions just so that a ChannelStrategy can use it for kompacts tell_serialised function.

Refactor ChannelStrategy to return an iterator of (Channel, ArconMessage) that can be dispatched from the Node/SourceNode itself. This way we also can handle potential buffer errors easier too.

The above change also relates to operators as we also pass a ComponentDefinition reference into the OperatorContext.

Rather than outputting events through ctx.output(event), make handle_element/handle_timeout return an element iterator.

// inside Operator trait
type ElementIterator: IntoIterator<Item = ArconElement<Self::OUT>> + 'static;

This change is also required for it to be possible to create an operator chain (#246).