search

cescoffier / fluid

Fluid - a substance that has no fixed shape and yields easily to external pressure
Apache License 2.0
4 stars 4 forks source link

Windowing and Error handling #32

Open cescoffier opened 6 years ago

cescoffier commented 6 years ago

Windowing: transforms the unbounded stream of data into windows of data (by time, size...). By default, we should provide window of 1 data. Sinks will only dispatch completed window.

Error handling: if something bad happens during the data processing, it should cancel the window and emit a {data: data from the window, error: error} on a separated Flowable.

Basically, sinks only dispatch completed window and cancel windows with an error. Because the data and the error are emitted on a separated stream, it is possible to recover (do a different manipulation).

hekonsek commented 6 years ago

Do you have a branch with preview of error handling, by any chance? I'd love to see it :) .

cescoffier commented 6 years ago

It's in the windowing branch. However, I've hit a wall with the current approach and fan-in operators. After discussing with @mbogoevi, we are going to introduce "control message" to implement watermarks (end of window) and avoiding relying on the end of flow for this.