TU-Berlin-DIMA / scotty-window-processor

This repository provides Scotty, a framework for efficient window aggregations for out-of-order Stream Processing.
https://tu-berlin-dima.github.io/scotty-window-processor/
Apache License 2.0
75 stars 23 forks source link

Fix Bug ouf-of-order Processing #45

Closed julianev closed 2 years ago

julianev commented 3 years ago

Fixes https://github.com/TU-Berlin-DIMA/scotty-window-processor/issues/44 to not process out-of-order tuples outside of the allowed lateness (i.e. with a timestamp < currentWatermark - maxLateness - maxFixedWindowSize).

Additional fix of same bug for out-of-order tuples outside of allowed lateness before any watermark occurred. Then, tuples with a timestamp < startsTsOfFirstSlice or timestamp < timestampOfFirstTuple - maxLateness are outside of allowed lateness and are discarded.

For context-ware windows (e.g. session window), a new slice is added with an addModification, if the out-of-order tuple is still in the allowed lateness and starts a new window.