search

TU-Berlin-DIMA / scotty-window-processor

This repository provides Scotty, a framework for efficient window aggregations for out-of-order Stream Processing.
https://tu-berlin-dima.github.io/scotty-window-processor/
Apache License 2.0
75 stars 23 forks source link

[Bug] Incorrect Slices of Sliding Window depending on Slide Value #49

Closed julianev closed 2 years ago

julianev commented 2 years ago

Scotty's StreamSlicer starts a new slice based on the window edge that is returned by the function assignNextWindowStart.

https://github.com/TU-Berlin-DIMA/scotty-window-processor/blob/2a6b33941ac1733fb7b8db2cece04b2afc61d40f/core/src/main/java/de/tub/dima/scotty/core/windowType/SlidingWindow.java#L41-L43

This does not work for sliding windows whose slide value is not a multiple of the window size, for instance, a sliding window with size = 10 and slide = 3. In these cases, the values of the window starts and the window ends are not identical. The function only returns the next window start which is based on the slide value, but does not return the next window end. Thus, the slices do not end when a window ends, leading to incorrect window results.

julianev commented 2 years ago

This issue occurred during the migration to the open-source project. It is not a limitation of Scotty. The concept of modeling a sliding window by two tumbling windows is correctly described in the paper Scotty: General and Efficient Open-source Window Aggregation for Stream Processing Systems (TODS 2021).