wandgitlabbot
commented
3 years ago In GitLab, by Daniel Oosterwijk on 2020-01-28
If a windowed operator is not used, out of order data is merely processed as though it was new, despite its older timestamp.
A class which wraps any KeyedProcessFunction in a ProcessWindowFunction in order to correctly order any elements which arrives during that window is a valid approach. Combined with maxLateness, this could allow us to process all elements in the correct order at the cost of some latency.
Some detectors, such as the Mode Detector, don't keep additional state on top of a list of recent measurements. These could be implemented natively as window functions, meaning they can also take advantage of the sliding window approach in order to definitely not miss any events. This would require some event de-duplication further down the stream, but that's probably not a bad idea for all detectors since it would let us stop reusing code which prevents events being emitted too often.
It could be feasible to determine or infer the frequency of each test stream, and create a custom windower or buffering process function which will wait events have been received in order. This would likely want a timeout in case an event is never found, since we don't want to stall forever. We could resort to using an InfluxHistoryConnection to see if we can retrieve missed measurements if they are lost in transit from the collector to us.
In GitLab, by Daniel Oosterwijk on 2019-12-08
Flink allows data to arrive out of order. Some detectors may have stronger order requirements than others, which might need us to adjust parameters like
flink.maxLateness. If there is a detector with a strong order requirement, our sink functions (like InfluxSink) might need to have some way to be signalled that a previously submitted event is now invalid. We don't really know a lot about what will happen if data does arrive out of order, but chances are it won't be an issue at all considering the usual magnitude of intervals between AMP measurements.