Implements stream level watermarks at barrier level granularity.
Limitations:
Depends on the barriers to transfer the watermarks down. If your checkpointing period is high, then you will only process the watermarks with that granularity. Future work will implement watermark messages which have higher granularity.
If you use trigger/emit and flush a lot of segments from a window downstream, you will see a corresponding memory increase. Future work should page these segments in incrementally.
Makes a breaking change to plugin interface. We should probably add an extra plugin interface (Watermarkable?) and then check whether the plugin implements it. That would prevent this from being a breaking change.
Implements stream level watermarks at barrier level granularity. Limitations:
Depends on the barriers to transfer the watermarks down. If your checkpointing period is high, then you will only process the watermarks with that granularity. Future work will implement watermark messages which have higher granularity. If you use trigger/emit and flush a lot of segments from a window downstream, you will see a corresponding memory increase. Future work should page these segments in incrementally. Makes a breaking change to plugin interface. We should probably add an extra plugin interface (Watermarkable?) and then check whether the plugin implements it. That would prevent this from being a breaking change.