[SE] Joins optimization: interval, temporal

volga-project / volga

Feature Engine for real-time AI/ML

Apache License 2.0

36 stars 4 forks source link

Open anovv opened 5 months ago

anovv commented 5 months ago

Current JoinOperator strores all of the processed data, this does not scale. We need to do 2 things:

Add option of specifying time-interval to store data (e.g. Flinks IntervalJoin)
Add option to stop storing data when bounded stream is fully processed an is in operators memory (possibly pre-load it?) - Flinks TemporalJoin