Introduce event windowing to the StreamPipes core/sdk

[dominikriemer]

Apache StreamPipes

Apache StreamPipes (incubating) is a self-service (Industrial) IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams. StreamPipes offers several modules including StreamPipes Connect to easily connect data from industrial IoT sources, the Pipeline Editor to quickly create processing pipelines and several visualization modules for live and historic data exploration. Under the hood, StreamPipes utilizes an event-driven microservice paradigm of standalone, so-called analytics microservices making the system easy to extend for individual needs.

Background

Currently, window logic can be individually defined per pipeline element. The whole windowing logic needs to be declared in the controller and runtime logic needs to be individually added based on the selected runtime wrapper (Java, Siddhi, Flink, etc...).

As many data processors benefit from using window-functions (i.e PEs such as Event Counter, Count Aggregation, Rate Limiter), windowing logic is often duplicated as it needs to be implemented for every new pipeline element. In addition, the feature set of supported window operators differs (and often depends on the developer) as it is unclear which windows and parameters should/can be offered.  

Therefore, adding support for explicit window semantics to the SDK/Core would make implementing data processors and sinks using windows much easier and less error-prone.

Tasks

• Design and introduce new processor and controller classes for windowed event processors (e.g., WindowedDataProcessor) which handle the windowing logic internally and only expose the higher-level methods to users (i.e onCurrentEvent, onExpiredEvent, etc...).
• Implement internal logic for few window functions (i.e TimeWindow, LengthWindow, TimeBatchWindow, LengthBatchWindow, etc...)
• Write a few sample pipeline-elements using your new API!

Relevant Skills

• Basic knowledge in StreamPipes core (cloning the repo, going through the codebase/documents would do).
• Basic knowledge of stream analytics window functions (this is not a must, but it's awesome if you know your way around analytics window functions).
• Some Java experience.

Learning Material

For StreamPipes:

• https://streampipes.apache.org/docs/
• https://streampipes.apache.org/media.html
• https://github.com/apache/incubator-streampipes
• https://github.com/apache/incubator-streampipes-extensions

For Streaming Analytics:

• https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-window-functions
• https://www.mikulskibartosz.name/difference-between-tumbling-and-sliding-window/
• https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.1.5/developing-storm-applications/content/understanding_sliding_and_tumbling_windows.html

For the context for the issue:

• https://www.mail-archive.com/dev@streampipes.apache.org/msg00868.html

Mentor

• Grainier Perera (grainier [at] apache.org).

Imported from Jira STREAMPIPES-292. Original Jira may contain additional context. Reported by: grainier.

[liuxiaocs7]

Hi, @dominikriemer, I am very interested in this task. Could you please assign it to me, thanks.