Define Scope for Apache Kafka Support

[kamir]

This issue is related to #386 .

In this issue we want to identify the scope of Apache Wayang - Apache Kafka support.

Apache Kafka does not offer a processing engine. KStreams and KSQLDB have been developed for "Kafka internal streaming processing". But it has been decided to discontinue KSQLDB. Instead, Apache Flink has been selected as the new event streaming processing system on top of Apache Kafka. But besides Apache Flink, there is the KStreams library. KSQLDB has been created using the KStreams event stream processing framework.

Why not to use ksqlDB? (from Google Search) ksqlDB is inefficient with long-running or high-cardinality aggregation. Routing, filtering, and running basic transformations over streaming data are the strengths of ksqlDB, and while it can perform some aggregations, it will suffer under more complex scenarios requiring large amounts of state.27.07.2023

With this in mind I suggest to define 2 scopes for Apache Kafka support in Apache Wayang:

Scope 1: Support Apache Kafka Source and Sink components in Apache Wayang plans. Scope 2: Support KStreams for "native Kafka Streaming processing" coordinated by Apache Wayang.

[kamir]

I suggest to start with implementing a Kafka-Source and a Kafka-Sink components, so that existing Apache Wayang applications can get input directly from Kafka topics and store results directly in such topics. This will not yet give us the full "Apache Kafka support as a processing platform for Apache Wayang" - but it is a starting point for interacting with the data which lives in Apache Kafka topics.

[zkaoudi]

+1

A kafka source+sink would be awesome.

[kamir]

I am back on this task working out a simple KafkaSource component, reading plain text messages from a Kafka cluster, comparable with the JavaFileSource, which can read file, line by line from HTTP URLs now.

[kamir]

Started merging the feature via branch mk-feature-2