ETL tool / framework research

[FlorianNachtigall]

• Option1: Kafka Consumer / Producer

• Option2: Spark Streaming (should we limit it to spark streaming? I think yes... )

Both Options can be implemented in python (confluent-kafka / kafka-python and pyspark) ->> two clients that share privacy library

Some examples:

• https://spark.apache.org/docs/latest/streaming-programming-guide.html#a-quick-example
• Kafka code examples

[ciklista]

• Option1: Kafka Consumer / Producer

• Option2: Spark Streaming (should we limit it to spark streaming? I think yes... )

Both Options can be implemented in python (confluent-kafka / kafka-python and pyspark) ->> two clients that share privacy library

Some examples:

• https://spark.apache.org/docs/latest/streaming-programming-guide.html#a-quick-example
• Kafka code examples