Apache Kafka as a Destination

[gervarela]

Problem

Apache Kafka is a common source of data for streaming and real-time analytics. In those setups it is also commonly used as a kind of 'data lake' or 'source of truth' where all the history is registered before any transformation and ingestion is done, so that in the future it can be 'replicated' in case of a bug or new feature downstream.

We propose to support the output of EventNative events (in streaming mode) to a Kafka topic (maybe one topic per collection). Downstream, this topic(s) could be used to any number of many other systems for real-time analytics like :

• ClickHouse using Kafka engine tables.
• Other real-time databases like Apache Druid.
• kSQL for stream processing with SQL.
• Streaming processing systems like Spark, Storm, Samza, Faust, etc.
• Almost any kind of storage system with support for Kafka connectors.

All of these systems already support Kafka ingestion.

Solution

Develop support for using Kafka topics as destinations.

A good solution would be allow one topic per each EventNative collection, so different types of events are written to different topics, and the schema of the events of each topic can be known in advance.

The pipeline will be like this: Tracker (JSON) --> EventNative --> Kafka (JSON) --> Others

A nice add-on feature would be to use also Kafka topics to store 'failed events'. In this case you will have one 'good' topic, and one 'bad' topic per each collection. This would allow, among other things, for the monitoring of the 'bad' events topic, in real-time, to detect and manage failures.

[azhard4int]

@absorbb @vklimontovich

As per our discussion on the Slack, the number of incoming files for the Jitsu increased while using the batch streaming mode with log.rotation_min: 1, when event rate is 2000/min. The Jitsu uploader process is sequential and does not process all of the events, hence leaving the backlog of events.

• The proposed solution was to set the log.rotation_min to 10 minutes or go with streaming mode. But, we are using Clickhouse and streaming mode will put a lot of pressure on the Clickhouse.
• The file processing mechanism could be parallelise, but at scale this could happen again. i.e 20,000-50,000 events/min

As a long term solution, Kafka can be added a a destination with Streaming mode, and the events are sent to the Kafka topics and then sent to the Clickhouse.