Closed romulogoncalves closed 5 years ago
Kafka is a data pipeline system with light stream processing
:
Kafka has light Stream processing which is possible with Kafka streams and KSQL. For data management Kafka streams uses RocksDB (developed and maintained by Facebook Database Engineering Team). It is a persistent key-value store.
Kafka API is simple and this blog explains in a nutshell how Kafka architecture looks like.
For complex stream processing the systems of choice are Apache Storm, Apache Flink and Apache Spark Streaming.
@romulogoncalves -- you had some slides about this? Could we possibly link to them here?
We should talk about what is going on in stream processing and who knows about it at NLeSC.
Some technologies: