search

nlesc-sigs / data-sig

Linked data, data & modeling SIG
Other
5 stars 3 forks source link

Streaming processing #26

Closed romulogoncalves closed 5 years ago

romulogoncalves commented 5 years ago

We should talk about what is going on in stream processing and who knows about it at NLeSC.

Some technologies:

romulogoncalves commented 5 years ago

Kafka is a data pipeline system with light stream processing:

Kafka has light Stream processing which is possible with Kafka streams and KSQL. For data management Kafka streams uses RocksDB (developed and maintained by Facebook Database Engineering Team). It is a persistent key-value store.

Kafka API is simple and this blog explains in a nutshell how Kafka architecture looks like. image

For complex stream processing the systems of choice are Apache Storm, Apache Flink and Apache Spark Streaming.

c-martinez commented 5 years ago

@romulogoncalves -- you had some slides about this? Could we possibly link to them here?