search

Hurence / logisland

Scalable stream processing platform for advanced realtime analytics on top of Kafka and Spark. LogIsland also supports MQTT and Kafka Streams (Flink being in the roadmap). The platform does complex event processing and is suitable for time series analysis. A large set of valuable ready to use processors, data sources and sinks are available.
https://logisland.github.io
Other
109 stars 28 forks source link

Support for elasticsearch write StructuredStream #592

Closed mathieu-rossignol closed 2 years ago

mathieu-rossignol commented 3 years ago

When reading from kafka and pushing to ES, this would avoid having to uselessly write to a kafka topic just to have the offset saved into the checkpoint location.

Also we currently use a BulkAdd to push into ES and using the native ES structured stream sink may improve performances.

Documentation Internals about implementation

mathieu-rossignol commented 2 years ago

Implemented with PR https://github.com/Hurence/logisland/pull/593