search

Hurence / logisland

Scalable stream processing platform for advanced realtime analytics on top of Kafka and Spark. LogIsland also supports MQTT and Kafka Streams (Flink being in the roadmap). The platform does complex event processing and is suitable for time series analysis. A large set of valuable ready to use processors, data sources and sinks are available.
https://logisland.github.io
Other
109 stars 28 forks source link

Improve performance of the IncrementalWebSession processor #591

Closed MiniPlayer closed 3 years ago

MiniPlayer commented 3 years ago

To be sure to be sync in events processing, the processor do some thread sleeping and refresh of ES indices. It would be good to add two degraded mode in term of fiability to improve performance. Default mode (Fast mode) => really fast (no refresh and no sleep) Moderate mode => no sleep but refresh are done with a configurable time to wait for Slow mode => sleep and refresh (with a configurable time to wait for refresh)

property name : processing.mode , values : [FAST, MODERATE, SLOW] by default the value is fast The configurable time to wait for refresh is configurable by

property name : es.refresh.wait.time.ms which should be a long (milliseconds), default to 100000

MiniPlayer commented 3 years ago

@jerome73 , @mathieu-rossignol Done (for information)