search

lightbend / cloudflow

Cloudflow enables users to quickly develop, orchestrate, and operate distributed streaming applications on Kubernetes.
https://cloudflow.io
Apache License 2.0
321 stars 89 forks source link

Add an option to add trigger.continuous to Sparklet #89

Closed blublinsky closed 4 years ago

blublinsky commented 4 years ago

Is your feature request related to a problem? Please describe. For simple queries trigger.continuous allows to produce results continuously, not as mini batch. This allows for faster processing in Sparklets putting them on par with Akka streams and Flink

Is your feature request related to a specific runtime of cloudflow or applicable for all runtimes? Only relevant for Spark runtime

Describe the solution you'd like I would like to allow to specify trigger option for Spark query

Describe alternatives you've considered There are none

Additional context None

olofwalker commented 4 years ago

@blublinsky We are working on improving the way configuration of streamlets, runners, etc is done, once this have been completed, this should be possible to achieve using that new system.