ArroyoSystems / arroyo

Distributed stream processing engine in Rust
https://arroyo.dev
Apache License 2.0
3.8k stars 220 forks source link

HA Controller support feature request #787

Open boer0924 opened 1 week ago

boer0924 commented 1 week ago

High-Availability (HA) for the Controller: Implement HA for the controller to enable a seamless failover mechanism. This enhancement would allow multiple controller instances, ensuring that if one controller instance fails, another can take over without disrupting the workflow.

Worker Independence During Controller Downtime: Modify the worker behavior so that existing tasks can continue to operate temporarily even when the controller is unavailable. Unlike Kubernetes, where workers continue operating independently until they reconnect to the controller, Arroyo's workers currently halt, leading to potential downtime. Improving this aspect will help avoid interruptions, making Arroyo more resilient.