ArroyoSystems / arroyo

Distributed stream processing engine in Rust
https://arroyo.dev
Apache License 2.0
3.67k stars 206 forks source link

Migrate node scheduler to arrow #587

Closed mwylde closed 5 months ago

mwylde commented 6 months ago

Migrates the node scheduler to the new architecture, hooks it into arroyo-bin, and modifies it to support running arrow programs.

This allowed us to simplify the controller -> node protocol, which previously needed to be able to stream large binaries over gRPC.