search

cda-group / arcon

State-first Streaming Applications in Rust
https://cda-group.github.io/arcon/
Apache License 2.0
175 stars 17 forks source link

Arrow Streaming Analytics #285

Open Max-Meldrum opened 2 years ago

Max-Meldrum commented 2 years ago

We are currently using Arrow mainly for converting Protobuf state into queryable Arrow data. I think it would be interesting to see how we could use Arrow for streaming operations and how this can be exposed in the API (e..g, Dataframes?).

As of now, the only thing we have is ArrowWindow that exposes (Arc<Schema>, Vec<RecordBatch>) per Window trigger.

Relevant references:

https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/trill-vldb2015.pdf https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/concepts/dynamic_tables/

Max-Meldrum commented 2 years ago

Maybe we could define a ArrowStream type which would support a number of transformations.