TimelyDataflow / differential-dataflow

An implementation of differential dataflow using timely dataflow on Rust.
MIT License
2.51k stars 182 forks source link

Merge batcher generic over containers #474

Closed antiguru closed 2 months ago

antiguru commented 2 months ago

Merge batcher that's generic over input containers and internal chains, with specific implementations.

Ideas

At the moment, a merge batcher receives a stream of vectors. It consolidates the input vectors, and inserts them into its queue structure. When sealing, it extracts ready data and presents it record-by-record to the builder. It inserts future updates into its queue.

This introduces several opportunities to introduce containers: