Merge batcher that's generic over input containers and internal chains, with specific implementations.
Ideas
At the moment, a merge batcher receives a stream of vectors. It consolidates the input vectors, and inserts them into its queue structure. When sealing, it extracts ready data and presents it record-by-record to the builder. It inserts future updates into its queue.
This introduces several opportunities to introduce containers:
The input the batcher: The stream of updates received from a timely operator. The batcher requires each batch to be consolidated, i.e., sorted and differences for same data accumulated. It can either take consolidated data, or consolidate it itself. We're missing an abstraction that allows the batcher to consolidate containers, and often containers are read-only, or we only get to look at the data.
We could sort by permuting a Vec<usize> and copying into a new container in sorted order.
Consolidating requires addition, which either requires storage or mutability, which is odd for read-only containers. Not sure what to do here.
The internal queue structure. We need to traverse the elements in order, and compare on data and time, accumulating diffs as we go. This requires specific knowledge about items of a container, which motivates moving the bulk of the implementation behind a trait to avoid generic parameters.
Ready data presented to the builder. We have an opportunity to present all data to the builder at once, which relieves it from incrementally building batches. It unlocks "looking ahead" in the input data.
Merge batcher that's generic over input containers and internal chains, with specific implementations.
Ideas
At the moment, a merge batcher receives a stream of vectors. It consolidates the input vectors, and inserts them into its queue structure. When sealing, it extracts ready data and presents it record-by-record to the builder. It inserts future updates into its queue.
This introduces several opportunities to introduce containers:
Vec<usize>
and copying into a new container in sorted order.