TimelyDataflow / differential-dataflow

An implementation of differential dataflow using timely dataflow on Rust.
MIT License
2.54k stars 183 forks source link

Move `consolidate` methods to inherent implementations #376

Closed frankmcsherry closed 1 year ago

frankmcsherry commented 1 year ago

This PR removes the Consolidate trait, and moves its associated methods to inherent implementations on Collection. There are no other implementors of the trait. The only plausible reason I could think of for the trait is to write code generic over Collection and Arrangement, but the latter didn't implement Consolidate yet in any case.

The change also alters consolidate_named to match arrange_named which takes a Tr: Trace generic argument, which can be specified by the user. Methods are not allowed to have generic arguments with default type values, so the consolidate method is the only way to do this. This removes the previous ability to easily specify a name without more awkwardly specifying the trace at the same time.

frankmcsherry commented 1 year ago

It's a fine question. We could always consider tearing about the batch building stages into their own operators, and allow a consolidate that does not form batches. Also, one that forms but does not unpack batches. The consolidate method won't end up with a maintained arrangement though, so no cost beyond per-batch memory used to consolidate.