TimelyDataflow / differential-dataflow

An implementation of differential dataflow using timely dataflow on Rust.
MIT License
2.53k stars 182 forks source link

Replace `OrdOffset` with `OffsetList` #437

Closed frankmcsherry closed 9 months ago

frankmcsherry commented 9 months ago

The OrdOffset trait allowed us to slot in u32 or u64 in for usize for offsets, in order to save the RAMs. That isn't terrible, but we can remove the complexity, and the potential surprise if we go too large, with the OffsetList struct that generally behaves as a Vec<u32> until we reach a point at which the entries exceed its size, at which point it behaves like a not-great Vec<u64>.

frankmcsherry commented 9 months ago

Parting thought: if we ever don't like the opinionated lock-in of OffsetList, it should be easy to add back special cases that just use one type and panics when things don't work (to recover the prior implementation).