TimelyDataflow / differential-dataflow

An implementation of differential dataflow using timely dataflow on Rust.
MIT License
2.51k stars 182 forks source link

Consolidate as part of TraceFrontier `map_times` #490

Open frankmcsherry opened 2 months ago

frankmcsherry commented 2 months ago

TraceFrontier brings all times up to a frontier (and discards updates at times beyond another frontier). This introduces the potential for consolidation, but map_times does not consolidate updates and relies on the interested user to do this. Recent experience indicates that most users are not aware of this, including your correspondent. This fix introduces consolidation as part of map_times, and is meant to broadly insulate folks who do not actively consolidate, which includes some parts of DD itself (seemingly just Cursor::to_vec; other users seem to have a prompt consolidation).

cc: @antiguru