search

TimelyDataflow / differential-dataflow

An implementation of differential dataflow using timely dataflow on Rust.
MIT License
2.51k stars 182 forks source link

Re-introduce safe consolidation #481

Closed frankmcsherry closed 2 months ago

frankmcsherry commented 2 months ago

Consolidation previously used an unsafe implementation because it improved performance over slice::split_at_mut. This no longer appears to be the case, and also unsafe code is bad and probably doesn't work.

At the same time, the current consolidation also appears to be almost 2x as slow as a version from @antiguru that copies from one container to another, and perhaps the in-place consolidation results in loop-carried dependencies or something like that. There is certainly some more to look into here, especially as consolidation is such a common event.

frankmcsherry commented 2 months ago

This is substantially faster than master on large consolidation benchmarks, and seems comparable to master or a little better on examples/bfs benchmarks. The improvement is within measurement error, but has been consistently on the side of "improvement".