search

TimelyDataflow / differential-dataflow

An implementation of differential dataflow using timely dataflow on Rust.
MIT License
2.51k stars 182 forks source link

Introduce generic consolidator #480

Closed antiguru closed 1 month ago

antiguru commented 2 months ago

Currently slightly slower than the existing in-place consolidate, mostly because it needs to copy the keys.

test consolidation::tests::test_consolidator_duration ... elapsed consolidator 3.956388988s ok test consolidation::tests::test_consolidator_duration_vec ... elapsed vec 3.652386365s

antiguru commented 2 months ago

It's improved:

➜  differential-dataflow git:(consolidator) taskset -c 4 cargo test --release --lib -- --nocapture duration --test-threads=1
   Compiling differential-dataflow v0.12.0 (/home/moritz/dev/repos/differential-dataflow)
    Finished `release` profile [optimized + debuginfo] target(s) in 36.85s
     Running unittests src/lib.rs (target/release/deps/differential_dataflow-59cc925a33da4197)

running 2 tests
test consolidation::tests::test_consolidator_duration ... elapsed consolidator 2.036071091s
ok
test consolidation::tests::test_consolidator_duration_vec ... elapsed vec 8.337145545s
ok
antiguru commented 1 month ago

Closing in favor of #494.