TimelyDataflow / timely-dataflow

A modular implementation of timely dataflow in Rust
MIT License
3.25k stars 273 forks source link

Container GATs, improve traits #541

Closed antiguru closed 5 months ago

antiguru commented 7 months ago

Cleanup the Container::Item type, move it to PushPartitioned and make it a GAT. Similarly, split the Map trait in two (Map, MapInPlace) and support GATs in the Map trait.

This has the benefit that Timely becomes less opinionated about the interface provided by containers, for example by allowing non-owned data when calling stream.map.

antiguru commented 6 months ago

I reworked the PR to only contain a minimum of changes. This compiles with differential once applying the following patch:

diff --git a/src/collection.rs b/src/collection.rs
index f2daf8ad..20cab0d2 100644
--- a/src/collection.rs
+++ b/src/collection.rs
@@ -416,10 +416,10 @@ impl<G: Scope, D: Data, R: Semigroup> Collection<G, D, R> where G::Timestamp: Da
     ///          .inspect_batch(|t,xs| println!("errors @ {:?}: {:?}", t, xs));
     /// });
     /// ```
-    pub fn inspect_batch<F>(&self, func: F) -> Collection<G, D, R>
+    pub fn inspect_batch<F>(&self, mut func: F) -> Collection<G, D, R>
     where F: FnMut(&G::Timestamp, &[(D, G::Timestamp, R)])+'static {
         self.inner
-            .inspect_batch(func)
+            .inspect_batch(move |time, batch| func(time, &batch[..]))
             .as_collection()
     }
     /// Attaches a timely dataflow probe to the output of a Collection.
(