search

TimelyDataflow / timely-dataflow

A modular implementation of timely dataflow in Rust
MIT License
3.25k stars 273 forks source link

Container builder #562

Closed antiguru closed 4 months ago

antiguru commented 4 months ago

Introduce a ContainerBuilder that sits on the outputs of operators and gets a chance to process data as produced by the operator. While the PR breaks many APIs, it tries to preserve compatibility for all users of output.session, which we hard code to a builder that's based on the length and capacity of a container. In other places, we need a type annotation, specifically when there is no call to session, or just a call to session_with_builder, which is the equivalent to session but doesn't force a specific builder.

The PR also replaces AutoflushSession with AutoflushSessionCore, and renames the latter to avoid Core. All uses were internal and easy to fix, which should apply to external crates, too.

This is compatible with differential to offer in-place consolidation.

Do not look commit-by-commit as the history represents a worklog instead of a grouping of changes. Readers are encouraged to first look at the timely-container library.

antiguru commented 4 months ago

https://github.com/TimelyDataflow/differential-dataflow/pull/478 shows how to integrate this with differential.

antiguru commented 4 months ago

Oh, if we want to have the return-position impl trait, we'd need to move to Rust 1.75. This, or refactoring to use explicit types both work for me.