TimelyDataflow / timely-dataflow

A modular implementation of timely dataflow in Rust
MIT License
3.25k stars 272 forks source link

Make MutableAntichain consolidate even without frontier advancement #504

Closed teskje closed 1 year ago

teskje commented 1 year ago

Currently, MutableAntichain consolidates its contained updates only on frontier changes. That means when we keep pushing updates at a MutableAntichain, the size of that object keeps growing unless some of these updates also change the frontier.

For example, consider a MutableAntichain that tracks two frontiers, one that advances every second and one that doesn’t move. Assuming the moving frontier is beyond the non-moving frontier, the MutableAntichain receives one pair of (-1/+1) updates every second and keeps them all around forever.

To avoid memory leaks, MutableAntichain should regularly perform consolidation of its contents, even if its frontier does not advance. A simple strategy would be to consolidate each time the list of updates doubles in size, relative to the last time consolidation was performed.