search

TimelyDataflow / differential-dataflow

An implementation of differential dataflow using timely dataflow on Rust.
MIT License
2.54k stars 183 forks source link

Allow custom exertion logic #411

Closed frankmcsherry closed 10 months ago

frankmcsherry commented 10 months ago

This PR allows users to set their own "exertion logic", replacing the hardwired reduced function for spines. This was the method that indicated whether there was work to perform in the absence of actual updates. This PR combines this with the effort argument to allow logic that produces an Option<usize> indicating how many updates should be virtually applied, as a function of the shape of the spine. It's a bit of a leaky abstraction (not all traces need to have a spine, right?) but the moment at which an opinion is currently injected requires the trace abstraction.

Some follow-on thoughts:

  1. We could change the Option<usize> output to usize, and commit to 0 to mean "do no work, and do not reschedule".
  2. We could change the idle_merge_effort config to be usize rather than isize, which I think it is because ultimately the exert function needs that. But it doesn't make a lot of sense to provide negative idle merge effort.
  3. We could try and put the method on a Spine type, rather than as part of Trace, but I'm not certain how to thread the system-wide config of "do this with a spine" other than across the trace.

cc: @antiguru, @teskje