search

TimelyDataflow / differential-dataflow

An implementation of differential dataflow using timely dataflow on Rust.
MIT License
2.55k stars 183 forks source link

Is this general purpose incremental framework? #275

Open Kesanov opened 4 years ago

Kesanov commented 4 years ago

Is it possible to model any kind of incremental algorithm with this library?

Does it for example make sense to try to model incremental compilation or sorting algorithm with differential dataflow?

frankmcsherry commented 4 years ago

Hello! It should be possible to model anything, but many things won't be good fits. You can sort, for example, by determining for each record x the number of other records in the collection with value < x, which reports the ordinal position for each record. At the same time, you probably don't actually want to do this.

There are other frameworks, like Adapton and Incremental, that probably look more appropriate for "general purpose" incremental programming languages. The main distinction, I think, is how easy certain tasks are. I would say differential is usually better for data-parallel tasks, and the other frameworks for more imperative tasks. Both types of tasks fit in both frameworks, but there is more friction implementing one in the other (e.g. compilation in DD, or Datalog in Adapton/Incremental).