mit-pdos / noria

Fast web applications through dynamic, partially-stateful dataflow
Apache License 2.0
4.98k stars 242 forks source link

Support online re-sharding #96

Open jonhoo opened 5 years ago

jonhoo commented 5 years ago

While we can currently remove all shards of an operator (and its children) to re-create it with a different sharding, this is pretty inefficient. Instead, we should support dynamically adding or removing a shard through a migration, which should cause the appropriate state transfer to take shard from/to the existing shards. We do have to be careful about concurrent writes and backfills though!