Dynamically resize / split shards if they grow too large

[ozgune]

We had several requests from prospective and current customers about dynamically resizing their shards. This for example could become handy if a customer starts with 32 shards, expands their cluster size (# of machines), and then wants to double the shard count to 64 to benefit from a larger cluster.

With tenant isolation (#844), we introduced new primitives to split a shard into smaller ones and rebalance those shards to new machines. We could use those primitives to enable shard resizing / splitting functionality, not only for tenants but for dynamic shard resizing as well.

[lfittl]

I'll make one additional note here: Any users of this feature will care a lot about for how long, and what kind of locks are taken as part of this operation.

As we implement this, lets maybe try to setup a corresponding citus_docs issue that documents the effective (write) downtime a customer has to expect here, based on an example shard with a specific size.

[jasonmp85]

@lfittl it's also worth maybe speccing out steps towards a solution which does not block writes.

[ozgune]

@lfittl -- Quick question. We currently run tenant isolation as a two step process: (1) split one shard into smaller shards and (2) move the smaller shards into new worker nodes.

When you're thinking about documenting effective write downtime, which one of the two steps are more concerning to you?

[lfittl]

@ozgune We have to document both, since in the end what matters is what you have to do in the application to queue up writes / turn off the application for that tenant, etc.

One detail there, and this applies more to tenant isolation, less splitting/resizing shards, is that I suspect (1) will also block other tenants in the same shard, whereas (2) really only blocks the tenant itself, so one could implement a "We're moving your data around" page in the application, just for that tenant. It might be worth describing as much.