Implement a utility class to abstract away sharded Durable Objects.
For example, I want to use N DOs instead of 1, where N is known upfront and will never change (for the initial version).
The utilities should allow reads/writes into a specific shard based on a shardKey: string, but also reads without specifying any shard key and in that case the aggregated results from all shards should be returned.
This can be done entirely with library code that will run in a Worker, I don't think I need to wrap custom logic on a DO.
We might need more storage than a single DO (1GB for SQLite, to become 10GB in GA).
We might need more throughput than a single DO.
We want to avoid a single DO being the single point of failure for all of our data.
Goals
Each shard (single DO) owns and manages the data for all shardKey values that hash into that shard.
Initial version is fixed N shards to avoid rebalancing with shards being removed or added (once we have more utilities to facilitate this we can extend it).
The reads operations across all shards should provide some kind of streaming or iteration to ease consumption of all the data.
Implement a utility class to abstract away sharded Durable Objects.
For example, I want to use N DOs instead of 1, where N is known upfront and will never change (for the initial version).
The utilities should allow reads/writes into a specific shard based on a
shardKey: string, but also reads without specifying any shard key and in that case the aggregated results from all shards should be returned.This can be done entirely with library code that will run in a Worker, I don't think I need to wrap custom logic on a DO.
Why
Goals
shardKeyvalues that hash into that shard.