Database Sharding on Document Tables

JasperFx / marten

.NET Transactional Document DB and Event Store on PostgreSQL

https://martendb.io

MIT License

2.79k stars 441 forks source link

Database Sharding on Document Tables #3315

Closed jeremydmiller closed 1 month ago

jeremydmiller commented 1 month ago

We've talked about this over the years, but it's relevant now to a current JasperFx client, so moves up in priority. For only the document tables, allow users to specify table shards. This is probably mostly an effort on the Weasel side of things. Not sure how we'll go about this and what we'll allow, but this will track that work.

Ideas:

Shard by tenant id for conjoined tenancy
Shard on soft delete flags
User defined shards???

jeremydmiller commented 1 month ago

Tasks and Use Cases

[x] Quick option for LIST partitioning on soft deletes
[x] Quick option for partitioning on tenant id with ranges
[x] Quick option for partitioning on tenant id with HASH
[x] Partitioning on tenant id with LIST
[x] External managed LIST partitions
[x] External managed RANGE partitions
[x] Partitioning off of a single user selected duplicate field
[x] Partitioning off of multiple user selected duplicate field
[x] Apply the multi-tenancy partitioning for all multi-tenanted documents