search

neondatabase / neon

Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, code-like database branching, and scale to zero.
https://neon.tech
Apache License 2.0
15.21k stars 444 forks source link

Epic: sharded pageserver ingest #9329

Open VladLazar opened 1 month ago

VladLazar commented 1 month ago

Background

We currently we send all WAL to all pageserver shards, and each shard filters out the data that it needs, in this epic we add a mechanism to filter the WAL on the safekeeper, so that each shard receives only the data it needs.

RFC in https://github.com/neondatabase/neon/pull/8754.

### Sub-issues
- [ ] https://github.com/neondatabase/neon/issues/9328
- [ ] https://github.com/neondatabase/neon/issues/9335
- [ ] https://github.com/neondatabase/neon/issues/9336
- [ ] https://github.com/neondatabase/neon/issues/9337
- [ ] https://github.com/neondatabase/neon/issues/9338
- [ ] https://github.com/neondatabase/neon/issues/9339
- [ ] https://github.com/neondatabase/neon/issues/5543
- [ ] https://github.com/neondatabase/neon/issues/9569
### Nice to Haves
- [ ] https://github.com/neondatabase/neon/issues/7002
- [ ] https://github.com/neondatabase/neon/issues/5962
### Review follow-ups
- [ ] https://github.com/neondatabase/neon/issues/9491
- [ ] https://github.com/neondatabase/neon/issues/9823
- [ ] https://github.com/neondatabase/neon/issues/9824
jcsp commented 1 month ago

Can we reconcile this with https://github.com/neondatabase/neon/issues/6345? (I don't mind replacing that issue, just want to avoid duplication)