Epic: evict inactive timelines on safekeepers to s3

neondatabase / neon

Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, code-like database branching, and scale to zero.

https://neon.tech

Apache License 2.0

13.21k stars 369 forks source link

Epic: evict inactive timelines on safekeepers to s3 #6220

Open arssher opened 6 months ago

arssher commented 6 months ago

### Tasks
- [ ] https://github.com/neondatabase/neon/issues/6336
- [ ] https://github.com/neondatabase/neon/issues/7751
- [ ] https://github.com/neondatabase/neon/issues/6337
- [ ] https://github.com/neondatabase/neon/issues/6338

problame commented 1 month ago

Progress on #6337 planned this week, but, multi-week project.

problame commented 1 month ago

This week:

monitor release of PR that was merged last week (TODO: link)
impl & create PR for https://github.com/neondatabase/neon/issues/6337

koivunej commented 3 weeks ago

Last week:

Refactoring PR no. 2 was released
Arseny found a bug related to refactoring PR, it was fixed https://github.com/neondatabase/neon/issues/7967

This week:

need to create the last PR to implement eviction
then this issue should be ready

petuhovskiy commented 1 day ago

Last week:

eviction PR was merged to staging, but not enabled anywhere
then I discovered a bunch of issues and merged some fixes
verified that no critical issues are left on staging

This week:

waiting for production deploy to ensure nothing break with DISABLED eviction
after that, we can enable eviction in staging and look for new issues

Next week:

if everything will be good in staging, we should enable eviction in prod

Later:

enable file deletion gradually in all regions (first staging, then prod)