search

neondatabase / neon

Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, code-like database branching, and scale to zero.
https://neon.tech
Apache License 2.0
15.12k stars 440 forks source link

Epic: PIT. Rewind use case #3064

Open vadim2404 opened 1 year ago

vadim2404 commented 1 year ago

Motivation

Users must be able to rewind at some point in time their workload.

DoD

Implementation ideas

### Tasks

Other related tasks and Epics

koivunej commented 1 year ago

Does this mean like git reset --hard <commit> instead of creating a new branch from history and continuing there, later on removing the first branch?

vadim2404 commented 1 year ago

Possibly yes, another use case:

  1. I had a branch "A"
  2. I created a branch "B"
  3. And I continued working with branch "B"
  4. I don't need a branch "A" anymore
  5. How can I delete this? "A" is a parent of "B"

Maybe we need to allow to change of "parents" for such cases, I don't know yet

knizhnik commented 1 year ago

I think that the difference is mostly in user expectation. Most of database (and not only) users are familiar with backups and know very well that it is necessary to make backups to prevent loose of data. We can say - in Neon you do not need to make backups, you can just return back to any point of time in the past. Ok, not to any point, but within 7 days (PiTR interval). And actually we do not always store information for all 7 days because there is also limit for WAL size...

I think that many clients will not be satisfied with such answer and will say: PiTR is fine, but can I still make some backup of the database stored just as file in some format from which database can be restored (and may be not necessarily in Neon, but in arbitrary Postgres instance).

And IMHO it is quit reasonable request: there can be some bug in Neon which cause storage corruption al loose of all data for all period of time. Backup file certainly also can be corrupted or lost. But if it is read-only file, stored in S3, then probability of it is much much smaller.

And once user have backup, he definitely wants to be able to restore data from backup. And may be periodically he will do it even if main database is not corrupted - just to check that backups are valid and it is able to restore database from it.

Restore from backup is not the same as switching to the branch (or creation of new branch). Once you switched to the branch, you are able to make some changes in this branch. But it actually means that you are not able to use this branch as backup any more. So this operation is different from restore from backup which is idempotent.

My opinion (although nobody asked it:) is that we should definitely make accent that Neon architecture allows you not to do backups and rely on PiTR. But at the same time we should provide for users "create backup" and "restore from backup" operations with expected semantic, doesn't matter that internally they can be implemented using branching.

shanyp commented 1 year ago

The rationale was that we have already several epics lying around about backups: https://github.com/neondatabase/neon/issues/915 and https://github.com/neondatabase/neon/issues/1541 for example. I would like us to have a concise backlog which will allow us to work efficiently Having said that, your opinion is indeed what our product requirements already state (https://github.com/neondatabase/neon_roadmap/issues/40) So I will revive https://github.com/neondatabase/neon/issues/3063 and merge the other related epics into it