search

neondatabase / neon

Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, code-like database branching, and scale to zero.
https://neon.tech
Apache License 2.0
13.26k stars 374 forks source link

Implement time_travel_recover for Azure blob storage #7546

Open arpad-m opened 2 months ago

arpad-m commented 2 months ago

The remote_storage code for Azure blob storage doesn't support the time_travel_recover function yet:

https://github.com/neondatabase/neon/blob/46f20308b0216a419a575c181bb1666f28b726fc/libs/remote_storage/src/azure_blob.rs#L505-L515

cc #5567

arpad-m commented 1 month ago

I've looked into this tonight and the list API provides version data. I'm just a bit confused about deletion markers, apparently there are none, but there is a deleted-time/DeletedTime value. How that interacts with versioning and soft delete, I don't know yet. I have to run some experiments.

arpad-m commented 1 month ago

According to the experiments, there is no indication of a version having been deleted, no deletion markers or anything. In other words, while we can restore the old version of the file, we don't know if it has been deleted in the meantime or not.

I also looked into the point in time recovery feature. Apparently it bases on the change feed, but at least for customers there is no index exposed that is per-prefix or something, only a time-indexed structure. Maybe there is internal per-prefix based indices, no clue. However, assuming the worst case, that's a bit sad because then the complexity of the restore operation grows with the size of the storage account, and it's not very scalable to recover tenants on huge regions.

There is a list of bugs/limitations of the point in time recovery feature here.

arpad-m commented 1 month ago

I think for the near future we should implement a wrapper over the builtin point in time recovery feature. Then, once it hits scaling problems we can think about building our own infrastructure that builds indices. Moving to paused as I will focus on different things this week.