search

neondatabase / neon

Neon: Serverless Postgres. We separated storage and compute to offer autoscaling, code-like database branching, and scale to zero.
https://neon.tech
Apache License 2.0
14.49k stars 421 forks source link

local manifest for atomic storage operation #4418

Open skyzh opened 1 year ago

skyzh commented 1 year ago

Currently we do not have local manifest. The compaction process will need to automatically remove/add set of files to the engine. It will also need to record some data (i.e., levels, start/end keys) in the manifest. We currently scan the directory to reconstruct the storage engine state, which would cause duplicated layers issue when restarted.

https://www.notion.so/neondatabase/Local-Manifest-948a76a35096465ba1be3a8bd2b84c7c

pseacrest commented 1 year ago

I can echo this. We don't have a local manifest right now. Automatic file removal and addition to the engine will be required as part of the compaction process. Additionally, some information (such as levels and start/end keys) must be recorded in the manifest.

koivunej commented 10 months ago

With crash-consistency through index_part.json I don't think this is needed anymore.