search

IntersectMBO / lsm-tree

A Haskell library for on-disk tables based on LSM-trees
Apache License 2.0
19 stars 5 forks source link

Allow storing snapshots outside of the session directory #272

Open jorisdral opened 2 weeks ago

jorisdral commented 2 weeks ago

This means that the snapshots can be stored on a different drive that is more suitable for persisting table snapshots.

The use case is a cloud instance where the local SSD should be considered ephemeral. Active tables can exist on the SSD, but persistent snapshots should live on a persistent network EBS volume.

dcoutts commented 2 weeks ago

With this feature, it allows multiple sessions to share the same snapshots dir, and thus be used from different processes that do not coordinate on the session or lock file. Thus, when implementing this, we should review the code to create, and delete snapshots to make sure it's safe to use from concurrent sessions.

This probably just means making sure that for creating snapshots we guarantee that the files are new (not existing), so any race results in the loosing side getting an exception. This can be done using the MustBeNew modifier to the OpenMode for the fs-api's hOpen.