Implement lock for currently loaded directories

GenSpectrum / LAPIS-SILO

Sequence Indexing engine for Large Order of genomic data

GNU Affero General Public License v3.0

12 stars 3 forks source link

Implement lock for currently loaded directories #612

Open Taepper opened 4 weeks ago

Taepper commented 4 weeks ago

Currently there is no way for a user to know whether SILO (running as api) only requires the folder with the latest valid timestamp or also previous folder are required (due to being in transition from old to new / newer folders are corrupted due to a bug).

A locking mechanism in the form of a file could solve this problem

corneliusroemer commented 4 weeks ago

There are good utilities for this that are part of UNiX so we don't need to reinvent, using flock for example, which makes it easy for multiple SILO instances running on the same data folder to each lock and prevent deletion.

See details from quick Chatgpt exploratory session: https://chatgpt.com/share/670e0ab2-5c78-8005-9373-246d6002ad47

theosanderson commented 4 weeks ago

It's worth considering as part of implementing this whether having multiple SILO/LAPIS instances sharing a directory (with a load balancer, as Cornelius mentioned) is something you intend to support.