oasysai / oasysdb

Hybrid vector database with flexible SQL storage engine & multi-index support.
https://docs.oasysdb.com/
Apache License 2.0
357 stars 12 forks source link

FIX: atomic collection write to disk #98

Closed edwinkys closed 4 months ago

edwinkys commented 4 months ago

Problem

Currently, when performing Database::save_collection method, OasysDB serialize the collection data into binary and store it directly into a file. When dealing with a large file that requires multiple second to write, the risk of a file being corrupted if the write process is interrupted is high.

Solution

We want to make save_collection operation atomic by writing the serialized collection into a temporary file before replacing the actual collection file. This way, if both write and replace operation fails, the actual collection file won't be corrupted.