oasysai / oasysdb

Hybrid vector database with flexible SQL storage engine & multi-index support.
https://docs.oasysdb.com/
Apache License 2.0
357 stars 12 forks source link

FEAT: optimize collection read and write from/to disk #96

Closed edwinkys closed 4 months ago

edwinkys commented 5 months ago

Purpose

Improve the performance when reading and writing collection from/to the disk. The scope of this optimization is backward compatibility.

Ultimately, to improve the speed, we need to reduce the overall collection size by off loading the storage of the vectors and metadata to disk permanently.

Proposed Solution

Use bincode::serialize_into() and bincode::deserialize_from().