search

kyr0 / vectorstore

In-browser, multi-lingual vector embedding and search
MIT License
5 stars 0 forks source link

Persisting the data #2

Open Torhamilton opened 3 weeks ago

Torhamilton commented 3 weeks ago

How to locally save embeddings. Thanks

kyr0 commented 2 weeks ago

Hi @Torhamilton the idea is to serialize the embeddings and store them via IndexedDB (when executed in browsers). IndexedDB has no size limitations and can be used cross-domain, when used in a service worker. To not deal with low-level the idea is to implement the storage layer based on dexie.js. For when used with Node.js or any other server-side JS runtime environment, the serialized vector embedding data can be stored in almost any database via a Blob-like datastructure (encoded as Unit8Array) or, via the same means, in files, on-disk, with binary encoding. I'm not there yet, but it will happen soon.