feat: Database Size for mobile

[MentalGear]

EDITED.

Voy looks really promising, especially with it's quick retrieval and low footprint.

My use case is running it locally in the browser, to index photos, and I expect the overall database to be around 1-2 GB. Does voy provide any optimization like sharding to ensure mobile browsers are not overwhelmed (or automatic release of least used items like vector-storage ) ?

( Tool to test memory size on mobile: https://jsfiddle.net/jasonklotzer/kvnouyz7/ )

[DawChihLiou]

Hey @MentalGear thanks for checking out Voy! Voy doesn't support automatic partitioning or caching at the moment. Voy is designed for limited environments like mobile and edge servers! The current tree-structured index in WebAssemly is able to provide efficient traversal through your data. I think for your use case it'll perform great. But if you start to experience performance issues as your dataset grows, please do create an issue.

I really like your idea about automatic sharding. I'll look into it.

[MentalGear]

Hey, glad you like the idea!

Indeed Voy is highly efficient in the retrieval! Here's some resource on sharding from the excellent RxDB

I found that overall in-Memory size per tab varies for mobile browsers quite a bit: For example (latest OS): iPad: 2GB, iPhone X: 1GB. Android is different again. Also, loading a chunk of 512mb at once on iOS might give an Out of Memory Error, while loading the memory in smaller chunks, allows to go well above the 512mb limit. Here's a test for in-browser memory limits.

On-Device Storage: Privacy and on-device storage has become quite prominent in regard of the new ML generative abilities.

I was wondering if Voy could integrate, besides a normal export function, storage adapters, most preferable with something like OPFS which is fast and supported by all mayor browsers now. (Reference: https://rxdb.info/rx-storage-opfs.html)

[DawChihLiou]

What about using web workers? Using a shared worker might give you more freedom from the memory limitation per tab.

[MentalGear]

Yes, I'm also interested in this, but could not find definitive resources for web workers. Shared Web workers allow the usage of the same worker across multiple tabs of the same web resource, however I don't think they expand overall available memory (but I don't know for sure).

There's a few APIs, however with very limited browser support (https://developer.mozilla.org/en-US/docs/Web/API/Performance/memory).

[DawChihLiou]

Hey @MentalGear I've done some more digging. Because Voy uses the embeddings as coordinates to store and locate minimal data from the resources, the requirement for the storage is greatly reduced (no embeddings are stored). For the resource data, like title, description, and url, I can provide you with an API that allows serialized and unstructured data to further minimize the size.

[MentalGear]

Yes, adding custom data to records would be a huge plus !