search

gusye1234 / nano-vectordb

A simple, easy-to-hack Vector Database
65 stars 3 forks source link

Loading the entire database NOT in memory for scalability #5

Open Yazington opened 1 month ago

Yazington commented 1 month ago

If this is being created for multiple users (maybe one day millions), we need to scale. Having in memory with numpy is simple, but if I was to query something, it would load the entire database in memory

Suggestions:

  1. sharding
  2. lazy loading
  3. caching
gusye1234 commented 1 month ago

Yeah! In fact I'm planning to improve this Project with multi-user support

gusye1234 commented 1 week ago

@Yazington Hi, I added a naive beta support for multi-tenancy, maybe you can do a proof-reading of this feature❤️