Mintplex-Labs / vector-admin

The universal tool suite for vector database management. Manage Pinecone, Chroma, Qdrant, Weaviate and more vector databases with ease.
https://vectoradmin.com/
MIT License
1.02k stars 173 forks source link

rely on provider for counting of vectors and only optionally fallback on DB due to slow query #58

Closed timothycarambat closed 9 months ago

timothycarambat commented 9 months ago

Until the data migration is done and organiztion_id is appended to document vectors we will need to rely on the provider for counting vectors.

  1. This can count documents or vectors that VectorAdmin isn't aware of because we are reading from remote
  2. The SQL query currently used is not great and with 50K documents results in an even larger ...IN(1,2,3,) query looking for document ids that are in document_vectors and would be easier to have a fixed organization_id key we can COUNT against since we will very easily reach the upper end of IN() parameters