[Backend] KMS upload limits - Githubissues

QuivrHQ / quivr

Opiniated RAG for integrating GenAI in your apps 🧠 Focus on your product rather than the RAG. Easy integration in existing products with customisation! Any LLM: GPT4, Groq, Llama. Any Vectorstore: PGVector, Faiss. Any Files. Anyway you want.

https://core.quivr.com

Other

36.68k stars 3.58k forks source link

[Backend] KMS upload limits #3170

Open linear[bot] opened 2 months ago

linear[bot] commented 2 months ago

KMS Limits:

Uploads (Local)

[x] Single file size ? (50Mb / file )
[x] Single Folder total size ? No limits
Concurrent file upload ? ( rate limit upload route, fornow can't upload whole folders)
[x] # Files per folder ? -> No limits
Total storage per tier of users ?
- 500Mb free
- 100GB premium

Knowledge linked to Brains ( sync & local )

Total size for all brains:
- free: 500 Mb
- premium: 100Gb for all brains ( get size users in production)
- No limits for # brains
Nb or files ?
- correct metric # of chunks in brain -> have the infos back to the user (too much chunks / files in brain .
File size across syncs + local ? No diff
# sync files per user to add to brain ?
- Limit to # chunks in a brain

KM Processing

# max concurrent processing of files ?
- max queue size of 10_000 (check ) 4 worker
# max concurrent processing files per user
- fairness scheduler / per user
Max time processing per file
- p99 celery worker
Max batch size sync
- Randomize across users
- Priorize refresh for connected users

linear[bot] commented 2 months ago

ENT-68 [Backend] KMS upload limits