[IDEA] Guide/process to build embeddings elsewhere and sync that to the server.

khoj-ai / khoj

Your AI second brain. Get answers to your questions, whether they be online or in your own notes. Use online AI models (e.g gpt4) or private, local LLMs (e.g llama3). Self-host locally or use our cloud instance. Access from Obsidian, Emacs, Desktop app, Web or Whatsapp.

https://khoj.dev

GNU Affero General Public License v3.0

12.63k stars 640 forks source link

[IDEA] Guide/process to build embeddings elsewhere and sync that to the server. #829

Open stevennt opened 3 months ago

stevennt commented 3 months ago

Describe the feature you'd like

Problem: Embedding on the server takes a lot of time, resource, and in many cases error prone, especially when GB-s of data need to be processed, and likely again and again.

If a script can be developed to allow users to run the embeddings elsewhere, then sync that up to the server / db, then it would be very convenient.

Thanks.

debanjum commented 2 months ago

Hey @stevennt, could you share a bit more about your use-case? What's the expected scale of usage?

Khoj can be configured to compute embeddings using the HuggingFace's serverless inference API. For production use-case you can use their dedicated inference endpoints. Just set the inference_endpoint and api_key fields when you create a embeddings model on the Khoj admin panel
Note: Khoj only recomputes embeddings of modified content, not all content sent to it. So unless there are GBs of new data, the embeddings to recompute should be smaller

stevennt commented 2 months ago

Hi @debanjum, my use-case is as follows: supposed you have a VIP user who you want to onboard. He has, say, 100GB of existing data in various places. Time is not a problem, he wants to start next week. But if he has to go through all the steps to index 100GB of data himself (the process which will certainly have issues here and there given that data size and data quality), then it will be a deal breaker.

So if you can have an onboarding script that takes care of all those workloads and let the user known when all is ready, then it will be a very convenient onboarding process.

This is true for organizations as well when we onboard for multiple users. Many of them may even share the same data set, so an assisted onboarding process can definitely help.