search

khoj-ai / khoj

Your AI second brain. Self-hostable. Get answers from the web or your docs. Build custom agents, schedule automations, do deep research. Turn any online or local LLM into your personal, autonomous AI (e.g gpt, claude, gemini, llama, qwen, mistral).
https://khoj.dev
GNU Affero General Public License v3.0
14.24k stars 705 forks source link

Feature Request: Add support for using local llms #501

Closed arian81 closed 1 year ago

arian81 commented 1 year ago

It's very easy to spin up a local llm and get an endpoint using ollama (https://ollama.ai/). I think that should be a feature supported by khoj. You can easily use langchain wrapper for Ollama to call local endpoints.

debanjum commented 1 year ago

I've just added support for running any GPT4All supported model in #494. You can try it with pip install --upgrade --pre khoj-assistant.

We'll add support for using other chat LLMs servers which expose an OpenAI style API interface (like local.ai, text-generation-webui etc.) once #407 is implemented.

I'll close this issue for now. But feel free to re-open it if you have other concerns

lrq3000 commented 1 year ago

GPT4All supports many more models than their list of curated models, such as dolphin-mistral. Is there a way to use custom models? Can I specify a path, or do I need to copy the model bin file to C:\Users\<username>\.cache\gpt4all ?

debanjum commented 1 year ago

You can update your chat-modeloffline-chat > chat-model`khoj.yml file (usually at ~/.khoj/khoj.yml) to

debanjum commented 1 year ago

You can update the chat-model field under processor > conversation > offline-chat section in your khoj.yml file at ~/.khoj/khoj.yml to any supported GPT4All chat model.

Current limitations of using non llama2 based models (like mistral) is that we only support the llama2 prompt template (for offline chat). But different chat models expect different chat templates. We'll add support for custom chat templates in a little while once we're done with current Khoj rearchitecture work to add multi-user support

lrq3000 commented 1 year ago

Ok I understand, you are working at a truly impressive speed! Thank you so very much! I wish you lots of success with your project!

yinzixie commented 10 months ago

Hmm, I'm not sure why I only find a log file under the /root/.khoj folder. I am using docker compose.

debanjum commented 9 months ago

Hmm, I'm not sure why I only find a log file under the /root/.khoj folder. I am using docker compose.

Hi @yinzixie, please see the khoj docs to configure khoj chat models etc. Khoj has been rearchitected to use postgres DB for configuration instead of the previous khoj.yml config file