Support Gemma 2 Model Family for Offline Chat - Githubissues

khoj-ai / khoj

Your AI second brain. Self-hostable. Get answers from the web or your docs. Build custom agents, schedule automations, do deep research. Turn any online or local LLM into your personal, autonomous AI (e.g gpt, claude, gemini, llama, qwen, mistral).

https://khoj.dev

GNU Affero General Public License v3.0

14.09k stars 702 forks source link

Support Gemma 2 Model Family for Offline Chat #855

Closed debanjum closed 3 months ago

debanjum commented 3 months ago

Overview

Gemma 2 is a new open model family by Google. They've released a 9B, 29B param model. A 2B model is also expected.
It performs really well on the Chatbot arena and shows good performance when testing within Khoj as well.
Llama.cpp support for Gemma 2 architecture seems to have stabilized
If Gemma 2 performs well in further testing, it can be made the default offline chat model for Khoj
- Once the 2B param model is released, the model size to download can be automatically chosen based on (V)RAM available

Major

Support Gemma 2 for Offline Chat
Improve and fix chat model prompts for better, consistent context

Minor

Fix and improve offline chat actor, director tests
Improve offline chat truncation to consider chat message delimiter tokens