Use set, inferred max token limits wherever chat models are used

khoj-ai / khoj

Your AI second brain. Get answers to your questions, whether they be online or in your own notes. Use online AI models (e.g gpt4) or private, local LLMs (e.g llama3). Self-host locally or use our cloud instance. Access from Obsidian, Emacs, Desktop app, Web or Whatsapp.

https://khoj.dev

GNU Affero General Public License v3.0

12.63k stars 640 forks source link

Use set, inferred max token limits wherever chat models are used #713

Closed debanjum closed 5 months ago

debanjum commented 5 months ago

User configured max tokens limits weren't being passed to send_message_to_model_wrapper
One of the load offline model code paths wasn't reachable. Remove it to simplify code
When max prompt size isn't set infer max tokens based on free VRAM on machine
Use min of app configured max tokens, vram based max tokens and model context window