nextcloud / context_chat_backend

GNU Affero General Public License v3.0
6 stars 5 forks source link

[install-help]: Is it possible to change the default model? #73

Closed aef5748 closed 1 month ago

aef5748 commented 1 month ago

If I want to change the default model dolphin-2.2.1-mistral-7b.Q5_K_M.gguf to another model like as Meta-Llama-3.1-8B-Instruct.Q4_K_M.gguf or a custom-defined model, how should I modify it?

kyteinsky commented 1 month ago

hello, I'd be happy to explain. The config contains many options but for this one, the llm option is important. All the listed items there llama, hugging_face, etc. are examples of configs for the LLM model to load/the LLM backend to use. Right now only the first one is used. That should be llama for you but the new config allows to use the TaskProcessing tasks for the answer generation too (nc_texttotext).

Now, to answer the real question,

  1. place the GGUF model file inside /nc_app_context_chat_backend_data/ inside the docker container nc_app_context_chat_backend. Use this command for that:
    docker cp <path_to_gguf_file> nc_app_context_chat_backend:/nc_app_context_chat_backend_data/
  2. modify the config file (/nc_app_context_chat_backend_data/config.yaml) and set the new model's filename at llm->llama->model_path
  3. restart the docker container
    docker restart nc_app_context_chat_backend

For llama, you can see all the available options here: https://api.python.langchain.com/en/latest/llms/langchain_community.llms.llamacpp.LlamaCpp.html

ctft1 commented 1 month ago

I'm currently using the nc_texttotext parameter in the config.yaml, so that my LocalAI instance (already setup in NC) handles the requests.

Everything works fine, but how can I change the default model used for the requests? Changing the "Default Completion Model to use" in "Admin Settings > Articifial Intelligence" in Nextcloud does not change anything, it is always the "gpt4" equivalent that seems to be used (Hermes-2-Pro-Llama-3-8B as I'm using the LocalAI AIO docker)

In the "Task: Context Chat" section, the selected item (and the only one available) is "Nextcloud Assistant Context Chat Provider"

kyteinsky commented 1 month ago

what are the options shown in the model selection dropdown in "Admin Settings > Articifial Intelligence > OpenAI Integration"? Is it only "gpt4"? Changing this value would be enough to switch the model for normal tasks and context chat.

If you're running background job workers, you should run them with a timeout like so, at least 2 of them:

set -e; while true; do sudo -u www-data occ background-job:worker -v -t 60 "OC\TaskProcessing\SynchronousBackgroundJob"; done
ctft1 commented 1 month ago

I have all my downloaded models appearing in the list (+ the default models from the LocalAI AIO like gpt-4, gtp-4o, and so on..)

It finally worked after stopping / restarting the Docker Nextcloud containers.

I'm running 2 workers (as I need 1 for processing the Context Chat request + 1 more to handle the request when it's forwarded to the nc_texttotext) and now I understand the workers had no timeout so that explains why they were caching the old model... For those looking for more information about this and how to automate the workers, it's described on the page https://docs.nextcloud.com/server/latest/admin_manual/ai/overview.html#ai-overview-improve-ai-task-pickup-speed

Thank you!

kyteinsky commented 1 month ago

Glad it worked out!