search

edgar971 / open-chat

A self-hosted, offline, ChatGPT-like chatbot with different LLM support. 100% private, with no data leaving your device.
MIT License
64 stars 8 forks source link

How to deploy Mistral-7B-Code-16K model? #14

Open charles-cai opened 7 months ago

charles-cai commented 7 months ago

OpenChat-Cuda works great with my 3080 10GB GPU under Unraid Docker environment.

Could someone explain how to swap the LLM model to https://huggingface.co/TheBloke/Mistral-7B-Code-16K-qlora-GGUF ?

I want to test code generation comparison between Llama 2 7B vs Mistral 7B Code.

Many thanks. Charles