Open ijean opened 9 months ago
Hi Jean,
To run on the GPU and assuming that you have an NVIDIA GPU, then you need to:
Install transformers with CUDA:
pip install ctransformers[cuda]
Replace the variable llm with:
llm = CTransformers(model="TheBloke/Mistral-7B-Instruct-v0.1-GGUF",
model_file="mistral-7b-instruct-v0.1.Q4_K_M.gguf",
config=config,
gpu_layers=50)
Hi,
I found your repo by reading the article on https://developer-service.blog/how-to-summarize-articles-with-streamlit-and-langchain-with-mistral-7b-on-cpu/. Great article.
How would you do the same but using a GPU instead of the CPU ?
Thanks, Jean