Summarization on GPU ?

[ijean]

Hi,

I found your repo by reading the article on https://developer-service.blog/how-to-summarize-articles-with-streamlit-and-langchain-with-mistral-7b-on-cpu/. Great article.

How would you do the same but using a GPU instead of the CPU ?

Thanks, Jean

[nunombispo]

Hi Jean,

To run on the GPU and assuming that you have an NVIDIA GPU, then you need to:

• Install transformers with CUDA: pip install ctransformers[cuda]

• Replace the variable llm with:

  llm = CTransformers(model="TheBloke/Mistral-7B-Instruct-v0.1-GGUF",
                      model_file="mistral-7b-instruct-v0.1.Q4_K_M.gguf",
                      config=config,
                      gpu_layers=50)