mistral chat example requires cuda

[MarcSkovMadsen]

When running the example it takes really a long time to download the model. And when its done I get

Could not find module 'C:\Users\USERNAME\AppData\Local\hatch\env\virtual\panel-chat-examples\yC6ANKqz\panel-chat-examples\Lib\site-packages\ctransformers\lib\cuda\ctransformers.dll' (or one of its dependencies). Try using the full path with constructor syntax.

"""
Demonstrates how to use the ChatInterface widget to create a chatbot using
Mistral through CTransformers.
"""

import panel as pn
from ctransformers import AutoModelForCausalLM

pn.extension(design="material")

llms = pn.state.cache["llms"] = pn.state.cache.get("llms", {})

async def callback(contents: str, user: str, instance: pn.widgets.ChatInterface):
    if "mistral" not in llms:
        instance.placeholder_text = "Downloading model; please wait..."
        llms["mistral"] = AutoModelForCausalLM.from_pretrained(
            "TheBloke/Mistral-7B-Instruct-v0.1-GGUF",
            model_file="mistral-7b-instruct-v0.1.Q4_K_M.gguf",
            gpu_layers=1,
        )

    llm = llms["mistral"]
    response = llm(contents, stream=True, max_new_tokens=1000)
    message = ""
    for token in response:
        message += token
        yield message

chat_interface = pn.widgets.ChatInterface(callback=callback, callback_user="Mistral")
chat_interface.send(
    "Send a message to get a reply from Mistral!", user="System", respond=False
)
chat_interface.servable()

Todo

We need working ui tests, images and videos for these examples.

• [ ] Thumbnails for all mistral examples
• [ ] Videos for all mistral examples
• [ ] Working UI tests in user.py

[ahuang11]

Do you know how to transfer the videos efficiently from test-result to docs/assets? Or did you do them one by one?

[MarcSkovMadsen]

I did them One by One

[ahuang11]

I added the thumbnails and videos here https://github.com/holoviz-topics/panel-chat-examples/pull/64