search

holoviz-topics / panel-chat-examples

Examples of Chat Bots using Panels chat features: Traditional, LLMs, AI Agents, LangChain, OpenAI etc
https://holoviz-topics.github.io/panel-chat-examples/
MIT License
105 stars 31 forks source link

Use built-in apply_chat_template #60

Open ahuang11 opened 10 months ago

ahuang11 commented 10 months ago

I tried to design my own apply_template function, but I found that there's one from transformers. https://github.com/holoviz-topics/panel-chat-examples/blob/main/docs/examples/mistral/mistral_with_memory.py

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
chat = [
  {"role": "system", "content": "You are a helpful bot."},
  {"role": "user", "content": "Hello, how are you?"},
  {"role": "assistant", "content": "I'm doing great. How can I help you today?"},
  {"role": "user", "content": "I'd like to show off how chat templating works!"},
]

print(tokenizer.apply_chat_template(chat, tokenize=False))
ahuang11 commented 10 months ago

Something about this isn't supported with ctransformers.

ahuang11 commented 10 months ago

Working example in https://github.com/holoviz/panel/pull/5764, but involves transformers lib too.