search

shroominic / funcchain

⛓️ build cognitive systems, pythonic
https://shroominic.github.io/funcchain/
MIT License
326 stars 22 forks source link

How does Funcchain handle different prompt templates? #19

Open ahuang11 opened 9 months ago

ahuang11 commented 9 months ago

Hi again. I'm wondering, does funcchain handle different chat templates internally?

e.g. for Llama

<s>[INST] <<SYS>>
You are a helpful, respectful and honest assistant. Always answer as helpfully as possible.
<</SYS>>

 [/INST]

vs Mistral

text = "<s>[INST] What is your favourite condiment? [/INST]"
"Well, I'm quite partial to a good squeeze of fresh lemon juice. It adds just the right amount of zesty flavour to whatever I'm cooking up in the kitchen!</s> "
"[INST] Do you have mayonnaise recipes? [/INST]"

References: https://www.reddit.com/r/LocalLLaMA/comments/1afweyw/comment/kofabzx/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button https://www.promptingguide.ai/models/mistral-7b#chat-template-for-mistral-7b-instruct

shroominic commented 9 months ago

Its just using the langchain templating so langchain_community is doing all the work converting every model provider.

ahuang11 commented 9 months ago

I don't think so; I think funcchain is hardcoding prompt formats. https://github.com/shroominic/funcchain/blob/main/src/funcchain/model/patches/llamacpp.py#L269-L283

Llama-cpp-python actually supports different chat_formats

https://github.com/abetlen/llama-cpp-python/blob/b1637c2319936df0ecf1b3eb18ca971b346a147e/llama_cpp/llama_chat_format.py#L609

shroominic commented 9 months ago

yess true we discussed in discord need to look into that!