search

langchain-ai / langchain-ibm

MIT License
10 stars 10 forks source link

Special tokens of newer models (e.g. Llama3.1) are not handled #17

Closed oezguensi closed 5 days ago

oezguensi commented 2 months ago

Models such as Llama3.1 use other special tokens which are not handled currently.

oezguensi commented 1 month ago

More context: In /libs/ibm/langchain_ibm/chat_models.py the _create_chat_prompt function only handles granite and llama2 models right now. Most modern LLMs use special tokens such as <SYS> which need to be added so that the model returns qualitative results. For example Llama3 will still function without its special tokens but the quality will be poor as it has a hard time to understand what is the System, the Human or the AI Message.

Mateusz-Switala commented 1 month ago

@oezguensi Support for dedicated chat templates for different models on watsonx.ai will be available soon. ETA - a few weeks.

MateuszOssGit commented 1 month ago

@oezguensi Could you please verify your issue with 0.3.0 version of langchain_ibm? Right now, we have covered system prompt in the backend side of watsonx.

Mateusz-Switala commented 5 days ago

When using ChatWatsonx the watsonx Chat API is used underneath. I'm closing the issue since everything seems to works fine now. Moreover, the proper chat template for a particular Chat model is maintained by the watsonx backend and not in the langchain_ibm source code.

fyi @MateuszOssGit