search

zhudotexe / kani

kani (カニ) is a highly hackable microframework for chat-based language models with tool use/function calling. (NLP-OSS @ EMNLP 2023)
https://kani.readthedocs.io
MIT License
558 stars 30 forks source link

LLaMA prompting: use itertools grouping rather than string buffer #23

Closed zhudotexe closed 1 year ago

zhudotexe commented 1 year ago

This improves the mechanism by which LLaMAv2 prompts are built to use an itertools group rather than a string buffer to merge consecutive system-user messages (which both need to be wrapped in a single [INST] [/INST] tag).

The PR also adds tests to assert that the prompt is built correctly (which you can view to see the expected grouping behaviour).

zhudotexe commented 1 year ago

Note: code style failures are a black 23.10 update; will go through and format the full project after pending PRs are merged