search

zhudotexe / kani

kani (カニ) is a highly hackable microframework for chat-based language models with tool use/function calling. (NLP-OSS @ EMNLP 2023)
https://kani.readthedocs.io
MIT License
560 stars 30 forks source link

Working in Huggingface #20

Closed evanbrociner closed 1 year ago

evanbrociner commented 1 year ago

Hi thank you for this great repo. Any chance you can provide any example of the Huggingface engine. I am having trouble getting it to work. Thank you!

zhudotexe commented 1 year ago

Sure - if you're working with LLaMA v2 or Vicuna (or models that use the same prompt format) we have engines implemented for each of these already and you can just pass model_id. If you are looking to implement your own chat model on HuggingFace, you can use the Vicuna engine as an example of how to implement build_prompt and message_len: https://github.com/zhudotexe/kani/blob/main/kani/engines/huggingface/vicuna.py

See also https://kani.readthedocs.io/en/latest/engines.html#huggingface for more information on the HuggingEngine.

zhudotexe commented 1 year ago

Moving this issue to discussions.