zhudotexe / kani

kani (カニ) is a highly hackable microframework for chat-based language models with tool use/function calling. (NLP-OSS @ EMNLP 2023)
https://kani.readthedocs.io
MIT License
560 stars 30 forks source link

Add CTransformers engine #1

Closed Maknee closed 1 year ago

Maknee commented 1 year ago

Slightly offtopic,: Using the free resources generously provided by HuggingFace, I created a gradio chat demo using kani's LlamaCTransformersEngine on HuggingFace spaces. However, it is somewhat slow since at this time as the HuggingFace CPU basic tier (free tier) provides only 2 VCPU resources and low memory.

Maknee commented 1 year ago

Thanks! Left a couple comments. To fix the code style, run black . from the project root, it'll autoformat the code.

Thanks! Formatted code using black

zhudotexe commented 1 year ago

Merging this into a working branch; I spent a bit of time looking more into the LLaMA tokenizer and I think some of the non-strict tokenization logic can be cleaned up and moved to a common file.