search

marella / chatdocs

Chat with your documents offline using AI.
MIT License
684 stars 99 forks source link

Does the falcon models work with chatdocs ? #35

Closed Ananderz closed 1 year ago

Ananderz commented 1 year ago

I know the falcon models are a little different and that they might not work with Chatdocs. Have anyone tried and got it to run. Which model did you use? GGML, GPTQ or maybe the standard models?

nickorrnah commented 1 year ago

I was able to get the model to work through GGML. Only tricky part was having to find GGML version of the Falcon model on Hugging Face, rather than just using the default one from the leaderboards.

marella commented 1 year ago

Support for the new GGML Falcon models was added recently in ctransformers 0.2.12 Make sure you have installed the latest version:

pip install 'ctransformers>=0.2.12'

You can find the falcon GGML models here. Example:

Normal HF and GPTQ versions should also work.