search

amaiya / onprem

A tool for running on-premises large language models with non-public data
https://amaiya.github.io/onprem
Apache License 2.0
684 stars 32 forks source link

update from ggml to gguf #1

Closed amaiya closed 12 months ago

amaiya commented 1 year ago

New model: https://huggingface.co/TheBloke/Yarn-Llama-2-7B-64K-GGUF

rabilrbl commented 1 year ago

@amaiya Is there any special bindings for GGML format in the project?

Just by updating llama-cpp-python to latest version should work right?

amaiya commented 1 year ago

@rabilrbl No, there are no special bindings. Updating llama-cpp-python should work (as long as you don't use the default GGML models and supply the URL to a GGUF model).

amaiya commented 12 months ago

https://huggingface.co/TheBloke/WizardLM-13B-V1.2-GGUF https://huggingface.co/TheBloke/Wizard-Vicuna-7B-Uncensored-GGUF