search

sobelio / llm-chain

`llm-chain` is a powerful rust crate for building chains in large language models allowing you to summarise text and complete complex tasks
https://llm-chain.xyz
MIT License
1.36k stars 132 forks source link

Is there a way to run GGUF models? #295

Open tirithen opened 5 months ago

tirithen commented 5 months ago

I want to run the tinyllama model and I wonder if there a way to run GGUF models with this crate. It seem much more common that models are using the GGUF format over the GGML format for models and converting from GGUF -> GGML seem non-tivial. At least from my limited skillset.

Juzov commented 5 months ago

Hi the underlying code is wrapped around llama.cpp. Check the requirements from that codebase. If we need to upgrade the llama.cpp code, a pr would be appreciated.