search

oobabooga / text-generation-webui

A Gradio web UI for Large Language Models.
GNU Affero General Public License v3.0
40.57k stars 5.31k forks source link

Use locally compiled llama.cpp #5816

Open bmtwl opened 7 months ago

bmtwl commented 7 months ago

Discussed in https://github.com/oobabooga/text-generation-webui/discussions/5479

Originally posted by **bmtwl** February 10, 2024 Hello, I'm working on a private branch of llama.cpp to add some features for an eventual PR, but I'd like to try to use it in oobabooga ahead of any PR going in as both a kind of regression test, and because I'd like to use my feature early : ) I didn't find anything in previous discussions, the wiki, the README or anywhere else I have been able to search. Is it possible, and if so is there a documented procedure? Thanks!

I think there are a variety of reasons someone might want to use a local compile of llama.cpp. Maybe there should be an official guide with steps?

Touch-Night commented 7 months ago

I think https://github.com/abetlen/llama-cpp-python?tab=readme-ov-file#development may help

StableLlama commented 7 months ago

With #5627 this would be trivially possible.

StableLlama commented 5 months ago

Sadly #5627 was auto closed so that nobody will see that this issue is still open