search

LostRuins / koboldcpp

Run GGUF models easily with a KoboldAI UI. One File. Zero Install.
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
4.84k stars 342 forks source link

A loading alternative via traditional "main" binaries #129

Closed appvoid closed 1 year ago

appvoid commented 1 year ago

I am running the latest code.

I was expecting a way to load the models from gptneox directly by running some kind of binary "main-gptneox" or something similar, instead, I see a ".so" library. Is there a way to load those models somehow?

I converted a pythia model using the neox conversion script from this repo but not look running it yet. I've tried running it with binaries from gptneox.cpp but not luck neither.

./koboldcpp_noavx2.so --model ../gptneox.cpp/pythia-1.4b-gpt4all-pretrain/ggml-model-f32.bin Segmentation fault (core dumped)

LostRuins commented 1 year ago

You're running the wrong file. The .so is a library that is used by the python script. You should run koboldcpp.py

./koboldcpp.py --model ../gptneox.cpp/pythia-1.4b-gpt4all-pretrain/ggml-model-f32.bin