search

LostRuins / koboldcpp

A simple one-file way to run various GGML and GGUF models with a KoboldAI UI
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
4.41k stars 319 forks source link

Model loads forever #766

Closed j2wx closed 3 months ago

j2wx commented 3 months ago

I tried to load some GGUF models, but the loading process is stuck at this line: [Threads: 2, BlasThreads: 2, SmartContext: False, ContextShift: True]

I already tried it without ContextShift and SmartContext, but it still won't load. With Oobabooga it works without problem.

henk717 commented 3 months ago

Which models are these?

j2wx commented 3 months ago

The models I tried are Toppy-M-7B and SnowLotus-v2-10.7B.

LostRuins commented 3 months ago

Do you have a link to the specific model quant file that doesn't work?

j2wx commented 3 months ago

Yes, here are the links:

Toppy-M-7B: https://huggingface.co/Undi95/Toppy-M-7B-GGUF/resolve/main/Toppy-M-7B.q4_k_s.gguf SnowLotus-v2-10.7B: https://huggingface.co/BlueNipples/DaringLotus-SnowLotus-10.7b-IQ-GGUF/resolve/main/SnowLotus-v2-10.7B-Q3_K.gguf

LostRuins commented 3 months ago

Hi, I tried Toppy-M-7B and it worked fine. Please try the latest version which is just released. Also, try reducing the number of offloaded layers.

j2wx commented 3 months ago

Thanks, I tried it with the latest version and I get this error while starting:

Warning, GUI failed to start: Reason: No module named 'customtkinter'
File selection GUI unsupported.
customtkinter python module required!
Please check command line: script.py --help

I installed customtkinter, but it still show this error message.

LostRuins commented 3 months ago

What is your OS? For windows and linux you can grab the latest release here.

Otherwise, you can run it without GUI with the command line. For example

python koboldpp.py --model /path/to/your/model.gguf

j2wx commented 3 months ago

I use Linux. I use the "koboldcpp.sh" and I also tried the program from the release.

I tried it with "koboldcpp.py" and I get a error in my language which translates to "`Invalid machine command (memory dump written)". Is the problem related to my OS?

LostRuins commented 3 months ago

Alright. Do you have a GUI? If yes, I suggest just trying this binary https://github.com/LostRuins/koboldcpp/releases/download/v1.62.2/koboldcpp-linux-x64 which should make things easier.

j2wx commented 3 months ago

This is the one I tried. I tried it in a live OS this time to see if it works there, but it gave the same error as with the "koboldcpp.py": Illegal instruction.

henk717 commented 3 months ago

I suspect your trying to use the regular backends on a CPU that does not have AVX2 support. Can you double check your CPU supports AVX2 and if not try the Old CPU or Failsave backends?

gustrd commented 3 months ago

Sometimes this is the error when you try to call a .py file like a .sh file. Try to call with python3 before, like:

python3 ./koboldcpp.py
j2wx commented 3 months ago

My CPU only supports AVX, but I read somewhere that it's enough. I tested it with the Old CPU backend and it works now. Thank you very much for your help.