search

LostRuins / koboldcpp

A simple one-file way to run various GGML and GGUF models with KoboldAI's UI
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
4.36k stars 312 forks source link

there is no way to use a gpu while safe mode? #794

Open blenderman94 opened 2 months ago

blenderman94 commented 2 months ago

hey im wondering about trying ot somethin special i have a cpu with sse but no avx somehow i can force kobold to use my gpu while in safe mode?

LostRuins commented 2 months ago

I think safe mode prevents loading of all non-essential drivers, so probably not.

blenderman94 commented 2 months ago

culd i just edit the source code and make a profile for my experiment? or its a bit more difficult then that?

aleksusklim commented 2 months ago

Name any single program that is able to use CUDA in safe mode? Safe mode is very limited, essentially as if you didn't have any GPU drivers installed. Instead, you should fix the main problem that prevents you to use normal Windows operation.

yggdrasil75 commented 2 months ago

if you are wanting to do it as a "minimum windows impact" test, could try running windows ce instead, dont know how well it handles cuda, but it can run on like 100mb memory or something.

blenderman94 commented 2 months ago

i was just wondering. just for the sake of experimentation.

httkl commented 1 month ago

Same problem here: My CPU has no avx2 instruction (AMD FX8350) and even though I have an Nvidia RTX 3060/12GB I cannot use it, neither with CUDA nor Vulkan. Only CPU gets used which is very slow. This is with Linux. (A1111 and oobabooga run fine here without problems.)

LostRuins commented 1 month ago

You can select (Old CPU) for CLBlast and Vulkan, try those.

httkl commented 1 month ago

Thank you. Did that and Vulkan works fine (and very fast - pygmalion-2-7b.Q5_1.gguf) for text generation (CLBlast works as well but is much slower), but the stable diffusion backend uses the CPU then and takes about 365 sec/it which means about 2 hours for 20 steps. :-| So I guess the best solution is to build kobaldcpp by myself without AVX2.