search

LostRuins / koboldcpp

A simple one-file way to run various GGML and GGUF models with KoboldAI's UI
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
4.35k stars 312 forks source link

Search for cuda_path on Windows #865

Closed jojorne closed 1 month ago

jojorne commented 1 month ago

Let's add dll directory for cuda on Windows too.

jojorne commented 1 month ago

I also delete the two cuda dlls (cudart64_110.dll, cudart64_12.dll) shipped with the repository so it can find the ones installed by my cuda installation. So changing the search path order will affect this.

jojorne commented 1 month ago

.gitignore

/cublas64_11.dll
/cublasLt64_11.dll
/cudart64_110.dll
/cublas64_12.dll
/cublasLt64_12.dll
/cudart64_12.dll

It would become unnecessary.

LostRuins commented 1 month ago

Thanks, will take some time to review this later

jojorne commented 1 month ago

btw this works too: pip install nvidia-cublas-cu11 nvidia-cuda-runtime-cu11 --index-url https://download.pytorch.org/whl/cu118 pip install nvidia-cublas-cu12 nvidia-cuda-runtime-cu12 --index-url https://download.pytorch.org/whl/cu121

Then we look inside the site-packages\nvidia\cublas\bin and site-packages\nvidia\cuda_runtime\bin.

These packages are intended for runtime use and do not currently include developer tools (these can be installed separately). - CUDA Installation Guide for Microsoft Windows