search

lemanschik / node-llama-cpp

Run AI models locally on your machine with node.js bindings for llama.cpp. Force a JSON schema on the model output on the generation level
https://withcatai.github.io/node-llama-cpp/
MIT License
0 stars 0 forks source link

on windows #1

Closed lemanschik closed 6 days ago

lemanschik commented 1 week ago

Issue description

does not work with cuda

Expected Behavior

D:\CUDA\v12.5\bin

Actual Behavior

// https://developer.nvidia.com/cuda-downloads // silent mode by executing the package with the -s flag. // npx ipull lmstudio-community/Meta-Llama-3-8B-Instruct-GGUF //On Linux, you can monitor GPU usage with this command: // watch -d nvidia-smi // set CUDACXX=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\bin\nvcc.exe // export CUDACXX=/usr/local/cuda-12.2/bin/nvcc // LLAMA_CUDA npx --no node-llama-cpp download --cuda

/**

-- Could not find nvcc, please set CUDAToolkit_ROOT. CMake Warning at llama.cpp/CMakeLists.txt:499 (message): CUDA not found

-- Warning: ccache not found - consider installing it for faster compilation or disable this warning with LLAMA_CCACHE=OFF */ // set CUDAToolkit_ROOT=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5 // set CUDAToolkit_ROOT=D:\CUDA\v12.5\bin

Steps to reproduce

sss

My Environment

Dependency Version
Operating System
CPU Intel i9 / Apple M1
Node.js version x.y.zzz
Typescript version x.y.zzz
node-llama-cpp version x.y.zzz

Additional Context

ssss

Relevant Features Used

Are you willing to resolve this issue by submitting a Pull Request?

Yes, I have the time, and I know how to start.

lemanschik commented 1 week ago

https://github.com/lemanschik/node-llama-cpp/blob/master/src/cli/commands/DownloadCommand.ts

lemanschik commented 1 week ago

https://github.com/lemanschik/node-llama-cpp/blob/2137c4634751efd0cd0ce284e1941fade3b860a0/src/utils/compileLLamaCpp.ts#L35

CMake Warning at llama.cpp/CMakeLists.txt:409 (message): LLAMA_CUBLAS is deprecated and will be removed in the future.

Use LLAMA_CUDA instead

lemanschik commented 1 week ago

-- Could not find nvcc executable in path specified by environment variable CUDAToolkit_ROOT=D:\CUDA\v12.5\bin CMake Warning at llama.cpp/CMakeLists.txt:499 (message): CUDA not found

chainstarup commented 1 week ago

npx --no node-llama-cpp download --gpu cuda Repo: ggerganov/llama.cpp Release: b3265 GPU: CUDA

chainstarup commented 1 week ago

in beta

chainstarup commented 1 week ago

13

I have tried it on a different PC now and it works fine. So I had absolutely no idea why it's not working on this one. As CUDA_PATH is correctly setup in my system variables.

Then looking into it further, by uninstalling the 'Build Tools' of Visual Studio and only having the Community IDE installed, CMake used the IDE instead of the Build Tools and then it started working fine.

Share Edit Follow Flag edited Jun 19, 2019 at 11:21 answered Jun 19, 2019 at 10:53 Mineral's user avatar Mineral 36711 gold badge22 silver badges1111 bronze badges 17

I just ran into the same issue with the Build Tools. If you want to keep the Build Tools installed, you just need to copy everything from: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.4\extras\visual_studio_integration\MSBuildExtensions To: C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\MSBuild\Microsoft\VC\v160\BuildCustomizations Change your CUDA and VS versions in those paths as necessary. For some reason the CUDA toolkit installer doesn't consider the Build Tools installs when choosing where to add the integrations

lemanschik commented 6 days ago

Final Conclusion is

NVIDIA CUDA TOOLS get Copyed only to the most recent Visual Studio\ eg 2022 and then there gets a older version used that is missing the files only uninstalling older versions and keeping 2022 would be sufficent