Closed zxcvxzcv-johndoe closed 9 months ago
@zxcvxzcv-johndoe llama.cpp
has support for AMD GPUs, so to use node-llama-cpp
with support for your GPU you'll have to build llama.cpp
from source with custom options.
To do that, run the following command inside your project:
npx --no node-llama-cpp download
You'll have to set custom environment variables before using the download
command to configure the AMD GPU support of llama.cpp
.
Since I do not have an AMD GPU to properly test it and add convenient support for it in node-llama-cpp
, I'd appreciate it if you could document here all the steps that you had to take to make it work, so I can add more convenient support for it and provide better documentation for this process.
Thanks!
I will add this to my to-do list, I hope I can try this later this week. My PC is still missing half the RAM (broken stick etc.) and it should be back to normal hopefully around Wednesday/Thursday.
Sorry for the late reply.
I tried to follow hipBLAS documentation but that got too complicated for me (I am no expert :) )
CLBlast could be easier alternative, someone commented in this link https://github.com/ggerganov/llama.cpp/issues/2533#issuecomment-1667894211 that he got his 6800XT working with that link's steps.
But if I manually compile llama.cpp with CLBlast support like that, how can I use that with node-llama-cpp?
@zxcvxzcv-johndoe You have to set custom environment variables before using the download
command to set custom cmake
build options, instead of the -D
parameters passed to the cmake
command when you compile llama.cpp
by yourself.
For example, instead of this command:
cmake .. -DBUILD_SHARED_LIBS=OFF -DOVERRIDE_MSVC_FLAGS_TO_MT=OFF -DTUNERS=OFF -DOPENCL_ROOT=C:/OpenCL-SDK-v2023.04.17-Win-x64 -G "Visual Studio 17 2022" -A x64
You'll have to do something like this on Windows:
set NODE_LLAMA_CPP_CMAKE_OPTION_BUILD_SHARED_LIBS=OFF
set NODE_LLAMA_CPP_CMAKE_OPTION_OVERRIDE_MSVC_FLAGS_TO_MT=OFF
set NODE_LLAMA_CPP_CMAKE_OPTION_TUNERS=OFF
set NODE_LLAMA_CPP_CMAKE_OPTION_OPENCL_ROOT=C:/OpenCL-SDK-v2023.04.17-Win-x64
npx --no node-llama-cpp download
@zxcvxzcv-johndoe You have to set custom environment variables before using the
download
command to set customcmake
build options, instead of the-D
parameters passed to thecmake
command when you compilellama.cpp
by yourself.For example, instead of this command:
cmake .. -DBUILD_SHARED_LIBS=OFF -DOVERRIDE_MSVC_FLAGS_TO_MT=OFF -DTUNERS=OFF -DOPENCL_ROOT=C:/OpenCL-SDK-v2023.04.17-Win-x64 -G "Visual Studio 17 2022" -A x64
You'll have to do something like this on Windows:
set NODE_LLAMA_CPP_CMAKE_OPTION_BUILD_SHARED_LIBS=OFF set NODE_LLAMA_CPP_CMAKE_OPTION_OVERRIDE_MSVC_FLAGS_TO_MT=OFF set NODE_LLAMA_CPP_CMAKE_OPTION_TUNERS=OFF set NODE_LLAMA_CPP_CMAKE_OPTION_OPENCL_ROOT=C:/OpenCL-SDK-v2023.04.17-Win-x64 npx --no node-llama-cpp download
Thanks! Those steps worked perfectly.
@zxcvxzcv-johndoe Glad to hear that it worked :)
Can you please check what's the minimum custom environment variables you need to make it compile?
Is only setting set NODE_LLAMA_CPP_CMAKE_OPTION_OPENCL_ROOT=C:/OpenCL-SDK-v2023.04.17-Win-x64
enough without the rest?
I'd want to make the handling of using OpenCL more automatic
Yes, it works only with that one variable you listed.
Also, it might be obvious, and I am not 100% sure, but I think you need to have the VS Community (or better) installed with "Desktop development with C++"?
Software I have installed on my Windows 11: git for windows (git version 2.42.0.windows.2) node.js LTS (20.9.0) python 3.11.6 Visual Studio Community 2022 17.8.0 (using the "Desktop development with C++" workload)
I ran the commands: npm install --save node-llama-cpp (here I added the "type": "module" to the package.json, not sure if necessary and no idea why it's not there automatically?) set NODE_LLAMA_CPP_CMAKE_OPTION_OPENCL_ROOT=C:\LLM\GitHub\OpenCL-SDK-v2023.04.17-Win-x64 npx --no node-llama-cpp download
Then you are able to chat: npx --no node-llama-cpp chat --model <path-to-a-model-file-on-y
:tada: This issue has been resolved in version 3.0.0-beta.12 :tada:
The release is available on:
v3.0.0-beta.12
Your semantic-release bot :package::rocket:
Feature Description
I cannot use this with AMD GPU.
The Solution
Support AMD GPU,
Considered Alternatives
I don't want to sell my AMD GPU.
Additional Context
No response
Related Features to This Feature Request
Are you willing to resolve this issue by submitting a Pull Request?
No, I don’t have the time and I’m okay to wait for the community / maintainers to resolve this issue.