Please add AMD ROCm support

[zxcvxzcv-johndoe]

Feature Description

I cannot use this with AMD GPU.

The Solution

Support AMD GPU,

Considered Alternatives

I don't want to sell my AMD GPU.

Additional Context

No response

Related Features to This Feature Request

• [ ] Metal support
• [ ] CUDA support
• [ ] Grammar

Are you willing to resolve this issue by submitting a Pull Request?

No, I don’t have the time and I’m okay to wait for the community / maintainers to resolve this issue.

[giladgd]

@zxcvxzcv-johndoe llama.cpp has support for AMD GPUs, so to use node-llama-cpp with support for your GPU you'll have to build llama.cpp from source with custom options.

To do that, run the following command inside your project:

npx --no node-llama-cpp download

You'll have to set custom environment variables before using the download command to configure the AMD GPU support of llama.cpp.

Since I do not have an AMD GPU to properly test it and add convenient support for it in node-llama-cpp, I'd appreciate it if you could document here all the steps that you had to take to make it work, so I can add more convenient support for it and provide better documentation for this process.

[zxcvxzcv-johndoe]

Thanks!

I will add this to my to-do list, I hope I can try this later this week. My PC is still missing half the RAM (broken stick etc.) and it should be back to normal hopefully around Wednesday/Thursday.

[zxcvxzcv-johndoe]

Sorry for the late reply.

I tried to follow hipBLAS documentation but that got too complicated for me (I am no expert :) )

CLBlast could be easier alternative, someone commented in this link https://github.com/ggerganov/llama.cpp/issues/2533#issuecomment-1667894211 that he got his 6800XT working with that link's steps.

But if I manually compile llama.cpp with CLBlast support like that, how can I use that with node-llama-cpp?

[giladgd]

@zxcvxzcv-johndoe You have to set custom environment variables before using the download command to set custom cmake build options, instead of the -D parameters passed to the cmake command when you compile llama.cpp by yourself.

For example, instead of this command:

cmake .. -DBUILD_SHARED_LIBS=OFF -DOVERRIDE_MSVC_FLAGS_TO_MT=OFF -DTUNERS=OFF -DOPENCL_ROOT=C:/OpenCL-SDK-v2023.04.17-Win-x64 -G "Visual Studio 17 2022" -A x64

You'll have to do something like this on Windows:

set NODE_LLAMA_CPP_CMAKE_OPTION_BUILD_SHARED_LIBS=OFF
set NODE_LLAMA_CPP_CMAKE_OPTION_OVERRIDE_MSVC_FLAGS_TO_MT=OFF
set NODE_LLAMA_CPP_CMAKE_OPTION_TUNERS=OFF
set NODE_LLAMA_CPP_CMAKE_OPTION_OPENCL_ROOT=C:/OpenCL-SDK-v2023.04.17-Win-x64

npx --no node-llama-cpp download

[zxcvxzcv-johndoe]

@zxcvxzcv-johndoe You have to set custom environment variables before using the download command to set custom cmake build options, instead of the -D parameters passed to the cmake command when you compile llama.cpp by yourself.

For example, instead of this command:

cmake .. -DBUILD_SHARED_LIBS=OFF -DOVERRIDE_MSVC_FLAGS_TO_MT=OFF -DTUNERS=OFF -DOPENCL_ROOT=C:/OpenCL-SDK-v2023.04.17-Win-x64 -G "Visual Studio 17 2022" -A x64

You'll have to do something like this on Windows:

set NODE_LLAMA_CPP_CMAKE_OPTION_BUILD_SHARED_LIBS=OFF
set NODE_LLAMA_CPP_CMAKE_OPTION_OVERRIDE_MSVC_FLAGS_TO_MT=OFF
set NODE_LLAMA_CPP_CMAKE_OPTION_TUNERS=OFF
set NODE_LLAMA_CPP_CMAKE_OPTION_OPENCL_ROOT=C:/OpenCL-SDK-v2023.04.17-Win-x64

npx --no node-llama-cpp download

Thanks! Those steps worked perfectly.

[giladgd]

@zxcvxzcv-johndoe Glad to hear that it worked :) Can you please check what's the minimum custom environment variables you need to make it compile? Is only setting set NODE_LLAMA_CPP_CMAKE_OPTION_OPENCL_ROOT=C:/OpenCL-SDK-v2023.04.17-Win-x64 enough without the rest?

I'd want to make the handling of using OpenCL more automatic

[zxcvxzcv-johndoe]

Yes, it works only with that one variable you listed.

Also, it might be obvious, and I am not 100% sure, but I think you need to have the VS Community (or better) installed with "Desktop development with C++"?

Software I have installed on my Windows 11: git for windows (git version 2.42.0.windows.2) node.js LTS (20.9.0) python 3.11.6 Visual Studio Community 2022 17.8.0 (using the "Desktop development with C++" workload)

I ran the commands: npm install --save node-llama-cpp (here I added the "type": "module" to the package.json, not sure if necessary and no idea why it's not there automatically?) set NODE_LLAMA_CPP_CMAKE_OPTION_OPENCL_ROOT=C:\LLM\GitHub\OpenCL-SDK-v2023.04.17-Win-x64 npx --no node-llama-cpp download

Then you are able to chat: npx --no node-llama-cpp chat --model <path-to-a-model-file-on-y

[github-actions[bot]]

:tada: This issue has been resolved in version 3.0.0-beta.12 :tada:

The release is available on:

• npm package (@beta dist-tag)
• GitHub release
• v3.0.0-beta.12

Your semantic-release bot :package::rocket: