GPU version build not using GPU

[dspasyuk]

Hi Everyone,

I am trying to build llama-node for GPU, I followed the guide in the readme https://llama-node.vercel.app/docs/cuda but the version of the llam-cpp I get from a manual build uses CPU not GPU. When I build llama-cpp directly in llama-sys folder using the following command:

make clean && LLAMA_CUBLAS=1 make -j It gives me perfectly fine GPU executable file which works no problem.

Am I missing something? Here is my full build commands:

git clone https://github.com/Atome-FE/llama-node.git cd llama-node/ rustup target add x86_64-unknown-linux-musl git submodule update --init --recursive pnpm install --ignore-scripts cd packages/llama-cpp/ pnpm build:cuda

Then I get libllama.so file in my ~/.llama-node which when used does not use GPU: Here my script to run it:

import { LLM } from "llama-node"; import { LLamaCpp } from "llama-node/dist/llm/llama-cpp.js"; import path from "path"; const model = path.resolve(process.cwd(), "~/CODE/models/vicuna-7b-v1.3.ggmlv3.q4_0.bin"); const llama = new LLM(LLamaCpp); const config = { modelPath: model, enableLogging: true, nCtx: 1024, seed: 0, f16Kv: false, logitsAll: false, vocabOnly: false, useMlock: false, embedding: false, useMmap: true, nGpuLayers: 40 }; const template = How do I train you to read my documents?; const prompt = A chat between a user and an assistant. USER: ${template} ASSISTANT:; const params = { nThreads: 4, nTokPredict: 2048, topK: 40, topP: 0.1, temp: 0.2, repeatPenalty: 1, prompt, }; const run = async () => { await llama.load(config); await llama.createCompletion(params, (response) => { process.stdout.write(response.token); }); }; run();

Any help appreciated

[shaileshminsnapsys]

I am facing the same issue. Can anyone please guide us on this?

[dspasyuk]

I ended up using just llama.cpp. Works very well on the GPU. You can write a simple wrapper in nodejs without rust. I can share the code if you want.

[shaileshminsnapsys]

@deonis1 it will be a great help. Please share the code.

[dspasyuk]

@shaileshminsnapsys no problem the code is here https://github.com/deonis1/llcui

[shaileshminsnapsys]

Thank you @deonis1 , I'll check with the code.

Thank you for your help.

[dspasyuk]

Let me know if you have any issues

[shaileshminsnapsys]

@deonis1 Thank you so much, your code help me alot to achieve my target.

Many Thanks !!

[dspasyuk]

@shaileshminsnapsys no problem, there is a new version if you are interested

[shaileshminsnapsys]

@deonis1 would love to see the new version. Thank you

[dspasyuk]

@shaileshminsnapsys The new version that supports embedding (mongodb or text document) is released. You can find it under the new url: https://github.com/deonis1/llama.cui

[shaileshminsnapsys]

@deonis1 Wow, its amazing.. Thanks, I'll give a try to it for sure.