How to use GPU to run the modes, like to set "-ngl N, -- n-Gpu-layers N" in the llama.cpp project?

[Void-nebula]

Here's what I've tried in my project:

ModelParameters modelParams = new ModelParameters
                .Builder()
                .setNGpuLayers(nGpuLayers)
                .build();

'nGpuLayers' is a integer which is the same value as when I used in llama.cpp project. However I found in the task manager that the GPU seems to not work at all when the model is running, may I ask why and thank you!

[kherud]

Hi, you are probably using the pre-compiled llama.cpp library of this repository. We currently only provide support for CPU inference since there are too many ways to compile the library. For GPU support, you have to compile the library yourself. Please refer to https://github.com/kherud/java-llama.cpp#setup-required

[Void-nebula]

Hi, you are probably using the pre-compiled llama.cpp library of this repository. We currently only provide support for CPU inference since there are too many ways to compile the library. For GPU support, you have to compile the library yourself. Please refer to https://github.com/kherud/java-llama.cpp#setup-required

Thank you so much!