search

kherud / java-llama.cpp

Java Bindings for llama.cpp - A Port of Facebook's LLaMA model in C/C++
MIT License
279 stars 28 forks source link

not getting gpu acceleration on windows #43

Closed siddhsql closed 7 months ago

siddhsql commented 7 months ago

i am running version 2.2.1 of this library on a windows PC with intel iris plus graphics 655 and do not see the gpu being used. is this because the underlying llama.cpp does not support it or something else? is there something i can do to get gpu acceleration on windows?

kherud commented 7 months ago

Hi @siddhsql the pre-compiled libraries this Java package provides are CPU only. For GPU support, you have to compile llama.cpp yourself. Please refer to these instructions https://github.com/kherud/java-llama.cpp#setup-required

siddhsql commented 7 months ago

Hi Konstantin,

Thanks for your reply. I built a custom binary of llama.cpp that uses CLBlast. I confirmed that when I run llama.cpp directly using the main.exe program, I get GPU acceleration but I do not get any GPU acceleration when using java-llama.cpp. I have set de.kherud.llama.lib.path to the location where I have jllama.dll and llama.dll. is there something I am missing? Can you provide more instructions?

when using main.exe I see following in the output:

llm_load_tensors: offloading 32 repeating layers to GPU
llm_load_tensors: offloading non-repeating layers to GPU
llm_load_tensors: offloaded 33/33 layers to GPU
llm_load_tensors:        CPU buffer size =    70.31 MiB
llm_load_tensors:     OpenCL buffer size =  3820.93 MiB

which is missing when I use java-llama.cpp

siddhsql commented 7 months ago

nevermind. i fixed it.

On Mon, Jan 29, 2024 at 12:12 PM Siddharth Jain @.***> wrote:

Hi Konstantin,

Thanks for your reply. I built a custom binary of llama.cpp that uses CLBlast. I confirmed that when I run llama.cpp directly using the main.exe program, I get GPU acceleration but I do not get any GPU acceleration when using java-llama.cpp. I have set de.kherud.llama.lib.path to the location where I have jllama.dll and llama.dll. is there something I am missing? Can you provide more instructions?

On Sun, Jan 28, 2024 at 11:58 AM Konstantin Herud < @.***> wrote:

Hi @siddhsql https://github.com/siddhsql the pre-compiled libraries this Java package provides are CPU only. For GPU support, you have to compile llama.cpp yourself. Please refer to these instructions https://github.com/kherud/java-llama.cpp#setup-required

— Reply to this email directly, view it on GitHub https://github.com/kherud/java-llama.cpp/issues/43#issuecomment-1913705204, or unsubscribe https://github.com/notifications/unsubscribe-auth/A6NWEK7FCAGSTNQVLH7OTBLYQ2UXBAVCNFSM6AAAAABCNWSDMCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJTG4YDKMRQGQ . You are receiving this because you were mentioned.Message ID: @.***>