Bug: llava doesn't use GPU on Windows

[qinbatista]

What happened?

after compiling the llava and running it on Windows, it is very slow because using the CPU, I don't know how can I switch to GPU acceleration.

Name and Version

version: 3658 (f1485161) built with cc (GCC) 14.2.0 for x86_64-w64-mingw32

What operating system are you seeing the problem on?

No response

Relevant log output

llama_kv_cache_init:        CPU KV buffer size =  1024.00 MiB
llama_new_context_with_model: KV self size  = 1024.00 MiB, K (f16):  512.00 MiB, V (f16):  512.00 MiB
llama_new_context_with_model:        CPU  output buffer size =     0.12 MiB
llama_new_context_with_model:        CPU compute buffer size =   164.01 MiB
llama_new_context_with_model: graph nodes  = 1030
llama_new_context_with_model: graph splits = 1

[leol333]

Same problem here! Have you fixed this?

[qinbatista]

Same problem here! Have you fixed this?

I am not an AI developer, so it is hard for me to check the source code, but I think I need to open the GPU setting somewhere,

[ggddddd]

-ngl, --gpu-layers N number of layers to store in VRAM this works for me

[qinbatista]

-ngl, --gpu-layers N number of layers to store in VRAM this works for me

could you let me know where should I use the command line? Thank you very much

[leol333]

Did you use this command line and fixed the problem? May I ask where you add the code? Thank youuu

[qinbatista]

Did you use this command line and fixed the problem? May I ask where you add the code? Thank youuu

it is not fixed, I found I didn't compile it with the GPU command line, so close the issue and opened another issue I

[qinbatista]

Did you use this command line and fix the problem? May I ask where you add the code? Thank you

I downloaded the compiled exe files from the tag, and run with -ngl 99 parameters behind and it works