Open soleblaze opened 1 year ago
When trying with the bindings are you following the steps in the readme?
BUILD_TYPE=metal make libbinding.a
CGO_LDFLAGS="-framework Foundation -framework Metal -framework MetalKit -framework MetalPerformanceShaders" LIBRARY_PATH=$PWD C_INCLUDE_PATH=$PWD go run ./examples -m "/model/path/here" -t 14
Yes. With that example it still appears to use cpu. llama expects -ngl 1
to be passed to it in order to use metal.
using the steps in the readme also outputs this error:
make: Circular llama.cpp/ggml-metal.o <- llama.cpp/ggml-metal.o dependency dropped.
When it uses metal it should output lines prefixed with ggml_metal_init:
when it uses metal. It doesn't do this. If I clone the repo, cd into llama.cpp, and run LLAMA_METAL=1 make
prior to building the libbinding.a then it works correctly.
Adding llama.SetGPULayers(1)
to the llama.New
call on line 33 allows the example to use metal. This fails to run due to a missing ggml-metal.metal file.
if I compile the examples/main.go and copy llama.cpp/ggml-metal.metal into my CWD then it works. running the example via go run fails to find the file.
FWIW I'm also not seeing GPU offload with -ngl 1 and from following the current README steps to build+use metal support.
That's odd. it's working fine for me. When it loads the model do you get any of the ggml_metal_init
lines? I do get sporadic gpu usage where it freezes every X amount of tokens and the gpu utilization dives.
When I try modifying the example to add llama.SetGPULayers(1) it doesn't appear to set that. The example is still using CPU without offloading to metal.
When I use it in local-ai it thinks that my q4_0 model is a f32 model. The same model works fine running llama.cpp directly.
You do need to copy the ggml-metal.metal file from the llama.cpp directory to your CWD for this to work. Otherwise it errors out with a can't find file (null).
Is there a different load path that go-llama.cpp should be using when loading a model in for metal?