'ggml-metal.metal' not found

[Solido]

Thanks for the initiative!

I've setup a quick sandbox on a mac M1. Here's my log when launching:

/de/kherud/llama/Mac/x86_64
'ggml-metal.metal' not found
Extracted 'libllama.dylib' to '/var/folders/kk/rpkp7skj60b2_srw93wnl9yh0000gn/T/libllama.dylib'

OSInfo run as Mac/x86_64 but ggml-metal is not in the jar archive and so cannot be decompressed to the tempdir.

[mbofb]

similar:

/de/kherud/llama/Mac/aarch64
'ggml-metal.metal' not found
Exception in thread "main" java.lang.UnsatisfiedLinkError: No native library found for os.name=Mac, os.arch=aarch64, paths=[/Users/userxyz/Downloads/git/llama.cpp/build/:/Users/userxyz/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.]
    at de.kherud.llama.LlamaLoader.loadNativeLibrary(LlamaLoader.java:141)
    at de.kherud.llama.LlamaLoader.initialize(LlamaLoader.java:66)
    at de.kherud.llama.LlamaModel.<clinit>(LlamaModel.java:27)
    at io.kgard.sandbox.textgenerate.JavaBindingsForllamacpp.main(JavaBindingsForllamacpp.java:17)

/Users/userxyz/Downloads/git/llama.cpp/build/ contains:

/Users/userxyz/Downloads/git/llama.cpp/build/ggml-metal.metal
/Users/userxyz/Downloads/git/llama.cpp/build/libggml_shared.dylib
/Users/userxyz/Downloads/git/llama.cpp/build/libllama.dylib

[ashwanthkumar]

I fixed this issue by creating the following folder structure and copying the ggml-metal.metal to that directory:

./src/main/resources/de/kherud/llama/Mac/aarch64

[kherud]

Thanks for reporting 👍 there was a problem with the github build workflow. This problem should hopefully be fixed with the latest version 2.2.1. Feel free to re-open otherwise.

[claudionieder]

Hi, I think the same happened now for version 3.0.0. No ggml-metal.metal in the jar file.

[kherud]

Hey @claudionieder did you test if it doesn't work? llama.cpp now has an option which is used to embed the metal file in the shared library. I tested it on my machine and it seems to work. If it doesn't for you I would be happy for feedback, though.

[claudionieder]

Hi @kherud,

please bear with me, I am no expert in this, so I don't know whether this ggml-metal.metal file is needed or just nice to have. So my little program that uses llama-3.0.0.jar works, did not break after switching from the previous version to 3.0.0. Everything looks working like before, except that at startup I now have this output where in the second line it tells me that ggml-metal.metal is not found:

/de/kherud/llama/Mac/aarch64
'ggml-metal.metal' not found
Extracted 'libllama.dylib' to '/var/folders/93/8hnm_v9x5dq9mwdhzlf0pmcr0000gq/T/libllama.dylib'
Extracted 'libjllama.dylib' to '/var/folders/93/8hnm_v9x5dq9mwdhzlf0pmcr0000gq/T/libjllama.dylib'

and I noticed, that while the previous version of the jar file had ggml-metal.metal in the Mac directories next to the dylib files, the new jar does not have it. That's why I reported it in this issue.

But as I have zero knowledge on what this file is good for I can't judge whether it missing is causing any adverse effect, besides outputting that error message. What I can confirm to you for sure is, it does not make my program non-functional. I just feared, that because of it missing, GPU acceleration would maybe not work, but I cannot tell.

If you know, that it missing is no big deal, except for that "not found" message, then please just ignore my issue.

Thanks!

[kherud]

So my little program that uses llama-3.0.0.jar works, did not break after switching from the previous version to 3.0.0.

That's nice to hear @claudionieder, thanks for the feedback!

But as I have zero knowledge on what this file is good for I can't judge whether it missing is causing any adverse effect, besides outputting that error message. What I can confirm to you for sure is, it does not make my program non-functional. I just feared, that because of it missing, GPU acceleration would maybe not work, but I cannot tell.

In the past the metal file was required to be present and I think the library wouldn't run without it at all. By the time version 3.0 was released, llama.cpp added an option to embed the file in the shared library (i.e., in libllama.dylib), however. If the library is compiled with -DLLAMA_METAL_EMBED_LIBRARY=ON (which is the case for the pre-compiled libraries of this repository) it should no longer be necessary to have the metal file separately.

Everything looks working like before, except that at startup I now have this output where in the second line it tells me that ggml-metal.metal is not found

The error is actually a left-over from previous versions where the file was necessary. I'll remove it in the next release!