Open RoryMB opened 6 months ago
@1-ashraful-islam curious if you have an idea on the easiest path since you were doing this previously #1701
I was thinking you could fork llama.cpp and whisper.cpp, modify the swift package dependencies and exclusions such that they both reference the same set of ggml sources, but is there an easier path of building whisper and llama frameworks independently? I haven't wrapped my head around the SPM / Xcode ecosystem..
Sorry for the late reply, I don't know of a better way to resolve this issue. I banged my head around this problem before and gotten nowhere until I separated ggml as dependency in both whisper and llama. I would suggest doing the fork and revert the mentioned commits - until someone figures out a better approach.
I actually just found a better way!! You can include the packages as framework (I got the idea from mlx-swift-examples)
Here's how you can do it:
llama
for llama.cpp. Set the other settings appropriately
llama
(icon should be yellow for framework). Then add llama.cpp
package dependency Under General > Frameworks and Libraries
Repeat the steps for whisper:
In step 2 set the Product name towhisper
for whisper.cpp
In step 3 add whisper.cpp
as package dependency
@ggerganov do you think this info would be useful to include somewhere?
Wow, thanks! This solved the duplicate symbol errors for me. Being new to the whole Apple/Swift landscape, I don't think I would have figured this solution out any time soon.
Hm, great! I haven't tried it, but since it seems to work for @RoryMB then this might be the way to do it. We can add a link to your comment in all relevant examples in llama.cpp
and whisper.cpp
One thing to note here: it seems like GGMLMetalClass is selected from either whisper.framework or llama.framework randomly. This is what I get when I run the application target:
objc[40428]: Class GGMLMetalClass is implemented in both /Users/.../whisper.framework/Versions/A/whisper (0x1020083c0) and /Users/.../llama.framework/Versions/A/llama (0x101e143c0). One of the two will be used. Which one is undefined.
At the moment I haven't seen this to be an issue for either transcription or llm use. If I run into any issue in the future, I will add notes here.
@1-ashraful-islam: Thank you for the instructions on importing Whistler and Llama into a project. While I was able to import them successfully, I encountered issues when trying to run both models simultaneously. Whistler operates as expected, but Llama does not produce any response. Did you expereince a similar issue, any clue why that could happen? Thank you.
I have the same problem. Project compiled with CMake llama.cpp depedency works perfectly. So does project with whisper.cpp only dependency. But when compiled with both dependencies simultaneously LLM functionality breaks at runtime. Using different models cause different errors for example loading Meta-Llama-3.1-8B-Instruct-IQ4_XS.gguf
fails with:
llama_model_load: error loading model: invalid model: tensor '' is duplicated
running exactly the same code just after adding whisper.cpp as library. Using other model I am getting error:
terminate called after throwing an instance of 'std::out_of_range'
As of these two commits: https://github.com/ggerganov/whisper.cpp/commit/3ffc83d90a958e3810f02e49de44abc3a85f9a35 https://github.com/ggerganov/llama.cpp/commit/df334a11251b81fd0b6a0e51e7146e0ba9e973f2
Xcode projects that depend on both whisper.cpp and llama.cpp fail to build with the following error:
Based on the comments in the accompanying pull requests I see that there is good reason for the commits, so I wonder if there is any alternative solution?
Thanks