search

guinmoon / LLMFarm

llama and other large language models on iOS and MacOS offline using GGML library.
https://llmfarm.site
MIT License
1.06k stars 64 forks source link

Will LLMFarm support llama2? #4

Closed realcarlos closed 11 months ago

guinmoon commented 11 months ago

if someone makes a ggml version

guinmoon commented 11 months ago

This models works fine.

jhen0409 commented 11 months ago

FYI: q2_k is not working in my tests, I got bad_alloc crash or unexpectedly reached end of file (enabling Extended Virtual Addressing). q4_0 works fine.

I also trying integrate llama.cpp to my iOS apps (latest commit), the problem is looks same with here.

guinmoon commented 11 months ago

FYI: q2_k is not working in my tests, I got bad_alloc crash or unexpectedly reached end of file (enabling Extended Virtual Addressing). q4_0 works fine.

I also trying integrate llama.cpp to my iOS apps (latest commit), the problem is looks same with here.

In the current version of llmfarm, I use this commit. q2_k works fine on iphone(without metal) and Mac.

guinmoon commented 11 months ago

FYI: q2_k is not working in my tests, I got bad_alloc crash or unexpectedly reached end of file (enabling Extended Virtual Addressing). q4_0 works fine.

I also trying integrate llama.cpp to my iOS apps (latest commit), the problem is looks same with here.

You may have compiled llmfarm_core with -DGGML_QKK_64 in Package.swift. You need to rebuild the project without this flag, or requantize the models for QKK_64.