search

guinmoon / LLMFarm

llama and other large language models on iOS and MacOS offline using GGML library.
https://llmfarm.site
MIT License
1.06k stars 64 forks source link

Crashes on macOS with Llama2 model #6

Closed aehlke closed 8 months ago

aehlke commented 9 months ago

Let me know if you don't have access to the crash logs via TestFlight. Latest stable macOS release, your Llama2 7B model file, default settings. After chatting a few times in a row, it crashes.

guinmoon commented 9 months ago

Let me know if you don't have access to the crash logs via TestFlight. Latest stable macOS release, your Llama2 7B model file, default settings. After chatting a few times in a row, it crashes.

Unfortunately, I don’t see any crash reports for the MacOS version. Tell me, does it crash when metal is on?

aehlke commented 9 months ago

LLMFarm crashes.zip I believe I was exclusively testing Metal because without it's unusably slow. I tried it without Metal once but didn't try to repro the crashes.

aehlke commented 9 months ago
image

this crashed with metal disabled

guinmoon commented 9 months ago

I have updated the version of llama.cpp and will soon upload it to testflight, I hope the crashes will stop in the new version. Unfortunately, from your reports, I was unable to draw an unambiguous conclusion about the cause of the crashes.

aehlke commented 9 months ago

it would be nice to use the newly merged llama.cpp ios build as well. no examples for it exist yet

guinmoon commented 9 months ago

Tell me, did the update solve the problem?

aehlke commented 9 months ago
image

doesn't seem to crash anymore! but randomly switches to talking in german mid-sentence sometimes, strange, and about topics I didn't bring up. I don't have experience yet with Llama2.

guinmoon commented 9 months ago

This is because the context is shifted incorrectly when overflowing. In the new version of llama.cpp it seems to be possible to do it correctly. I will add it to the update as soon as I figure it out.

guinmoon commented 8 months ago

Tell me, did the update solve the problem?

aehlke commented 8 months ago

unfortunately it started reverting to german and then long rambling about a book I didn't mention after a couple messages (continuing the chat I had from above, without creating a new one - let me know if there's something else I should try)

aehlke commented 8 months ago

Is there a config that is working for you on your device?

guinmoon commented 8 months ago

i use this config for test llama2 llama-2-7b-chat-q4_K_M_1693728799.json

aehlke commented 8 months ago

Thanks. I just realized Orca 3B works with Metal. So it was just llama2 3B that spews nonsense. I'll try your config.

aehlke commented 8 months ago

Is it possible to reduce the UI hang on model initialization? I couldn't find any open src yet for doing multi-process control of it in swift

guinmoon commented 8 months ago

Is it possible to reduce the UI hang on model initialization? I couldn't find any open src yet for doing multi-process control of it in swift

I know how to make swiftui not hang during model loading. I will do it in one of the next releases. Right now there are many more priority tasks. But thanks for the offer of help.

aehlke commented 7 months ago

FYI I was able to easily do it in my fork by changing AI to an actor (with a couple associated changes). I won't contribute the changes back because the fork's diverged a lot but it's there if you need the reference.

guinmoon commented 7 months ago

FYI I was able to easily do it in my fork by changing AI to an actor (with a couple associated changes). I won't contribute the changes back because the fork's diverged a lot but it's there if you need the reference.

I seem to have solved the problem in the new version. I am new to swift so would love to see your solution. But I don't see your fork for some reason.

aehlke commented 7 months ago

https://github.com/lake-of-fire/SwiftWhisperStream/blob/master/Sources/SwiftLlama/AI.swift

(which will be launched at https://chatonmac.com if you're curious)

just FYI some minor style feedback, you don't need parens for if-statements and the like in swift (can save you some typing)