Support for Phi-3 models

[retteghy]

see huggingface for the models

[guinmoon]

Hi. work normal with this template

<|user|>
{{prompt}}<|end|>
<|assistant|>

And BOS option enabled.

[paulilioaica]

Hi. How can I make it generate until EOS? If I select the option, the app crashes.

[retteghy]

Hi. work normal with this template

<|user|>
{{prompt}}<|end|>
<|assistant|>

And BOS option enabled.

BOS is enabled, I have set that prompt, but I am getting an error as reply for every message: Load Model Error: [Error] modelLoad Error Load Model Error: [Done]

[jekriske-lilly]

@guinmoon when you say "works normal" are you referring to the development version or the version in the App store?

The stable version from the app store isn't honoring the end token and the app crashes if you try enabling EOS.

[guinmoon]

development version

[Cimplex]

Hi. work normal with this template

<|user|>
{{prompt}}<|end|>
<|assistant|>

And BOS option enabled.

BOS is enabled, I have set that prompt, but I am getting an error as reply for every message: Load Model Error: [Error] modelLoad Error Load Model Error: [Done]

In the TestFlight version I’m using ‘Phi-3-mini-4k-instruct-q4.gguf’

When setting up, I used the “Phi 2” setting template and then wrote the recommended prompt. On my iPhone 14 Pro I’m getting around 2-5 token per second.

Sometimes the <|end|> tag isn’t handle correctly, and it just skips over it and starts a new answer

[savkinavmono]

Make sure Metal=on, BOS=on, EOS=off. And try setting contextsize=1024. I got 8-9 Tok/sec.

Officially phi3 is only supported starting with llama.cpp release b2717. The latest LLMFarm commit uses b2692. The Testflight version uses b2135 which officially supports only phi2.