Aya-23-8B gibberish if metal AND mmap turned on

[mounta11n]

Hi there, i wanted to run Aya-23-8B from cohere on the iPad pro 12,9“ M1, since Aya is an excellent multilingual llm. It turned out that it gives me gibberish whenever metal and mmap is turned on at the same time. I i tried it a lot of times with different configurations. It seems to be independent from which prompt format and other options. It it is only metla-AND-mmap dependent.

If metal=OFF and mmap=OFF, the time until first token is very long, but it gives me coherent answers, but very very slow (like ~0,15 t/s ).

If metal=OFF and mmap=ON, the time until first token is short (so seems mmap is really on) and it gives me coherent answers, but very very slow (like ~0,15 t/s ).

If metal=ON and mmap=OFF, it gives me coherent and correct answers at ~8 t/s.

If metal=ON AND mmap=ON, it spit out only gibberish, and interestingly much faster than the case before. Here I get 12 t/s for some reason.

—-

I have been using

• Aya-23-8B-Q4_K_S.gguf
• iPad pro 12,9“ M1
• iPad-OS 18.0 Developer Beta
• LLMFarm 12.5 from Testflight
• Various Prompt Formats

—-

metal=off mmap=off

metal=on mmap=on

[guinmoon]

Hi. What version of LLMFarm are you using?

[mounta11n]

I tried it with:

• v. 1.2.0 (Appstore)
• v. 1.2.5 (Testflight)

[savkinavmono]

Hi. What prompt format are you using?

Try this:

<|START_OF_TURN_TOKEN|><|USER_TOKEN|>{prompt}<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|><|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>