search

guinmoon / LLMFarm

llama and other large language models on iOS and MacOS offline using GGML library.
https://llmfarm.site
MIT License
1.05k stars 62 forks source link

Sampling stuck in greedy #42

Closed Maw1979 closed 2 months ago

Maw1979 commented 4 months ago

The sampling is reverted back to greedy if I try to change it. Device: iPhone 15 pro max

In greedy mode the app is unusable. The llm keeps inferencing.

guinmoon commented 4 months ago

Could you please send the chat configuration and a link to the model so I can reproduce the error?

Maw1979 commented 4 months ago

I have tried tiny llama 1.1b 4K_M and mistral 7b 0.1 4K_M. The problem is the configuration itself. No matter what I try the settings are not saved. Even when trying different templates it is reverted back to custom with the greedy setting.image

Maw1979 commented 4 months ago

I just tried the app on a iPhone 14 Pro Max. Although the template also reverts back to custom, the settings are applied.

The answers on the 15 pro max are completely random and incoherent. But this is probably because of the temperature settings?

guinmoon commented 4 months ago

I am thinking about your question, but so far I have not been able to reproduce this bug.

Maw1979 commented 4 months ago

I can get it to work now on the 15 pro. When I choose the llama2 7b iPhone 12 template and don't change anything, it works. The sampling setting stays on temperature. However, the template setting reverts back to custom.

guinmoon commented 3 months ago

Can you tell me if this problem still occurs in the new version?