search

LostRuins / koboldcpp

Run GGUF models easily with a KoboldAI UI. One File. Zero Install.
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
4.86k stars 342 forks source link

Halucinations after certain questions #1097

Open Amestern opened 3 weeks ago

Amestern commented 3 weeks ago

With the new install of Koboldcpp I find the models struggle with narration. I am using previous models, no change there and the latest release of Koboldcpp. When I type something like "continue" normally the model would keep on going with coherent naration. I've noticed now the model replies with incoherent word salad. My computer hasn't changed, I run the models a little hot, I enjoy the randomness. I bet you want my settings although nothing has changed but the cpp version. I will amend this post with my settings.

My settings: Found 1 Vulkan devices: Vulkan0: Radeon (TM) RX 480 Graphics (AMD proprietary driver) | uma: 0 | fp16: 0 | warp size: 64 llm_load_tensors: ggml ctx size = 0.32 MiB llm_load_tensors: offloading 32 repeating layers to GPU llm_load_tensors: offloading non-repeating layers to GPU llm_load_tensors: offloaded 33/33 layers to GPU llm_load_tensors: Radeon (TM) RX 480 Graphics buffer size = 3577.56 MiB llm_load_tensors: CPU buffer size = 70.31 MiB

I have used the same model and settings for many months now. Tempreture .83 Max tokens 1424 Amount to gen 220 rep penalty 1.1 Top sampling .5

EDIT: An example: ME: Continue Solar: Kult, Words of type of service for Car, there are definitely cares or 23. ME: Sorry? Solar: Blue Bok, Big Humb Chit, Care: Green Worth: Wide Out, Kara.

And then I rebooted

ibrainventures commented 3 weeks ago

I got the same issue / feelings about 1.73.1 .. But i also changed the last days to much to declare the Kobold as the culprit.

What was your last "less hallucinations" KoboldCPP Version Number? May i can re-check and stepping through the commits for investigation.

Amestern commented 3 weeks ago

I think it was version 1.63 as my last non-hallucinating version. I kept many versions of Koboldcpp, but unfortunately cleaned up my folder recently. One thing I noticed is I left it running overnight and picked up a conversation in the morning and it fell apart, I had to reboot. I find the hallucinations fascinating by iunteruptive. A.

On Thu, Aug 29, 2024 at 9:05 AM ibrainventures @.***> wrote:

I got the same issue / feelings about 1.37.1 .. But i also changed the last days to much to declare the Kobold as the culprit.

What was your last "less hallucinations" KoboldCPP Version Number? May i can re-check and stepping through the commits for investigation.

— Reply to this email directly, view it on GitHub https://github.com/LostRuins/koboldcpp/issues/1097#issuecomment-2317605253, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQ4QYFFKAISKMWUWJY3YPSLZT4MCDAVCNFSM6AAAAABNI24SPSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJXGYYDKMRVGM . You are receiving this because you authored the thread.Message ID: @.***>

-- Andrew Mestern @. @. 519-274-0477