Closed Alihkhawaher closed 2 months ago
Is this version 1.62.2 ? Have you tried 1.61.2 ?
I played with mixtrals up to 64k several times without any problems. …Oh wait, you said mistral, not mixtral. Hm-m, is it a RoPE issue? Anyway, try the older version!
I see you use mmproj. Will the problem persist without it?
Thanks, I tried with both versions I got the same issue, also even LM Studio stopped working, so I am not sure how it worked previously. The issue appears mostly with MoE, the models I am using are 2x7b. I suspected my P40 may have an issue and I tested VRAM using OCCT and I have no errors. I will update you if I have more information.
removing mmproj does not change the result
If LM studio is also not working, then that seems to point to the model being the culprit.
Try changing your RoPE scaling settings, how about --ropeconfig 1 32000
, and see if that works for you. Or try --ropeconfig 0.5 10000
Great, I tried your custom ropeconfig, 1 32000 and I was able to go beyond 8k, I reached 11K without an issue.
I needed to fix it because macadeliccc/laser-dolphin-mixtral-2x7b-dpo-GGUF has been the smartest 7b model I tried.
Thanks,
Dears,
I tried a few mistral models with context 32k, but when I go over 8k koboldcpp started returning gibberish, at the start I thought it was the issue with the model then I tired LM Studio and I easily reach 11K without the same issue.
I tried disabling/enabling mmq and contextshift but the issue is still the same.
maybe the following could help