search

Vali-98 / ChatterUI

Simple frontend for LLMs built in react-native.
GNU Affero General Public License v3.0
521 stars 28 forks source link

Context shifting problem? #104

Open inspir3dArt opened 4 days ago

inspir3dArt commented 4 days ago

I had a longer roleplay chat using 0.8.1 with a Q4 k_m gguf quant of L3-8B-Lunar-Stheno. It worked well, until the #42 message, there it took 19 minutes before it replied, like it had to reprocess the entire chat. The models context size is set to 8192.

The log doesn't show anything different than on all other replies, except the big time jump.

The device is a Samsung Galaxy S24 ultra.

Vali-98 commented 4 days ago

A few questions:

  1. Was this working in 0.8.0?
  2. Is Bypass Context Length in settings disabled?
  3. What fields are enabled in the chat? (eg. Examples, Scenario, Personality etc)
inspir3dArt commented 4 days ago

  1. I haven't tested 0.8.0, but it was a problem back in v0.7.9g too, with is the last version I think I tested before.

  2. Yes, it is disabled. The model showed the context length correctly as 8192 in the models overview, but was at a lower setting in the model settings by default, so I changed it to 8192 there too.

  3. The following fields are used / Enabled by the character card:

    • Name
    • Description
    • First message
    • Scenario
    • Example messages

The model card has also entries in the "System Prompt" and "Jailbreak (Post history instructions)" that seems to be not used. I copied the System Prompt text manually to the beginning of the Description field inside the character card edit menu in chatter ui before starting the roleplay.