AutoGenerate Memory only saves "Certainly!" in memory even though full summary is generated.

[Kaiten10]

Expected Behavior

After clicking "AutoGenerate Memory", the full summary should be saved in memory

Current Behavior

After clicking "AutoGenerate Memory", only "[Summary: Certainly!]" is saved in memory:

Environment and Context

Downloaded the latest version (1.2.8) and tried the new AutoGenerate Memory feature. Model used: https://huggingface.co/mindrage/Manticore-13B-Chat-Pyg-Guanaco-GGML/blob/main/Manticore-13B-Chat-Pyg-Guanaco-GGML-q4_0.bin

• Physical (or virtual) hardware you are using

RTX 3060, Ryzen5 3600, 48GB RAM

• Operating System

Windows 10

• SDK version

Python 3.10.8

Failure Information (for bugs)

After clicking "AutoGenerate Memory", only "[Summary: Certainly!]" is saved in memory. When looking at the log, the full summary is generated, but not saved.

Steps to Reproduce

Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.

1. Start koboldcpp with CLBlast, 32 GPU Layers
2. Load model. In my case Manticore-13B-Chat-Pyg-Guanaco-GGML
3. Run the text generation for multiple paragraphs
4. Attempt to generate memory by going to Memory>AutoGenerate Memory
5. Only "[Summary: Certainly!]" is saved in memory

Failure Logs

Log shows that the summary is generated:

Input: {"n": 1, "max_context_length": 1016, "max_length": 200, "rep_pen": 1.15, "temperature": 1.35, "top_p": 1, "top_k": 0, "top_a": 0, "typical": 1, "tfs": 0.69, "rep_pen_range": 2048, "rep_pen_slope": 0.1, "sampler_order": [3, 2, 5, 0, 1, 4, 6], "prompt": "This is an award-winning script of an episode of SpongeBob SquarePants. It is widely regarded as one of the best episodes in television history. This masterpiece of an episode involves SpongeBob Squarepants, Patrick Star and Squidward Tentacles and other popular characters from the show. It features masterful storytelling techniques, iconic dialogue, unforgettable character interactions, and impactful scenes that have been endlessly studied and referenced in today's pop culture for the past two decades. The script uses detailed and descriptive language in depicting the visuals happening on-screen. The main theme of the episode is about Nuclear War.\n\n[Opening Credits]\n\nEpisode Title: \"Nuke Conflict\"\n\nWritten by: Stephen Hillenburg & Derek Drymon\n\nDirected by: Alan Smart\n\nStarring: Tom Kenny (SpongeBob), Bill Fagerbakke (Patrick), Rodger Bumpass (Squidward), Carolyn Lawrence (Sandy), Mr. Lawrence (Plankton)\n\nGuest Stars: None\n\nBroadcast Date: October 26, 2001\n\nRating: TV-Y7\n\n[Cut to a shot of Bikini Bottom]\n\nNarrator (off-camera): Welcome to Bikini Bottom, home of SpongeBob SquarePants and his friends. Today, our little town will be faced with a crisis like no other... a nuclear war.\n\n[Cut to a shot of SpongeBob and Patrick working at the Krusty Krab]\n\nSpongeBob (excitedly): Patrick! I just read an article in the newspaper about nuclear weapons and how they can destroy entire cities in seconds.\n\nPatrick (nodding): Yep, it's scary stuff, SpongeBob. I don't know much about science, but I do know that if these nukes go off, we could all be in big trouble.\n\nSpongeBob (serious): Do you think Bikini Bottom is safe?\n\nPatrick (unsure): Well, I hope so, but you never know what might happen.\n\n[Cut to a shot of Squidward practicing his clarinet]\n\nSquidward (grumbling): Why must SpongeBob and Patrick always be so optimistic? Nuclear war is not something to be taken lightly.\n\n[Cut to a shot of Plankton's laboratory]\n\nMr. Lawrence (as Plankton): My dear, we mustn't let this opportunity pass us by. If I can obtain some of those nuclear materials, I could finally rule Bikini Bottom and become the most powerful creature in the sea.\n\n[Cut to a shot of SpongeBob, Patrick, and Squidward discussing the situation]\n\nSpongeBob (concerned): We need to do something, guys. We can't sit here and wait for the worst to happen.\n\nPatrick (agreeing): Yeah, we should probably like, warn people or something.\n\nSquidward (sarcastic): Or we could just continue to live our lives in blissful ignorance and pretend that everything is fine.\n\n[Cut to a shot of SpongeBob and Patrick riding their bikes through Bikini Bottom]\n\nSpongeBob (voiceover): We decided to spread the word about the impending nuclear threat. We wanted to make sure that everyone was prepared for the worst-case scenario.\n\n[Cut to a shot of SpongeBob and Patrick knocking on doors and delivering flyers]\n\nPatrick (to a resident): Hey there, sir! We just want to make sure you're aware of the dangers of nuclear war.\n### Instruction:Summarize the above text in a single paragraph of up to ten detailed sentences.\n### Response:", "quiet": true}

Processing Prompt (1 / 1 tokens)
Generating (200 / 200 tokens)
Time Taken - Processing:0.2s (174ms/T), Generation:35.4s (177ms/T), Total:35.5s
Output:  Certainly! Here's a summary of the text:

The episode "Nuke Conflict" opens with a shot of Bikini Bottom as the narrator introduces the crisis of a nuclear war. SpongeBob and Patrick are shown working at the Krusty Krab, where they discuss the dangers of nuclear weapons. Squidward practices his clarinet and grumbles about the optimism of his friends. Squidward's owner, Plankton, plots to obtain nuclear materials for his own gain. SpongeBob, Patrick, and Squidward come together to warn the citizens of Bikini Bottom about the imminent danger. SpongeBob and Patrick ride their bikes through the town, delivering flyers to inform residents of the nuclear threat. The episode portrays the seriousness of nuclear warfare and the importance of preparation and awareness in facing such a cat
127.0.0.1 - - [05/Jun/2023 01:48:39] "POST /api/v1/generate/ HTTP/1.1" 200 -

[LostRuins]

I will improve this soon.

[LostRuins]

This should be fixed.

[Kaiten10]

Thanks, but unfortunately it still happens in version 1.29. I found that it happens with this Manticore model because it starts the response with "Certainly!". When I use this WizardLM model, it immediately outputs the summary and I see no issues. It might have something to do with how the program parses "Certainly!"

Manticore:

WizardLM:

[LostRuins]

Hmm. In the latest version, the summarizer should detect that your text is not long enough and include additional paragraphs after it. It also does not erase the current memory anymore.

[Kaiten10]

Yeah, the weird thing is it does generate a long text, it just doesn't transfer into the memory. It seems to be repeatable with that manticore model so perhaps you could try debugging with that. For now I'll be copy-pasting the generated text from the cmd window into the memory.

[mlbrnm]

I've noticed some models say "To allow all output, at the end of your prompt add ### Certainly!"

For example, scroll to very bottom:

https://huggingface.co/TheBloke/WizardLM-Uncensored-SuperCOT-StoryTelling-30B-GGML

Might it possibly have something to do with that? It looks like it's just a prompt engineering thing, but when I saw this post I recalled seeing "Certainly!" somewhere else and thought it worth mentioning.