search

chrisrude / oobabot

A Discord bot which talks to Large Language Model AIs running on oobabooga's text-generation-webui
MIT License
101 stars 34 forks source link

when using streamed responses, message history is lost #63

Closed chrisrude closed 1 year ago

chrisrude commented 1 year ago 
          > Yup, working on it soon! It's been a minute since the last release, and I have a lot of improvements that need to get out. :)

I'm not planning to give the speed limit itself a UI widget, but on release you'll be able to change it via the advanced yaml editor.

I'm still testing out your latest 0.2.0 changes, and I'm finding that using stream-responses: True still causes the messages to be omitted from chat history. 1688757366_07_07

Originally posted by @jmoney7823956789378 in https://github.com/chrisrude/oobabot/issues/50#issuecomment-1625936170

chrisrude commented 1 year ago

@jmoney7823956789378 the fix above should work. Let me know if you're still seeing it!

I really should have included this in 0.2.0, but now that that is out it's much easier to do smaller, more frequent updates.

jmoney7823956789378 commented 1 year ago

@jmoney7823956789378 the fix above should work. Let me know if you're still seeing it!

I really should have included this in 0.2.0, but now that that is out it's much easier to do smaller, more frequent updates.

Works great, you're the best!

jmoney7823956789378 commented 1 year ago

On second thought we may need an even slower stream speed, given that I have noticed that the occasional "User says" at the end of a block of a bot's message. This is probably because it isnt able to detect the stopping strings BEFORE streaming half of it out.

chrisrude commented 1 year ago

Yeah, I noticed the same... another approach, rather than slowing down the stream speed, is to delay posting strings who are match with "User says" (Use... User... User sa... etc.) until we have enough information to either post it or abort.

chrisrude commented 1 year ago

Ok, not quite what I suggested, but I put in a change... now if we notice an "immersion-breaking line", we will remove any previously posted parts of those lines before finishing the response.

So, the new experience will be that it will post the desired response, then update with "User ..." when those tokens arrive, but then once it's clear the line should have been filtered, it will remove the parts it shouldn't have posted.

Not super clean but cleaner than what was there.

jmoney7823956789378 commented 1 year ago

Ok, not quite what I suggested, but I put in a change... now if we notice an "immersion-breaking line", we will remove any previously posted parts of those lines before finishing the response.

So, the new experience will be that it will post the desired response, then update with "User ..." when those tokens arrive, but then once it's clear the line should have been filtered, it will remove the parts it shouldn't have posted.

Not super clean but cleaner than what was there.

I pulled and gave it some tests, but it seems like it isn't quite consistent enough. Sometimes I'll get a response that ends in "User sa-" and then it will get edited off the end. Other times it will continue the "User says:" into the full dialogue of the user. From the logs, it seems the former will only happen if the received message is presented on a line by itself (surrounded by /n or otherwise?).