No Discussions Tab (Also we're talking about persisting data)

[afewvowels]

Hi I have things I'd like to talk about and discuss (functionality like persisting memories across chats, database integrations, api calls etc.) that aren't issues with the existing codebase.

Thanks!

[Spencer-Dawson]

It would be nice to have a discord for that. Just my two cents.

Re: chat persistence

From a web server arch perspective the options are basically server side storage or local storage. Any discussion about implementation would probably hang on deciding that detail first. Normally this decision is pretty straightforward for web developers because local storage scales better, but server storage is generally more convenient for the end user and SaaS companies although it typically requires logging in. I don't think those norms necessarily apply here however because this is ostensibly a self hosted application and not necessarily intended to scale for thousands of users across dozens of servers.

The default option for local storage is browser cookies. These are convenient from the perspective of refreshing the page to still see your settings loaded, but inconvenient in that if you accidentally clear your session data your data is lost. There are size limitations to storing cookies, but I don't think that's a big issue for this use-case.

The other local storage option is exporting session data to a file at the users request and then loading that file later to another browser session at a later time. I suspect an export chat/settings/character option would be viewed as desirable regardless of other implementation details.

The obvious main benefit of local storage from a server arch perspective is that it doesn't matter what server you send your requests to. Ten thousand users could hit 10 servers at the same time and all the data needed to compute the individual responses are included in the calls. IE no looking up data on a server in the cloud anywhere. It's also a less invasive approach for the user in that the server only ostensibly remembers what you send it for as long as you send it (except for logs)

For a professional SaaS model website server side storage is usually preferred. It’s generally implemented with a combination of databases saving files in block storage on a storage service like Amazon S3. Generally this is typically tied to a user account id which is tied to your login. This is the approach CharacterAI and ChatGPT have taken. It's a huge pain to implement, but appropriate for SaaS in that it provides user convenience and server monitoring(and potentially metering).

For a self-hosted service NOT designed to scale across multiple servers the distinction between local and server storage can get a little muddled. stable-diffusion-ui for example is guilty of saving files straight to the server's runtime folder. It then reads the data saved locally when it loads up again. That makes it impossible to scale multiple server instances to meet the needs of multiple users at the same time. Ie it is really a one server per user solution and worse the way it’s designed doesn’t separate the servers own code from the users all that well. From an application perspective this is permissible, but it is not ideal.

ATM I think text-generation-webui stores some session state in ram in the python server code and some in the browser which it then feeds to the python server when generating text etc. IMO the obvious way to go for this project is probably local storage, but that decision is really up to the person doing the work(IE oobabooga presumably).

I think once the decision on how to handle persistence is resolved most other major decisions on features will be easier to prioritize and implement.

[afewvowels]

Wow that's a lot bigger in scale/scope than I was thinking!

I guess from my perspective I'm using these webui's for generative ai models from a personal use case but I would like to be able to have functionality like bigger apps that have scaled possess. In my case i'm thinking about adding likes/dislikes/reactions to chat messages and eventually fine tuning the model based on my personal message history. To do that I would probably like to run a local instance of a database wheter that's mongo or postgresql or mariadb (I don't have a lot experience with understanding different database relational models and which would work best for storing chat history). Also long term storage for adding memories to spice up the context would be nice to have as well.

I found the experience of Automatic1111's webui to be very easy to use and I've found this repo to be very easy to get up and running. I don't know what's going on with my IDE but I'm trying to push an update to the UI on my fork (and hopefully get it approved) to put all the items after the input box into an accordion and hide the footer with some CSS for the --cai-chat option to make the interface mobile friendly but the push is hanging right now.

[oobabooga]

@afewvowels Issues can also have the categories "general question" and "feature request". Is that not enough for more open-ended discussions? Feel free to create new issues with your ideas, preferably one issue per topic.

I am glad that you found the installation of this web UI easy. Easy installation and good defaults are two of its main goals.

[Spencer-Dawson]

So re likes, dislikes and reactions. I think a misunderstanding could be in play. I don't think the language models this ui can use would be improved by adding likes, dislikes and reactions to this ui at this time. The models basically work by feeding character information and chat history in for each generation then ask the model to generate new text that naturally follows what was previously generated(it's a glorified autocomplete technique).

CharacterAI presumably uses likes, dislikes, etc as feedback for model fine tuning, but atm this ui doesn't do any model training. Ergo adding that functionality now would probably just confuse the user. If model training was added though it could be useful.

It certainly would be cool to have character specific fine tuning as a feature. Personally I'd like to see "characters" as their own trained extensions of specific language models instead of json configs with example text. My thought is you could save an indefinitely long character log as a training set and feed it through to generate some kind of pruned model component that ideally is just a diff from the base model to save memory. That idea is more of a data science / model developer concern then a ui one though. In summary I think this kind an ideal 'character' would be character description + chat log + some kind of language model extension trained on character specific chat logs ideally with likes, dislikes etc integrated into how the model is trained

[oobabooga]

@Spencer-Dawson a possibility to do what you are describing is to use softprompts. See here: https://rentry.org/pygsoft

The web UI now supports them (there is a default menu to upload the .zip files).

[SwannSchilling]

Thumbs up for a Discord server! But also keeping it organized over here on Github is fine atm... Looking forward to see where this is all heading, a place for open discussion would be a good thing though, once more people are joining in.

[oobabooga]

Well, why not, I have enabled Discussions for the repository.

The chat history has been made persistent - every time a new message is generated, the chat is exported to logs/character_name_persistent.json. This file is automatically imported when the character is loaded. https://github.com/oobabooga/text-generation-webui/commit/b397bea387c9a7a4296dfad57f3636a74d3bc5ae

Interface persistence across browser sessions is an open problem, but there is an open issue for this.

I will close this issue. Feel free to create new ones about each new suggestion, or to move this discussion to the Discussions.