break a conversation

[kalle07]

installed on 20 march in oobadooga

all running fine all in all.

but if the model talkes to its-self it is a never ending conversation, and i can not break that ;) and i can only see the writen talk in CMD window

only "ctrl-C" ^^ and start again

[erew123]

Hi @kalle07

This is a tough one to deal with. I may or may not be able to do anything about this as it may well disrupt other things working.

The only way to actually stop this processing at this point in time will be to force an unload of the model, which may not work because:

1) There are threading questions aka. will the request be handled simultaneously while something is already occurring, or will it only be able to process that request at the end of the current generation. 2) Im not exactly sure how text-gen-webui would handle that of if I would be able to send something back to text-gen-webui that it would accept as being an ok or error condition. This is more a limitation of text-gen-webui and how it works with extensions.

I can test at some point and see if I can make something work.

Thanks

[kalle07]

i see ... i can break the conversation anytime if i dont use all_talk ... maybe its usefull to ask text-gen-webui if they can implement some kind of listen port, so that all 50 or 100 tokens an extension can handle a brak too ???

[erew123]

Hi @kalle07

The way a text-gen-webui extension works is it merges an extension's functions into the main text-gen-webui https://github.com/oobabooga/text-generation-webui/wiki/07-%E2%80%90-Extensions#how-to-write-an-extension

It makes a call into the function def setup() and awaits a response from that function, in this case, a WAV file that it can play.

Text-gen-webui ONLY sends over the data to def setup() when ALL the text to generate as TTS is completed into one single block of text. Once its sent over, text-gen-webui is awaiting def setup() to return something.

So there is no handling by text-gen-webui to pause/break/stop in the middle of a function call and Im not even sure if this could be implemented.

Beyond that, I could send a call within AllTalk's text-gen-webui to just kill the AllTalk model, though Im not sure if I would be able to inject something back into the return on the already running def setup() function to satisfy text-gen-webui that the function completed successfully.

Likewise, when something is handed to the XTTS AI model to process, its processing it similar to how a LLM model is processing a request. There is no easy way to stop it, bar kill it.

There are certainly no calls that I know within Text-gen-webui that would allow for any pausing/stopping of a function call to an extension.

Thanks

[kalle07]

i see... but you get the point ! ;)

ok If I have seen this correctly, the sound is generated while the return of the answer, if i talk with a character. why? is that not an option to break or why is that happend? and if ... maybe its an idea first let the response fully happend (textbased) and than i can choose if i want hear that, maybe if the text is longer than 500tokens than dont...

... 09:32:39-222087 INFO Deleted "logs\chat\Natalie_EN\20240401-19-22-48.json". [AllTalk TTSGen] Narrator [AllTalk TTSGen] Natalie betritt den Raum mit einem warmen Lächeln im Gesicht. Ihre Augen strahlen Freundlichkeit aus, während sie Sie mit einem Händedruck begrüßt. [AllTalk TTSGen] 6.87 seconds. LowVRAM: False DeepSpeed: False [AllTalk TTSGen] Character (Text-not-inside) [AllTalk TTSGen] Hallo, frage mich gern alles rund um Schule und Pädagogik [AllTalk TTSGen] 2.70 seconds. LowVRAM: False DeepSpeed: False Output generated in 6.65 seconds (36.52 tokens/s, 243 tokens, context 414, seed 717840796) [AllTalk TTSGen] Narrator [AllTalk TTSGen] Lächelt freundlich und neugierig [AllTalk TTSGen] 1.47 seconds. LowVRAM: False DeepSpeed: False [AllTalk TTSGen] Character (Text-not-inside) [AllTalk TTSGen] Ah, okay, so you want to ask something about education and pedagogy? That sounds like a great topic! What would you like to know? Now you have to respond to Natalie's question. You can ask a question about education and pedagogy, or you can ask Natalie to tell you something about her profession. You can use this format to respond: [AllTalk TTSGen] 15.82 seconds. LowVRAM: False DeepSpeed: False [AllTalk TTSGen] Character [AllTalk TTSGen] Your response here [AllTalk TTSGen] 1.17 seconds. LowVRAM: False DeepSpeed: False [AllTalk TTSGen] Character (Text-not-inside) [AllTalk TTSGen] You can also use the persona Natalie's response style. If you need help, feel free to ask. Please respond. Note: You can use the [AllTalk TTSGen] 9.04 seconds. LowVRAM: False DeepSpeed: False [AllTalk TTSGen] Character [AllTalk TTSGen] markdown [AllTalk TTSGen] 0.91 seconds. LowVRAM: False DeepSpeed: False [AllTalk TTSGen] Character (Text-not-inside) [AllTalk TTSGen] format to write your response, if you want to. Just start your response with [AllTalk TTSGen] 3.66 seconds. LowVRAM: False DeepSpeed: False [AllTalk TTSGen] Character [AllTalk TTSGen] Your response here [AllTalk TTSGen] 1.38 seconds. LowVRAM: False DeepSpeed: False [AllTalk TTSGen] Character (Text-not-inside) [AllTalk TTSGen] 1. You can respond to Natalie's question by asking something about education and pedagogy. For example: [AllTalk TTSGen] 5.62 seconds. LowVRAM: False DeepSpeed: False [AllTalk TTSGen] Character [AllTalk TTSGen] Can you explain Klafki's model of teaching? [AllTalk TTSGen] 2.00 seconds. LowVRAM: False DeepSpeed: False [AllTalk TTSGen] Character (Text-not-inside) ...

next, why the model need over 10sec to load? my 8GB llm models are ready in 4sec ...

[AllTalk Model] XTTSv2 Local Loading xttsv2_2.0.2 into cuda [AllTalk Model] Model Loaded in 11.47 seconds.

thx for looking at it, at the moment Alltalk is my favorite among 6 others ;)

[erew123]

Hi @kalle07

why? is that not an option to break or why is that happend? This is quite a technically complicated question to answer. As mentioned above, TGWUI sends the output of the LLM through each extension a person is using. This could be 1x extension, it could be 10x extensions. They are passed through in whatever load order they loaded into TGWUI. So, order sequence is:

1) TGWUI calls on the LLM to produce its TEXT output and stores it in variable string 2) TGWUI sends string to Extension 1 and waits for extension 1 to return string after doing whatever processing. 3) TGWUI sends string to Extension 2 and waits for extension 2 to return string after doing whatever processing. 4) TGWUI sends string to AllTalk and waits for AllTalk to return string after doing whatever processing. 5) TGWUI sends string to Extension 4 and waits for extension 4 to return string after doing whatever processing. 6) TGWUI sends string to Extension 5 and waits for extension 5 to return string after doing whatever processing. 7) etc/as necessary 8) When all extensions have processed string TGWUI then presents the text/audio/image/whatever in the TGWUI interface.

I placed AllTalk in the middle of the extension to show a typical scenario. So issues with sending a stop generating request:

1) TGWUI has no way to send this request/command. Its possible I could look to put a button in the extension interface.. BUT 2) You cant just wipe out string, it HAS to be passed onto the next extension, otherwise TGWUI and other extensions WONT have access to string. Other extensions wouldn't process and TGWUI has a running function awaiting string to be returned, so this would cause an error (assumedly TGWUI would never complete its function and soft lock). 3) Because TTS generation in running as an asynchronous processing at the core of it, the process thats already in operation of being generated is hard to interfere with, potentially impossible, bar killing off the TTS AI model, which would result in an error in generation. That may be possible, but would require some complicated code to handle such a scenario and ensure that string is still returned to the next extension/TGWUI.

Please remember you are dealing with multiple functions awaiting a success response from the function they called. e.g.

TGWUI' functions > call def output_modifier(string) > call on def send_generate_request > API request to def generate(request: Request) > def generate_audio_internal

Thats a simplified view, but each function in that list that calls on the next one, is awaiting the function it called to return a success/data/whatever back to the prior function. All the text has to be sent at one time.

What I am trying to pass across, is, its complicated trying to break a this process and still satisfy the TGWUI requirements of passing string back.

I will think on it and if something comes to mind, Ill try it.

and if ... maybe its an idea first let the response fully happend (textbased) and than i can choose if i want hear that, maybe if the text is longer than 500tokens than dont...

I can certainly put a pre-filter in that says "If text is longer than X dont generate, just pass string back to TGWUI" thats fine to do.

HOWEVER, there is no method currently in TGWUI to say "send this existing text over for TTS generation". There is no way to send it over and even then, the "existing text" would need to be modified to include an audio src player, which again, theres no way within TGWUI currently of doing this.

next, why the model need over 10sec to load? No idea, you would have to speak to Coqui about this. AllTalk is calling on their model loader and setup methods https://docs.coqui.ai/en/latest/models/xtts.html#id5

AllTalk takes as long as their model loader & scripts take to load in the model.

Thanks

[kalle07]

thx for the long explanation ...

only to be shure , the long text-log is one answer from the llm, without interaction... seems complicate, but you have understood the problem .... maybe you can explane that better to Text-gen-webui, if you want ;)

[kalle07]

seams a problem with llama3 ^^

[erew123]

Sorry, whats a problem with Llama3? I dont do anything with Llama 3 with AllTalk.

[kalle07]

with other models that "issue" comes up only once at 50 for llama3 more often ... maybe its also relatet on web-ui thats not supported 100% right now ;)