Auto submit of text after audio input

[AravindKaay]

Feature suggestion: I wish there were a prop that enables auto-submit of text based on audio input. It won't be very pleasant for the user to say a word to submit every time. I mean it should auto-submit after some seconds of inactivity from the user

[bharathmorg]

I am also facing this issue, could you please provide an solution for this issue or is there any alternate method

[OvidijusParsiunas]

Hi, submitting a message on a pause is difficult to standardise due to dynamic human speech intensions. For example, if the user speaks and later decides to think for a moment, automatically sending a message to the server would be the last thing that they would want. Hence, this is a relatively volatile problem depending on your chat audience.

I will try to see if I can implement something to help you out, but it may take some time depending on what needs to be changed. I'll let you know more details later tonight when I get some free time. Thanks!

[OvidijusParsiunas]

Hi, I did some experimentation and was able to create automated submit on silence functionality. I will create an API for this and deploy it in dev tomorrow so you can test it out.

One small issue I ran into was that whilst this works - the functionality to keep the recording going after submission is quite difficult. It does work well for the submit command because the [Speech To Element]() library that is used has been preconfigured to automatically restart after after a particular phrase has been spoken, but I can't do that on silence.

Hence, right now - stopping the recording after submit has happened works perfectly, and if you want to keep the recording going after submit - there is a 300 millisecond interim where the microphone is not available and then the recording automatically restarts again. Let me know if this fits your use-case (or if the first option is enough for you).

I also noticed a bug where upon recording the speech - the submit button is not clickable, I will also be fixing that. Thanks!

[AravindKaay]

I want the microphone to be on continuously after submission and that 300 milliseconds time gap is not a big deal for me .

[marioseixas]

Hi, I did some experimentation and was able to create automated submit on silence functionality. I will create an API for this and deploy it in dev tomorrow so you can test it out.

One small issue I ran into was that whilst this works - the functionality to keep the recording going after submission is quite difficult. It does work well for the submit command because the Speech To Element library that is used has been preconfigured to automatically restart after after a particular phrase has been spoken, but I can't do that on silence.

Hence, right now - stopping the recording after submit has happened works perfectly, and if you want to keep the recording going after submit - there is a 300 millisecond interim where the microphone is not available and then the recording automatically restarts again. Let me know if this fits your use-case (or if the first option is enough for you).

I also noticed a bug where upon recording the speech - the submit button is not clickable, I will also be fixing that. Thanks!

very nice, gonna use this also too, thanks!

the 300 ms is not a big deal for me either

[OvidijusParsiunas]

Hi, I have made the changes and updated our deep-chat-dev and deep-chat-react-dev packages version 9.0.137 to have all the latest functionality.

The API in this comment has been updated in the comments below.

To note, if you are using the default webSpeechAPI for the speech to text service, it automatically stops after around 5-7 seconds of silence (by browser), therefore if you are defining a custom ms value - please be aware of that.

Final thing to mention. The dev packages behave the same way as the original ones - except their names are different. The next release for Deep Chat is going to be a big one as we are migrating a lot of its logic to another web component called active chat and with that we are changing quite a few property names, such as changing the request property to connect and initialMessages to history. The old properties will still keep working as usual except you will see warnings in the console for their deprecation. Most importantly, because this is going to be a very big release - it is likely that it will take place some time next month, so all the submitAfterSilence will have to remain in the dev packages until it happens. Hopefully this is ok for you folks.

Let me know if you have any questions or issues. Thanks!

[OvidijusParsiunas]

The bug where the submit button is not enabled after speech transcription takes place has now been fixed in deep-chat-dev and deep-chat-react-dev version 9.0.138.

[AravindKaay]

I found a bug like the microphone button icon color is changed to default after a response from service but after that also I can speak and my voice is transcribed to text. I need to persist the icon color even after response from server

[OvidijusParsiunas]

Hey @AravindKaay. Thankyou for reporting the bug. I had another look into the silence functionality and have updated the overall Speech To Text API. The speechToText interface now looks like this:

...
stopAfterSubmit?: false;
submitAfterSilence?: true | number;

The stopAfterSubmit property is used to toggle whether the recording stops after the message has been submitted. The submitAfterSilence property is used to toggle the automatic submit after silence functionality. You can either set it to true or use a number to represent the number of milliseconds of silence required before submit.

The defaults for these values are:

...
stopAfterSubmit: true,
submitAfterSilence: false

If submitAfterSilence is set to true, then the default number of milliseconds is 2000 (2s).

Example usage (Vanilla JavaScript):

speechToText = {submitAfterSilence: true}

speechToText = {submitAfterSilence: 3000}

speechToText = {stopAfterSubmit: false}

speechToText = {submitAfterSilence: true, stopAfterSubmit: false}

speechToText = {submitAfterSilence: 3000, stopAfterSubmit: false}

This has been published in deep-chat-dev and deep-chat-react-dev versions 9.0.142. Let me know if this works for you and fixes your problem. If the issue is still occurring could you share the Deep Chat configuration you are using.

Thankyou!

[AravindKaay]

I have used stopafterSubmit to restrict the microphone to listen to what the system is speaking but I have to start it again for the next conversation. Could you help me in solving this issue?