Issue with Voice utterance when using Cloudformation Deployment

[klotniaa]

Hi,

I have deployed the lex-web-ui for a healthcare specific use case at this web address: https://d1bpmlzuocay9s.cloudfront.net/. Text communication and response works absolutely fine.

However, I am having an issue when I try to interact with the bot with my voice. The frontend ingests the voice input correctly. That voice input is then converted to text and sent to a lambda function I am using to interact with Bedrock. The lambda function processes the input properly, interacts with Bedrock, bedrock produces an answer in text format, and then the lambda function returns the text to Lex. I am able to effectively use voice to interact with the bot when testing in the Lex console. However, when I attempt to use the same process to interact with the bot from my deployed CloudFront lexwebui frontend, I get "Sorry, I had an error handling this conversation." I have tested this process in both Chrome and Firefox and get the same error.

Any guidance or thoughts are appreciated, thank you!

[bobpskier]

@klotniaa First want to confirm that you are using Lex V2. For the voice use case, I believe the response from the Lambda function back to lex should return a message using a type of SSML for a voice response and the content should have tags as part of the response.

"message": { "contentType": "SSML", "content": "Message to convey to the user. For example, Thanks, your pizza has been ordered. " },

[klotniaa]

Hi, I am using Lex V2.

The response I am using to test is:

{
       'messages': [{
            'contentType': "SSML",
            'content': "<speak>What symptoms are you experiencing?</speak>" (message in SSML format with tags)
        }]
}

I built the response based on this documentation for Lex V2: https://docs.aws.amazon.com/lexv2/latest/dg/lambda-response-format.html.

[klotniaa]

I tried both with and without SSML tags since it seems like the presence of SSML tags doesn't matter. I spoke "I have back pain" into the Lex testing console and the lexwebui frontend. In the former case, I get a response that's the same as I get when I type "I have back pain". In the latter, I get the below error.

[klotniaa]

@bobpskier , appreciate your help so far. Is there any update on whether this issue is something that needs to fixed within the lex-web-ui codebase itself?

[selvaraj-xemplar]

Hello, I am also facing same issue, can anyone help me is this resolved or not yet ?

@klotniaa have you find any workaround for this?

I am getting error here,

lex-web-ui.min.js:29 converser error: TypeError: Cannot read properties of undefined (reading '0')

e.slotToElicit = t.dialogAction.slotToElicit) : (e.intentName = t.interpretations[0].intent.name,

My Lambda code

 return {
            "sessionState": {
                "dialogAction": {'slotElicitationStyle': 'Default','type': 'ElicitIntent'},
                "intent": {
                    'name':intent,
                    'slots': slots
                    },
                "sessionAttributes": event['sessionState']['sessionAttributes']
            },
            'messages': [{
                'contentType': "SSML",
                'content': "What symptoms are you experiencing?"
            }]
        }

[bobpskier]

@selvaraj-xemplar Usually this means there is an error in the backend. Perhaps a lambda handling fulfillment is throwing an exception or returning content that the lex service is not expecting. You can usually see the error detail using the browser debugger in the network tab as part of the response to the content request from the browser. Another place to check if using a lambda is the cloudwatch log from the function. Can you post a test URL that demonstrates the problem and if using lambda the response being returned by the function?

[selvaraj-xemplar]

@bobpskier Thanks for quick response, can you correct me on the lambda function, also check my errors screenshot

return { "sessionState": { "dialogAction": {'slotElicitationStyle': 'Default','type': 'ElicitIntent'}, "intent": { 'name':intent, 'slots': slots }, "sessionAttributes": event['sessionState']['sessionAttributes'] }, 'messages': [{ 'contentType': "SSML", 'content': "What symptoms are you experiencing?" }] }

[bobpskier]

@selvaraj-xemplar I didn't see the screenshot with errors in your post. I'll try and recreate the problem based on the example response.

[selvaraj-xemplar]

Thanks for looking into the issue,

Adding the URL with Lambda function

https://d3a0xuu4h1fpm5.cloudfront.net/

def lambda_handler(event, context):
    slots = event['sessionState']['intent']['slots']
    intent = event['sessionState']['intent']['name']

    if event['inputMode'] == 'Speech':
        print("Coming in the speech mode of the lambda", event)
        response = {
            "sessionState": {
                "dialogAction": {
                    'slotElicitationStyle': 'Default',
                    'type': 'ElicitIntent'
                },
                "sessionAttributes": event['sessionState']['sessionAttributes'],
                "intent": {
                    'name': intent,
                    'slots': slots
                }
            },
            "messages": [
                {
                    "contentType": "SSML",
                    "content": "<speak>This is Voice Output</speak>"
                }
            ]
        }
    else:
        response = {
            "sessionState": {
                "sessionAttributes": event['sessionState']['sessionAttributes'],
                "intent": {
                    'name': intent,
                    'slots': slots
                },
                "dialogAction": {
                    'slotElicitationStyle': 'Default',
                    'type': 'ElicitIntent'
                }
            },
            "messages": [
                {
                    "contentType": "PlainText",
                    "content": "Text Output from backend"
                }
            ]
        }

    return response

In the default chatbot LEX ui voice output is coming, but with the Vue-ui is not working

[bobpskier]

@selvaraj-xemplar I was able to duplicate the problem. I'm looking into a solution.

[selvaraj-xemplar]

@selvaraj-xemplar I was able to duplicate the problem. I'm looking into a solution.

Thanks @bobpskier

[bobpskier]

@selvaraj-xemplar The problem is in the lex-webui lex client implementation contained in lex-web-ui/src/lib/lex/client.js. For this type of response using ElicitIntent, the client is not handling the case where Lex has not identified any interpretations. The javascript code is throwing an exception when trying to access interpretations. I'm working on refactoring the code to handle this case. This also leads to another problem handling the automatic end of the conversation with an "All Done" response in voice mode. This needs to be cleaned up as well. I'll let you know when both fixes are ready for testing.

[selvaraj-xemplar]

@bobpskier thank you so much for looking into this issue as priority

[bobpskier]

@selvaraj-xemplar I've made an update which you can test. I've not submitted this as a pull request back to the AWS team yet as I'd like to confirm first that it does solve the problem. When using ElicitIntent please make sure to provide a response with instructions to the user in voice mode as well as text mode about what they should do next. When in voice mode, lex-web-ui will play this response and will automatically turn on the microphone waiting for input back from the user. The browser will wait for a period of time. If no voice input is received in this time, the "All done" response will be played and the current conversation will end until the user hits the microphone button again to start a new intent. I've placed the lex-web-ui build/templates that you can use for an update to your stack in cloudformation at "https://s3.amazonaws.com/tioth-dev-us-east-1-lex-web-ui/artifacts/templates/master.yaml". Please let me know if this solves the issue.

There are other less than desirable behaviors in the UI related to voice that I'm not that pleased with. A second is the play button that appears next to the user input message bubble allowing a user to play back what the system heard from them. This only works on Chrome. None of the other browsers support this playback mechanism. Need to either remove this button or see if it can work with the other browsers. Thoughts?

[selvaraj-xemplar]

sure @bobpskier I will test this and I will update you, give me a day I will respond back

[selvaraj-xemplar]

Hello @bobpskier, sorry for delay in response I struck in other things

great work appreciated your effort 👏 , its working fine now I tested in Firefox, Chrome and Safari, I have seen recorded audio is not playing in firefox and safari but working in chrome. I think we can remove that playback button if not able to fix.

I have one more problem with voice when I tried with HTML response with contentType:CustomPayload``, its not converting as HTML.

Attached the screenshot