search

aws-samples / aws-lex-web-ui

Sample Amazon Lex chat bot web interface
Other
744 stars 471 forks source link

Mobile voice not working #58

Closed JohnCalhoun closed 7 months ago

JohnCalhoun commented 6 years ago

1) in mobile browsers where voice is not supported the microphone still appears 2) voice gets stuck in listening mode in some browsers

zenglicheng commented 6 years ago

Hello John and other Lex experts,

Is there any progress on this issue? We are also trying to enable LexWebUI to interact with LEX from a mobile device and encounter similar problems. I'd like to share more details of the problems we encountered here.

Our development context We are building a custom chat bot using AWS LEX. In our requirement, users need to access LEX service from our mobile app, which is developed on IONIC framework. Therefore, our technical objective is to embed or access the Lex web UI sample code (https://github.com/aws-samples/aws-lex-web-ui) in our IONIC app to interact with AWS LEX. Note because we are not developing native Android or iOS apps, we don't use AWS Mobile SDK to integrate with LEX.

Technical issue encountered We host the LexWebUI app on a S3 bucket and access it through https from IONIC app through the "cordova-plugin-crosswalk-webview", i.e. the Chromium webview. The native Android webview gives us more issues.

The text-based interaction works perfectly, however voice input/output experiences issues. We saw the following error in our logs:

error in startSpeechConversation Error: setting audio auto play failed [Object event]: at HTMLAudioElement.audio.onError: (xxx/lex-web-ui.js: 7223:16) (anonymous function@InputContainer.vue:2222

Interestingly, if we access the same lexWebUI URL from a Chrome browser on Android, voice IO can work. But with a native Internet Browser on Android, voice IO cannot work either.

Any guidance on this will be highly appreciated. We are still trying to resolve the issue and will share our experience/findings if we can make progress.

regards zx

zenglicheng commented 6 years ago

We did some research and it is understood by us that this issue could be due to a restriction of webkit that requires a “user gesture” to play or pause an audio or video element, which is enabled in Opera for Android, Chrome for Android, the default Android browser, Safari for iOS and probably other browsers. This means audio auto play is by default not possible with webview.

So I am wondering if there is a way for disabling "auto audio play" in lex web ui and uses a touch button for user to enable audio playback?

Another way is to use the trick as documented in "https://blog.foolip.org/2014/02/10/media-playback-restrictions-in-blink/" to have an initial gesture to allow for auto-play and all subsequent audio sources can be auto played?