search

alexa / alexa-smart-screen-sdk

⛔️ DEPRECATED Active at https://github.com/alexa/avs-device-sdk
Apache License 2.0
76 stars 25 forks source link

No smart screen content displayed while AVS is speaking. #128

Closed zhaokai3000 closed 2 years ago

zhaokai3000 commented 2 years ago

IMPORTANT: Before you create an issue, please take a look at our Issue Reporting Guide.

Briefly summarize your issue:

After saying "alexa, tell me a story", alexa state changed from thinking to idle immediately. Nothing but the initial "say alexa to talk" string is displayed on smart screen while AVS is telling me a story.

What is the expected behavior?

Smart screen displays the story information while AVS is speaking.

What behavior are you observing?

Nothing but the initial "say alexa to talk" string is displayed on smart screen while AVS is telling me a story.

Provide the steps to reproduce the issue, if applicable:

1st we customized the avs device sdk to use a audio input process engine, avs sample app works fine, the state seems right while we make a call like "alexa, tell me a joke", the state change from idle to listening to thinking to speaking, and to idle again after speaking. 2nd port the changes to avs sample app to alexa smart screen sample app. 3rd run the alexa smart screen sample app.

Tell us about your environment:

avs device sdk version 1.26.0; alexa smart screen sdk version 2.9.0; websocketpp version 0.8.2; apl-core-library version 1.8.1; apl-client-library version 1.8.2.

Tell us what hardware you're using:

Tell us about your OS (Type & version):

Have you tried the same use case with AVS Device SDK SampleApp?

logs maybe relevant (more logs refer to the log file attached): #####################################

ALEXA STATE: THINKING

#####################################

2022-04-08 09:08:14.168 [ 14] 0 GUIManager:ChannelFocusChanged:channelName=Content,newFocus=BACKGROUND 2022-04-08 09:08:14.169 [ 5] E AplClientBridge:invokeExtensionEventHandlerFailed::Root context is missing #################################

ALEXA STATE: IDLE

#################################

2022-04-08 09:08:14.312 [ 14] 0 GUIManager:ChannelFocusChanged:channelName=Dialog,newFocus=NONE 2022-04-08 09:08:14.312 [ 14] 0 GUIManager:ChannelFocusChanged:channelName=Content,newFocus=FOREGROUND 2022-04-08 09:08:16.693 [ 5] E AplClientBridge:invokeExtensionEventHandlerFailed::Root context is missing 2022-04-08 09:08:16.697 [ 5] E GUIManager:getAudioItemOffset:reason=Null MediaPropertiesInterface 2022-04-08 09:08:16.713 [ 5] E GUIManager:getAudioItemOffset:reason=Null MediaPropertiesInterface 2022-04-08 09:08:16.729 [ 5] E GUIManager:getAudioItemOffset:reason=Null MediaPropertiesInterface 2022-04-08 09:08:16.746 [ 5] E GUIManager:getAudioItemOffset:reason=Null MediaPropertiesInterface 2022-04-08 09:08:16.762 [ 5] E GUIManager:getAudioItemOffset:reason=Null MediaPropertiesInterface 2022-04-08 09:08:16.778 [ 5] E GUIManager:getAudioItemOffset:reason=Null MediaPropertiesInterface 2022-04-08 09:08:16.794 [ 5] E GUIManager:getAudioItemOffset:reason=Null MediaPropertiesInterface

alexa smart screen error log.txt

powj commented 2 years ago

Hi @zhaokai3000 , I am unable to reproduce this on our reference SampleApp, can you confirm if you see any visuals for other utterances, such as weather?

From your logs I see a few errors when APL package dependencies are downloaded: 2022-04-08 09:09:21.466 [ 1b] E CachingDownloadManager:downloadFromSourceFailed:statusCode=HTTP_RESPONSE_CODE_UNDEFINED,reason=nonSuccessStatusCodeFromGetHeader

The failure to download APL dependencies will cause an issue so you may wish to investigate why your device is receiving undefined response codes when attempting to perform the download. This function is likely where you would want to start the investigation: https://github.com/alexa/alexa-smart-screen-sdk/blob/master/modules/Alexa/SampleApp/src/CachingDownloadManager.cpp#L198

zhaokai3000 commented 2 years ago

Thanks for your quick response @powj Nothing is displayed for utterances like weather, but the log of status seems ok (alexa status changed from thinking to speaking before change to idle): wakeup sucess second!!! wakeup success --------------------wakeup:8000,17440,34880 Amazon Keyword startIndex 4965600 endIndex 4975040 2022-04-11 03:44:22.706 [ e] 3 AlexaPresentation:executeProvideState:token=8 ######################################

ALEXA STATE: LISTENING

######################################

2022-04-11 03:44:22.715 [ 16] 0 GUIManager:ChannelFocusChanged:channelName=Dialog,newFocus=FOREGROUND vad begin #####################################

ALEXA STATE: THINKING

#####################################

2022-04-11 03:44:37.189 [ 1e] 0 GUIManager:ChannelFocusChanged:channelName=Dialog,newFocus=FOREGROUND #####################################

ALEXA STATE: SPEAKING

#####################################

#################################

ALEXA STATE: IDLE

#################################

2022-04-11 03:44:48.508 [ 9] 0 GUIManager:ChannelFocusChanged:channelName=Dialog,newFocus=NONE 2022-04-11 03:44:49.163 [ d] W AlexaPresentation:handleRenderDocumentDirectiveInExecutor::Failed to find presentationSession autoInitializedExtensions 2022-04-11 03:44:49.163 [ d] 0 AlexaPresentation:handleRenderDocumentDirectiveInExecutor::PresentationSessionChanged:previousSkillId=,newSkillId=Domain\:Application\:Weather 2022-04-11 03:44:49.164 [ 21] W AplClientBridge:getAplClientRendererFromWindowId:targetWindowId=:Unable to find renderer for this windowId 2022-04-11 03:44:49.165 [ 9] 0 GUIManager:ChannelFocusChanged:channelName=Visual,newFocus=FOREGROUND 2022-04-11 03:44:49.165 [ d] 3 AlexaPresentation:executeRenderDocumentEvent:prevState=IDLE,nextState=ACQUIRING 2022-04-11 03:44:49.167 [ d] 3 AlexaPresentation:executeRenderDocumentCallbacks:previousToken=,newToken=amzn1.as-tt.v1.Domain\:Application\:Weather#TID#165ed515-177a-4ec7-b2ca-2a5aadc34e24,isClear=false,windowId=tvOverlayLandscape 2022-04-11 03:44:49.171 [ d] 3 AlexaPresentation:executeOnFocusChangedEvent:prevState=ACQUIRING,nextState=DISPLAYING 2022-04-11 03:44:49.171 [ 15] 0 AplClientBridge:AplCoreEngine::AplCoreExtensionManager:getExtension : aplext:backstack:10 2022-04-11 03:44:55.733 [ e] 3 GUILogBridge:GUILog:component=WSClient:message sent, type: deviceWindowState 2022-04-11 03:44:55.766 [ e] 3 GUILogBridge:GUILog:component=APLRenderer:APL Renderer Set View Size: 1520 x 860 2022-04-11 03:44:59.018 [ 19] 3 GUILogBridge:GUILog:component=WSClient:message sent, type: aplEvent 2022-04-11 03:44:59.077 [ 19] 3 GUILogBridge:GUILog:component=APLRenderer:APL Renderer Set View Size: 1520 x 860 2022-04-11 03:44:59.082 [ 15] W AplClientBridge:handleBuild::Unable to inflate document with current chosen scaling. 2022-04-11 03:44:59.123 [ 15] W AplClientBridge:handleBuild::Unable to inflate document with current chosen scaling. 2022-04-11 03:44:59.124 [ 15] E AplClientBridge:handleBuildFailed::Unable to inflate document 2022-04-11 03:44:59.130 [ 19] 3 GUILogBridge:GUILog:component=APLRenderer:APL Renderer Set View Size: 1520 x 860 2022-04-11 03:44:59.142 [ 20] 3 AlexaPresentation:processRenderDocumentResultExecutor:token=amzn1.as-tt.v1.Domain\:Application\:Weather#TID#165ed515-177a-4ec7-b2ca-2a5aadc34e24,result=false 2022-04-11 03:44:59.150 [ 20] 3 AlexaPresentation:executeRenderDocumentCallbacks:previousToken=amzn1.as-tt.v1.Domain\:Application\:Weather#TID#165ed515-177a-4ec7-b2ca-2a5aadc34e24,newToken=,isClear=true,windowId= 2022-04-11 03:44:59.152 [ 15] 0 AplClientBridge:AplCoreEngine::AplCoreExtensionManager:getExtension : aplext:backstack:10 2022-04-11 03:44:59.153 [ 20] 3 AlexaPresentation:executeClearCardEvent:prevState=DISPLAYING,nextState=RELEASING 2022-04-11 03:44:59.156 [ 20] 3 AlexaPresentation:executeProvideState:token=9 2022-04-11 03:44:59.157 [ 20] 3 AlexaPresentation:executeOnFocusChangedEvent:prevState=RELEASING,nextState=IDLE 2022-04-11 03:44:59.158 [ 1d] 0 GUIManager:ChannelFocusChanged:channelName=Visual,newFocus=NONE 2022-04-11 03:44:59.212 [ 19] 3 GUILogBridge:GUILog:component=WSClient:message sent, type: deviceWindowState 2022-04-11 03:44:59.218 [ 15] E AplClientBridge:handleUpdateDisplayStateFailed::Root context is missing 2022-04-11 03:44:59.218 [ 19] 3 GUILogBridge:GUILog:component=WSClient:message sent, type: aplEvent

powj commented 2 years ago

Can you share your smart screen sdk configuration json file?

It appears that you are using a non-standard screen resolution which is causing APL document inflation to fail. There are a number of pre-defined viewport profiles listed here , the APL runtime will attempt to scale documents, however if your configuration does not fall into one of these categories then APL documents may either fail to render correctly, or will not render at all.

zhaokai3000 commented 2 years ago

I tried to display the smart screen in a 760*430 pixel window on a lcd screen, the resolution I used may'be is not standard as you mentioned. Which profile do you suggest that I should follow? Here is my smart screen sdk configuration json file: { "sampleApp": { "websocketInterface":"127.0.0.1", "websocketPort":8933, "websocketCertificateAuthority":"/etc/avs/certs/ca.cert", "websocketCertificate":"/etc/avs/certs/server.chain", "websocketPrivateKey":"/etc/avs/certs/server.key" }, "alexaPresentationCapabilityAgent": { // The minimum state reporting interval in milliseconds for the AlexaPresentation CA // "minStateReportIntervalMs": 600 // The interval between checks for context changes // "stateReportCheckIntervalMs": 2000 }, "gui": { "appConfig": { "description": "Smart Screen Sample UI", "mode": "TV", "emulateDisplayDimensions": true, "scaleToFill": true, "audioInputInitiator": "WAKEWORD", "windows": [ { "id": "tvFullscreen", "templateId": "tvFullscreen", "sizeConfigurationId": "fullscreen", "interactionMode": "tv", "supportedExtensions": [ "aplext:backstack:10" ] }, { "id": "tvOverlayLandscape", "templateId": "tvOverlayLandscape", "windowPosition": "bottom", "sizeConfigurationId": "landscapePanel", "interactionMode": "tv_overlay", "supportedExtensions": [ "aplext:backstack:10" ] } ], "defaultWindowId": "tvFullscreen", "deviceKeys": { "talkKey": { "code": "KeyA", "keyCode": 65, "key": "a" }, "backKey": { "code": "KeyB", "keyCode": 66, "key": "b" }, "exitKey": { "code": "Escape", "keyCode": 27, "key": "Escape" }, "toggleCaptionsKey": { "code": "KeyC", "keyCode": 67, "key": "c" }, "toggleDoNotDisturbKey": { "code": "KeyD", "keyCode": 68, "key": "d" } } }, "visualCharacteristics": [ { "type": "AlexaInterface", "interface": "Alexa.InteractionMode", "version": "1.1", "configurations": { "interactionModes": [ { "id": "tv", "uiMode": "TV", "interactionDistance": { "unit": "INCHES", "value": 130 }, "touch": "UNSUPPORTED", "keyboard": "SUPPORTED", "video": "SUPPORTED", "dialog": "SUPPORTED" }, { "id": "tv_overlay", "uiMode": "TV", "interactionDistance": { "unit": "INCHES", "value": 130 }, "touch": "UNSUPPORTED", "keyboard": "SUPPORTED", "video": "UNSUPPORTED", "dialog": "SUPPORTED" } ] } }, { "type": "AlexaInterface", "interface": "Alexa.Presentation.APL.Video", "version": "1.0", "configurations": { "video": { "codecs": [ "H_264_42", "H_264_41" ] } } }, { "type": "AlexaInterface", "interface": "Alexa.Display.Window", "version": "1.0", "configurations": { "templates": [ { "id": "tvFullscreen", "type": "STANDARD", "configuration": { "sizes": [ { "type": "DISCRETE", "id": "fullscreen", "value": { "unit": "PIXEL", "value": { "width": 1520, "height": 860 } } } ], "interactionModes": [ "tv" ] } }, { "id": "tvOverlayLandscape", "type": "OVERLAY", "configuration": { "sizes": [ { "type": "DISCRETE", "id": "landscapePanel", "value": { "unit": "PIXEL", "value": { "width": 1520, "height": 860 } } } ], "interactionModes": [ "tv_overlay" ] } } ] } }, { "type": "AlexaInterface", "interface": "Alexa.Display", "version": "1.0", "configurations": { "display": { "type": "PIXEL", "touch": [ "UNSUPPORTED" ], "shape": "RECTANGLE", "dimensions": { "resolution": { "unit": "PIXEL", "value": { "width": 1520, "height": 860 } }, "physicalSize": { "unit": "INCHES", "value": { "width": 56.7, "height": 31.9 } }, "pixelDensity": { "unit": "DPI", "value": 160 }, "densityIndependentResolution": { "unit": "DP", "value": { "width": 1520, "height": 860 } } } } } } ], "liveViewControllerOptions": { "physicalMicButtonHint": { "micButtonHintType": "Text", "micButtonHintSource": "A" }, "viewingDeviceMicUnsupported": false } } }

powj commented 2 years ago

The TV viewport is designed for 16:9 aspect ratios (such as 1920x1080). For your display, to get visuals which render reliably you may need to use/modify a hub profile (https://github.com/alexa/alexa-smart-screen-sdk/blob/master/modules/GUI/config/guiConfigSamples/GuiConfigSample_SmartScreenLargeLandscape.json).

Please note however that if you require content to be navigable with a remote then this is only supported with a uiMode of "TV"

zhaokai3000 commented 2 years ago

@powj Thanks! I'll have a try.