How to use Alexa Presentation states for launching browser wrt Render directive

[abkant]

IMPORTANT: Before you create an issue, please take a look at our Issue Reporting Guide.

Briefly summarize your issue:

This is the continuation of - https://github.com/alexa/alexa-smart-screen-sdk/issues/121

Thanks for the clarification. I have a few more queries for my trials done as following and hence reopening this

1. Launch the browser much ahead (may be at the listening/thinking state) before receiving the render Directive - Browser displays the display card successfully

2. Launch the browser just when we receive render Directive - Browser misses the display of APL card and nothing gets displayed in the browser. In this case, we do see that the communication is established between websocket client and server with all the commands received by the js client. If we receive another render Directive in this situation, then browser is able to display the card successfully. Please refer the attached logs where the relaunch is done around this timestamp [2022-02-10 08:17:29.061] AlexaLogs_ReLaunch.txt

3. Relaunch/reload the browser in between of the active directive - Browser misses the display of remaining portion of APL card and nothing gets displayed in the browser. If we receive another render Directive in this situation, then browser is able to display the card successfully.

Can you please help us in understanding the dependency of SSDK with browser in more depth. As our requirement was to launch the browser only when we receive render Directive, for all other cases we can skip launching the browser. With these trials it seems like browser should be active always, is this the requirement of SSDK? Can you please comment.

What is the expected behavior?

NA

What behavior are you observing?

NA

Provide the steps to reproduce the issue, if applicable:

NA

Tell us about your environment:

What version of the AVS Device SDK are you using?

    <2.8.0>

Tell us what hardware you're using:

• [ ] Desktop / Laptop
• [ ] Raspberry Pi
• [ X] Other - tell us more: Setup Box

Tell us about your OS (Type & version):

• [ X] Linux
• [ ] MacOS
• [ ] Raspbian Stretch
• [ ] Raspbian Jessy
• [ ] Other - tell us more:

Have you tried the same use case with AVS Device SDK SampleApp?

• [X] The issue is not applicable to the AVS Device SDK SampleApp.
• [ ] The issue is not reproducible on the AVS Device SDK SampleApp.

[sngoc-amazon]

Hi abkant,

Once a websocket connection is established for every browser session, initResponse/initResponse, guiConfiguration/deviceWindowState should be sent from C++ side/to the GUI browser before the renderDocument message should be sent to GUI browser.

Can you share GUI logs and corresponding C++ SDK logs so we can take a look further?

Thanks, Kelly

[abkant]

Hi Kelly, Ok, so it means handshake happens between the SDK and GUI browser side. I have SDK side logs here, please have a look [ AlexaLogs_ReLaunch.txt ]. I am figuring out how to take GUI browser side logs, will upload it soon.

Thanks, Abhay

[1cengizk]

@abkant To collect GUI browser side logs, you could use Option+Command+J (on Mac) while the browser is open and then choose all levels in the console window to collect verbose logs. Or you could go to Settings in Chrome, and choose More Tools > Developer Tools.

Revisiting your questions; for (3), I think you mean refreshing the browser when an APL document is being rendered (for example a Wikipedia article). If you refresh the browser in that case, the websocket connection between the native layer and GUI is reset as well causing the active APL card not to be displayed anymore. You would see the following in the SDK logs:

2022-02-22 01:31:37.449 [ 13] I WebSocketServer:onConnectionClose

I am not sure if I understand your use case in (2). In that case, if the browser is not open, SDK will stay in the CONNECTING state anyway; so, you should not be able to send an utterance to Alexa and get a response back. If you somehow have a renderDocument() directive ready beforehand, opening the browser will cause SDK to establish the websocket connection. Unless that connection is active, nothing will reach the GUI layer.

Could you describe case (2) and (3) above with specific lines from the log file you attached?

[abkant]

Actually my device where i am validating this is chromium based browser but doesn't have rich debugging interfaces. I am trying to find alternative for this.

For case 2, I am trying to control the browser from GuiClient. Code snippet is as following void GUIClient::onRenderDirectiveReceived(..) { .... // Launch the browser which loads "index.html" }

void GUIClient::clearDocument(..)
{
    ...
    // Close the browser
}

With this changes, we see the browser is getting launched succesfully, but browser doesn't display the APL card. 
Yes, you are right the SDK is in connected state, but because of re-handshake between SDK and JS client the APL card is missed?

For case 3: I got your point, because of websocket connection reset, the current APL card is not getting displayed and i hope this is the expected behaviour. Please comment. I haven't collected logs for this case.

[1cengizk]

@abkant For case (3), once the websocket connection is reset, SDK won't remember the most recent APL card that was being rendered. As a result, no APL card will be displayed until a new renderDocument() directive comes in.

For case (2), can you try putting your browser startup logic a bit earlier in the stack? For example, right after the following line: https://github.com/alexa/alexa-smart-screen-sdk/blob/d436bda61cd9dc11076c56bd90f7c89655d33dd8/modules/Alexa/SmartScreenCapabilityAgents/AlexaPresentation/src/AlexaPresentation.cpp#L949

Here is the flow:

(1) AlexaPresentation::executeRenderDocument() (2) GUIManager::renderDocument() ... (3) GUIManager::onRenderDirectiveReceived() (4) GUIClient::onRenderDirectiveReceived() (5) AplClientBridge::onRenderDirectiveReceived()

Currently, you have the browser startup logic in (4). It could be a lot earlier in the stack; for example in (1).

[abkant]

Thanks for the explanation, i am trying to do as following. Can you please comment void AlexaPresentation::handleDirective() { // Launch the Browser and wait till we get deviceWindowState update in GUIManager::handleDeviceWindowState(..) // After getting GUIManager::handleDeviceWindowState, proceed below ....... } void AlexaPresentation::executeClearCardEvent() { ...... // Wait for ~200ms for websocket communication and then Close the browser }

[mdaix-az]

Hi Abhay, I have noticed you are proposing to perform the "launch browser" action in AlexaPresentation::handleDirective() method. While this could potentially work I am worried there might be extra work for you to identify the correct directive type that should invoke the launch browser action.

On the contrary, as my colleague suggested - AlexaPresentation::executeRenderDocument() might be a better place to try since this method is still part of the "document rendering flow", which seems more suitable to what you are pursuing here.

Thanks, Michael

[abkant]

Hi Michael, Got your point, i will move the browser launch to executeRenderDocument. I see handleDirective being handled in other components like TemplateRuntime and LiveViewControllerCapabilityAgent. Do i need to handle this there too. Or if i move my logic to executeRenderDocument, then i don't need to handle it separately? Plz comment. Thanks, Abhay

[mdaix-az]

Hi Abhay, handleDirective is generally used for redirect incoming directives to correct handler functions base on their directive type/content. I am not sure I understand your question correctly - but AlexaPresentation::handleDirective still needs to be implemented and called for directives such as render document to be handled correctly. Let me know if you would like to elaborate on this.

Michael

[abkant]

Hi Michael, What I meant here is, do I need to handle my browser control logic in each handleDirective implementation. Because I see the implementation of handleDirective in TemplateRuntime and some other place too. So i need to repeat the same in these places too? Can you please help me in understanding where all we handle directives, so that i can design my logic accordingly. Thanks, Abhay

[mdaix-az]

Hi Abhay, Thanks for clarify - the answer is yes, you will need to implement or call your browser control logic from both handleDirective implementations.

The SDK utilizes both APL and TemplateRuntime to render visual content. For TemplateRuntime it actually depends on the GUI implementation to resolve templates into locally generated APL documents. So the GUI app needs to be loaded before we pass on the TemplateRuntime directives from the SDK.

Michael

[abkant]

Thanks Michael, understood now. Will try to implement the same and update here.

One more thing, i see the following in the logs, from where we get this 30s value. For few this value comes as 4s and for others 30s. Is this depends on the directive or ? "AlexaPresentation:executeStartOrExtendTimer:timeoutInMilliseconds.hasValue=true,timeoutinMilliseconds.value=30000" Abhay

[powj]

Hi Abhay, Yes, the timeout depends on the directive (see https://github.com/alexa/alexa-smart-screen-sdk/blob/d436bda61cd9dc11076c56bd90f7c89655d33dd8/modules/Alexa/SmartScreenCapabilityAgents/AlexaPresentation/src/AlexaPresentation.cpp#L623) . It can also be overridden by the APL document (https://developer.amazon.com/en-US/docs/alexa/alexa-presentation-language/apl-document.html#document_settings_property)

[abkant]

Appreciate the in-depth and detailed explanation.