microsoft / botbuilder-js

Welcome to the Bot Framework SDK for JavaScript repository, which is the home for the libraries and packages that enable developers to build sophisticated bot applications using JavaScript.
https://github.com/Microsoft/botframework
MIT License
685 stars 281 forks source link

Skill Bot not working with Alexa channel #4307

Closed breakingram closed 2 years ago

breakingram commented 2 years ago

JS sample 80.skills-simple-bot-to-bot Skill Bot does not work with Alexa channel.

[PROBLEM DESCRIPTION] Customer deployed to Azure the sample https://github.com/microsoft/BotBuilder-Samples/tree/main/samples/javascript_nodejs/80.skills-simple-bot-to-bot.

Bots work fine with Webchat channel but fail with Alexa one.

Root bot replies correctly to questions from Alexa channel, but when Skill bot is called and a question is sent from Alexa channel, no reply is received.

[TROUBLESHOOTING STEPS] I deployed the same sample and was able to reproduce the issue on Alexa channel. Apart from that, I added Webchat and Teams channels and issue does not happen on them.

Per my troubleshooting, Alexa only receive the first welcome message when connecting to Skill bot, after that whatever the question is sent from Alexa, no reply is received and Alexa close the connection.

I could see Skill bot receives messages from Alexa instead.

Below, I am appending image with working and non-working scenarios.

Teams channel: Connecting to Root bot image

Connecting to Skill bot You can see Skill bot sends two welcome messages, then it echoes every message I send. image

Alexa channel:

Connection to Root bot. It replies fine. image

Connecting to Skill bot.

Here, only first welcome message is received, then it does not echo messages I send and on debugging console I can see Alexa disconnects from Skill bot after sending the second message. I guess it is because Alexa does not receive the reply to the first message I sent. image image

And on Alexa debugging information, connection is closed.

{ "header": { "namespace": "SkillDebugger", "name": "CaptureDebuggingInfo", "messageId": "ded9bc2d-67ff-43e8-8801-80cd0e04fc2a" }, "payload": { "skillId": "amzn1.ask.skill.d10705f7-1027-4841-9699-dc731ce140f5", "timestamp": "2022-04-22T12:58:07.704Z", "dialogRequestId": "48999c37-8b95-4364-9d7c-e20e8a22c6e8", "skillRequestId": "amzn1.echo-api.request.21f74d56-1118-495b-a0e6-1d0e2652bb3d", "type": "SkillExecutionInfo", "content": { "invocationRequest": { "endpoint": "https://alexa.botframework.com/api/v1/bots/DfvBOSQBoYf/simple-root-bot-80-js", "body": { "version": "1.0", "session": { "new": false, "sessionId": "amzn1.echo-api.session.c1fb3a06-df9c-47e5-9ede-60ea8dfc3faf", "application": { "applicationId": "amzn1.ask.skill.d10705f7-1027-4841-9699-dc731ce140f5" }, "attributes": {}, "user": { "userId": "amzn1.ask.account.AESB36F2J4RID2B46HL5EUANMAAVEG2LV37G33NDOJL7R76U3XOUN2QFR6PCC7EWHTBDFMWDEVTLX6AUI64SQNHAC5D2DTWNMYM3MUGLKMYGJPEK2SKNTPNN3QMM3ZI67OGO7SHZ4W2UXPQIIKSPJICD4QJIJHN4AZMQB6Z6RUBXCJ675DMKZOZPNLRAAFUEIX2UVPARXGJ3LOA" } }, "context": { "Viewports": [ { .... }, "System": { "application": { "applicationId": "amzn1.ask.skill.d10705f7-1027-4841-9699-dc731ce140f5" }, "user": { "userId": "amzn1.ask.account.AESB36F2J4RID2B46HL5EUANMAAVEG2LV37G33NDOJL7R76U3XOUN2QFR6PCC7EWHTBDFMWDEVTLX6AUI64SQNHAC5D2DTWNMYM3MUGLKMYGJPEK2SKNTPNN3QMM3ZI67OGO7SHZ4W2UXPQIIKSPJICD4QJIJHN4AZMQB6Z6RUBXCJ675DMKZOZPNLRAAFUEIX2UVPARXGJ3LOA" }, "device": { "deviceId": "amzn1.ask.device.AGTYF2U3G3BJP4LGZ4SQOVZ5F5JF3R2JBENJRLUXAM3VPIGUJ4JUEGWOU2MYJERWNYB3VEJ633P4YPU347O7FG45AAIEYRRG2E4QAS2X7TP7LJMPZZKMXBCSKFIZAMEUV7RRKYHHL6NW7D6JSTKQTTK3U5GJXKSHF6I4FAMIPWQ2MMTULOGXY", "supportedInterfaces": {} }, "apiEndpoint": "https://api.amazonalexa.com", "apiAccessToken": "ey........" } }, "request": { "type": "IntentRequest", "requestId": "amzn1.echo-api.request.21f74d56-1118-495b-a0e6-1d0e2652bb3d", "locale": "en-US", "timestamp": "2022-04-22T12:58:07Z", "intent": { "name": "GetUserIntent", "confirmationStatus": "NONE", "slots": { "phrase": { "name": "phrase", "value": "pizza", "resolutions": { "resolutionsPerAuthority": [ { "authority": "amzn1.er-authority.echo-sdk.amzn1.ask.skill.d10705f7-1027-4841-9699-dc731ce140f5.phrase", "status": { "code": "ER_SUCCESS_NO_MATCH" } } ] }, "confirmationStatus": "NONE", "source": "USER", "slotValue": { "type": "Simple", "value": "pizza", "resolutions": { "resolutionsPerAuthority": [ { "authority": "amzn1.er-authority.echo-sdk.amzn1.ask.skill.d10705f7-1027-4841-9699-dc731ce140f5.phrase", "status": { "code": "ER_SUCCESS_NO_MATCH" } ..... }, "invocationResponse": { "body": { "version": "1.0", "response": { "outputSpeech": { "type": "PlainText", "text": "" }, "shouldEndSession": true, "type": "_DEFAULT_RESPONSE" } } }, "metrics": { "skillExecutionTimeInMilliseconds": 469 }

breakingram commented 2 years ago

Hi @JuanAr.

Same behavior occurs for C# sample 80.skills-simple-bot-to-bot as well.

The below code only resolves for DotNet C# sample. In RootBot.cs file, update the SendToSkill method with the following code:

private async Task SendToSkill(ITurnContext turnContext, BotFrameworkSkill targetSkill, CancellationToken cancellationToken)
{
    // NOTE: Always SaveChanges() before calling a skill so that any activity generated by the skill
    // will have access to current accurate state.
    await _conversationState.SaveChangesAsync(turnContext, force: true, cancellationToken: cancellationToken);

    // Clone activity and update its delivery mode.
    var activity = JsonConvert.DeserializeObject<Activity>(JsonConvert.SerializeObject(turnContext.Activity));
    activity.DeliveryMode = DeliveryModes.ExpectReplies;

    // Create a conversationId to interact with the skill and send the activity
    var options = new SkillConversationIdFactoryOptions
    {
        FromBotOAuthScope = turnContext.TurnState.Get<string>(BotAdapter.OAuthScopeKey),
        FromBotId = _botId,
        Activity = turnContext.Activity,
        BotFrameworkSkill = targetSkill
    };
    var skillConversationId = await _conversationIdFactory.CreateSkillConversationIdAsync(options, cancellationToken);

    using var client = _auth.CreateBotFrameworkClient();

    // Route the activity to the skill.
    var expectRepliesResponse = await client.PostActivityAsync<ExpectedReplies>(_botId, targetSkill.AppId, targetSkill.SkillEndpoint, _skillsConfig.SkillHostEndpoint, skillConversationId, activity, cancellationToken);

    // Check response status.
    if (!expectRepliesResponse.IsSuccessStatusCode())
    {
        throw new HttpRequestException($"Error invoking the skill id: \"{targetSkill.Id}\" at \"{targetSkill.SkillEndpoint}\" (status is {expectRepliesResponse.Status}). \r\n {expectRepliesResponse.Body}");
    }

    // Route response activities back to the channel.
    var responseActivities = expectRepliesResponse.Body?.Activities;

    foreach (var responseActivity in responseActivities)
    {
        if (responseActivity.Type == ActivityTypes.EndOfConversation)
        {
            await EndConversationAsync(responseActivity, turnContext, cancellationToken);
        }
        else
        {
            await turnContext.SendActivityAsync(responseActivity, cancellationToken);
        }
    }
}

/// <summary>
/// Clears storage variables and sends the end of conversation activities.
/// </summary>
/// <param name="activity">End of conversation activity.</param>
/// <param name="turnContext">Context for the current turn of conversation.</param>
/// <param name="cancellationToken">CancellationToken propagates notifications that operations should be cancelled.</param>
/// <returns>A <see cref="Task{TResult}"/> representing the result of the asynchronous operation.</returns>
private async Task EndConversationAsync(Activity activity, ITurnContext turnContext, CancellationToken cancellationToken)
{
    // Forget delivery mode and skill invocation.
    await _activeSkillProperty.DeleteAsync(turnContext, cancellationToken);

    // Show status message, text and value returned by the skill
    var eocActivityMessage = $"Received {ActivityTypes.EndOfConversation}.\n\nCode: {activity.Code}.";
    if (!string.IsNullOrWhiteSpace(activity.Text))
    {
        eocActivityMessage += $"\n\nText: {activity.Text}";
    }

    if (activity.Value != null)
    {
        eocActivityMessage += $"\n\nValue: {JsonConvert.SerializeObject(activity.Value)}";
    }

    await turnContext.SendActivityAsync(MessageFactory.Text(eocActivityMessage), cancellationToken);

    // We are back at the host.
    await turnContext.SendActivityAsync(MessageFactory.Text("Back in the host bot."), cancellationToken);

    await _conversationState.SaveChangesAsync(turnContext, false, cancellationToken);
}
breakingram commented 2 years ago

Perhaps translating the above code to Javascript might fix JS sample 80.skills-simple-bot-to-bot in respect with Alexa channel?

erquirogasw commented 2 years ago

Hi, @ramfattah .

We were able to reproduce the problem. While testing, we noticed that the first activity the bot receives from Alexa has the delivery mode set to expect replies. So the channel is waiting for this mode of communication, that's why the normal delivery mode, with asynchronous responses, was not working. We tested the proposed changes in the JS samples, setting the delivery mode to "expect replies" and sending all the activities we receive from the skill in a single response to the channel and it worked. However, these samples don't support expect replies and probably they don't need to do it. (issue#2313). You can use the bots from the BotFramework-FunctionalTests repository as samples as they support both delivery modes and we confirmed that they are working with Alexa. Let us know if you have any questions.

Regards.

breakingram commented 2 years ago

Hi @erquirogasw,

Looks like the skill is still not working in Alexa channel using SimpleHostBot as rootbot and EchoSkillBot as skill bot. Also, in Alexa, I noticed the suggestion choices were not showing up. Used numbers to select choices.

Alexa (skill not working): image

Screenshot in WebChat (working):

any thoughts? looking forward to hearing from you soon.

erquirogasw commented 2 years ago

Hi @ramfattah .

We tested the SimpleHostBot and EchoSkillBot and worked as expected in Alexa. Could you provide us with information on how you have done the tests? Have you made any modifications to the bots? We were able to visualize through the ngrok web ui that in our test the deliveryMode is correct. Can you check using ngrok if the message sent to the skill is using expectReplies?

This screenshot shows the bots working in Alexa.

image

Regards.

breakingram commented 2 years ago

Hi @erquirogasw, Thank you so much.

Indeed I made code changes I did not realize. Shame on me.

Using a fresh SimpleHostBot as rootbot and EchoSkillBot as skill bot works as expected with Alexa channel.

image

Thanks for the help @erquirogasw :)