search

microsoft / botbuilder-dotnet

Welcome to the Bot Framework SDK for .NET repository, which is the home for the libraries and packages that enable developers to build sophisticated bot applications using .NET.
https://github.com/Microsoft/botframework
MIT License
872 stars 480 forks source link

Microsoft.Bot.Builder.Adapters.Facebook return 400 (failure) on multiturn answer from QnA or adaptive card. #5049

Closed jarekgoslawski closed 3 years ago

jarekgoslawski commented 3 years ago

Version

4.9.4

Describe the bug

We use Facebook adapter in our virtual assistant build on template from MS to connect to our Workplace Integration (Workchat)

We use LUIS and QnA and Dispatcher in virtual assistant.

All works in Bot Framework Emulator (simple answer from QnA, Welcome card, multiturn answer, etc.) In Workchat (via Facebook Adapter) user see correct QnA answer simple answer, but when we have answer as Multi-Turn (with additional buttons) or it should to show Welcome card ... our bot fails. In debugger we see then Response 400.

Chat for some time is unresponsive and ends with "The bot encountered an error or bug."

To Reproduce

Steps to reproduce the behavior: 1.Start chat 2.Ask question to QnA which return Multi-Turn answer

Expected behavior

We shoudl to see answer and additional button/Link with additional question (classic multiturn answer)

Screenshots

what we see on debug console:

correct answer: image

failure when answer is Multi-Turn image

failure when answer is a Card

image image

Additional context

In Emulator it works as expected, but over Workchat not.

v-kydela commented 3 years ago

Thank you for the report, @jarekgoslawski

Can you tell us what Workchat is? Do you mean Workplace?

jarekgoslawski commented 3 years ago

Yep, WorkChat aka Workplace Chat is a Messenger in Workplace world

v-kydela commented 3 years ago

@jarekgoslawski - I can see that you said you don't encounter the problem in Emulator. In order to help diagnose the problem, can you try your bot in Facebook Messenger (non-Workplace) and see if you get the problem there?

jarekgoslawski commented 3 years ago

Is the way to pick-up full API call which our botengine does into Workplace? I have another one ticket open in our Workplace support, and they need this full API call to check why it is not correctly responded.

Best regards

[cid:image001.png@01D6D3D3.30690F00] Jarek GOSLAWSKI - Messaging Specialist DANONE ITCC EMEA

From: Kyle Delaney notifications@github.com Sent: Tuesday, December 15, 2020 10:59 PM To: microsoft/botbuilder-dotnet botbuilder-dotnet@noreply.github.com Cc: GOSLAWSKI Jaroslaw Jaroslaw.GOSLAWSKI@external.danone.com; Mention mention@noreply.github.com Subject: Re: [microsoft/botbuilder-dotnet] Microsoft.Bot.Builder.Adapters.Facebook return 400 (failure) on multiturn answer from QnA or adaptive card. (#5049)

@jarekgoslawskihttps://github.com/jarekgoslawski - I can see that you said you don't encounter the problem in Emulator. In order to help diagnose the problem, can you try your bot in Facebook Messenger (non-Workplace) and see if you get the problem there?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/microsoft/botbuilder-dotnet/issues/5049#issuecomment-745593852, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AMUNEB6TYVRIHLM7GYDSWETSU7L3LANCNFSM4U2ZTCYQ.

Danone Sp. z o.o., 01-103 Warszawa, ul. Redutowa 9/23, KRS 0000014227, Sąd Rejonowy dla m. st. Warszawy w Warszawie, XII Wydział Gospodarczy KRS, Kapitał Zakładowy 53 550 000,00 zł, NIP 527-020-44-71, BDO: 000013290

Ce message électronique et tous les fichiers attachés qu'il contient sont confidentiels et destinés exclusivement à l'usage de la personne à laquelle ils sont adressés. Si vous avez reçu ce message par erreur, merci de le retourner à son émetteur. Les idées et opinions présentées dans ce message sont celles de son auteur, et ne représentent pas nécessairement celles de DANONE ou d'une quelconque de ses filiales. La publication, l'usage, la distribution, l'impression ou la copie non autorisée de ce message et des attachements qu'il contient sont strictement interdits.

This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual to whom it is addressed. If you have received this email in error please send it back to the person that sent it to you. Any views or opinions presented are solely those of its author and do not necessarily represent those of DANONE or any of its subsidiary companies. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its associated attachments is strictly prohibited.

v-kydela commented 3 years ago

@jarekgoslawski - I'm sorry, I'm not sure I understand your question. Are you able to connect your bot to Facebook Messenger (non-Workplace)?

jarekgoslawski commented 3 years ago

Sorry I was not enough clear.

I meant, how to see (catch in Debug or save to file) full URI of API request made by bot-engine to Workplace/Facebook (I can’t test it on Facebook (private) as I Use only Workplace (in company) and don’t have FB identity.)

Best regards

[cid:image001.png@01D6D560.6DD6AAD0] Jarek GOSLAWSKI - Messaging Specialist DANONE ITCC EMEA

From: Kyle Delaney notifications@github.com Sent: Thursday, December 17, 2020 7:50 PM To: microsoft/botbuilder-dotnet botbuilder-dotnet@noreply.github.com Cc: GOSLAWSKI Jaroslaw Jaroslaw.GOSLAWSKI@external.danone.com; Mention mention@noreply.github.com Subject: Re: [microsoft/botbuilder-dotnet] Microsoft.Bot.Builder.Adapters.Facebook return 400 (failure) on multiturn answer from QnA or adaptive card. (#5049)

@jarekgoslawskihttps://github.com/jarekgoslawski - I'm sorry, I'm not sure I understand your question. Are you able to connect your bot to Facebook Messenger (non-Workplace)?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/microsoft/botbuilder-dotnet/issues/5049#issuecomment-747628594, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AMUNEBYXAQ5BZ5Y3TIUCKRTSVJHFJANCNFSM4U2ZTCYQ.

Danone Sp. z o.o., 01-103 Warszawa, ul. Redutowa 9/23, KRS 0000014227, Sąd Rejonowy dla m. st. Warszawy w Warszawie, XII Wydział Gospodarczy KRS, Kapitał Zakładowy 53 550 000,00 zł, NIP 527-020-44-71, BDO: 000013290

Ce message électronique et tous les fichiers attachés qu'il contient sont confidentiels et destinés exclusivement à l'usage de la personne à laquelle ils sont adressés. Si vous avez reçu ce message par erreur, merci de le retourner à son émetteur. Les idées et opinions présentées dans ce message sont celles de son auteur, et ne représentent pas nécessairement celles de DANONE ou d'une quelconque de ses filiales. La publication, l'usage, la distribution, l'impression ou la copie non autorisée de ce message et des attachements qu'il contient sont strictement interdits.

This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual to whom it is addressed. If you have received this email in error please send it back to the person that sent it to you. Any views or opinions presented are solely those of its author and do not necessarily represent those of DANONE or any of its subsidiary companies. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its associated attachments is strictly prohibited.

v-kydela commented 3 years ago

@jarekgoslawski - While I cannot test a Workplace bot because I am not a Workplace admin, I believe I've discovered the cause of the problem by examining the source code and by testing a Facebook Messenger bot.

While the Azure Bot Service channel connectors do convert Bot Framework cards to their equivalents in the respective chat platforms, the Bot Framework channel adapters do not. This means the Facebook Messenger channel connector service will convert Bot Framework cards to Facebook Messenger templates, but the Facebook adapter does not support Bot Framework cards. If you try to send hero cards through the Facebook adapter, such as the hero cards generated by the QnA Maker library, you will get an error.

In order to solve your problem, you will need to manually convert the QnA Maker follow-up prompts into Facebook attachments or quick replies that the Facebook adapter will recognize, or just not try to send buttons to the user at all. You can have a look at the QnA Maker docs to see examples of the information QnA Maker sends to your bot for multi-turn conversations. You may be able to figure out how to build an adequate Facebook message just by looking at the Facebook adapter source code, but it's also a good idea to familiarize yourself with the Messenger platform docs.

Does this resolve your issue?

v-kydela commented 3 years ago

@jarekgoslawski - Are you still working on this?

jarekgoslawski commented 3 years ago

Yes, I’m trying to find way to export from bot engine full URI string sent to Workplace API to validate what is not aligned with Facebook standard.

Best regards

[cid:image001.png@01D6D910.08D96F60] Jarek GOSLAWSKI - Messaging Specialist DANONE ITCC EMEA

From: Kyle Delaney notifications@github.com Sent: Tuesday, December 22, 2020 8:22 PM To: microsoft/botbuilder-dotnet botbuilder-dotnet@noreply.github.com Cc: GOSLAWSKI Jaroslaw Jaroslaw.GOSLAWSKI@external.danone.com; Mention mention@noreply.github.com Subject: Re: [microsoft/botbuilder-dotnet] Microsoft.Bot.Builder.Adapters.Facebook return 400 (failure) on multiturn answer from QnA or adaptive card. (#5049)

@jarekgoslawskihttps://github.com/jarekgoslawski - Are you still working on this?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/microsoft/botbuilder-dotnet/issues/5049#issuecomment-749729368, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AMUNEBYRXGGHMBBD5T5ABGTSWDWTZANCNFSM4U2ZTCYQ.

Danone Sp. z o.o., 01-103 Warszawa, ul. Redutowa 9/23, KRS 0000014227, Sąd Rejonowy dla m. st. Warszawy w Warszawie, XII Wydział Gospodarczy KRS, Kapitał Zakładowy 53 550 000,00 zł, NIP 527-020-44-71, BDO: 000013290

Ce message électronique et tous les fichiers attachés qu'il contient sont confidentiels et destinés exclusivement à l'usage de la personne à laquelle ils sont adressés. Si vous avez reçu ce message par erreur, merci de le retourner à son émetteur. Les idées et opinions présentées dans ce message sont celles de son auteur, et ne représentent pas nécessairement celles de DANONE ou d'une quelconque de ses filiales. La publication, l'usage, la distribution, l'impression ou la copie non autorisée de ce message et des attachements qu'il contient sont strictement interdits.

This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual to whom it is addressed. If you have received this email in error please send it back to the person that sent it to you. Any views or opinions presented are solely those of its author and do not necessarily represent those of DANONE or any of its subsidiary companies. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its associated attachments is strictly prohibited.

v-kydela commented 3 years ago

@jarekgoslawski - It's not the URI string. It's the Bot Framework cards that Facebook doesn't recognize, and those are contained in the body of the request. This document might help you understand Bot Framework cards, but remember that you will want to use Facebook Messenger templates instead.

jarekgoslawski commented 3 years ago

Maybe I used wrong words (or shortening) how I want to investigate, sorry for another misunderstanding. I understand that (using your words from one of topic on stackoverflow.com, where you tried to help other users) “(…)If you send a hero card through the Facebook connector then it will automatically be converted to a button template (…)”

And I want to see effect of this conversion. How can I see this full request body (with encoded Json) which adapters sends to Facebook/Workplace? It allows me to check if request “composed” by bot and send over adapter is ok from FB/WP perspective

Best regards

[cid:image001.png@01D6D988.4ABFED00] Jarek GOSLAWSKI - Messaging Specialist DANONE ITCC EMEA

From: Kyle Delaney notifications@github.com Sent: Wednesday, December 23, 2020 5:52 PM To: microsoft/botbuilder-dotnet botbuilder-dotnet@noreply.github.com Cc: GOSLAWSKI Jaroslaw Jaroslaw.GOSLAWSKI@external.danone.com; Mention mention@noreply.github.com Subject: Re: [microsoft/botbuilder-dotnet] Microsoft.Bot.Builder.Adapters.Facebook return 400 (failure) on multiturn answer from QnA or adaptive card. (#5049)

@jarekgoslawskihttps://github.com/jarekgoslawski - It's not the URI string. It's the Bot Framework cards that Facebook doesn't recognize, and those are contained in the body of the request. This documenthttps://docs.microsoft.com/azure/bot-service/rest-api/bot-framework-rest-connector-add-rich-cards might help you understand Bot Framework cards, but remember that you will want to use Facebook Messenger templates instead.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/microsoft/botbuilder-dotnet/issues/5049#issuecomment-750384596, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AMUNEB4PMFEQN42XN4CQUOLSWINZFANCNFSM4U2ZTCYQ.

Danone Sp. z o.o., 01-103 Warszawa, ul. Redutowa 9/23, KRS 0000014227, Sąd Rejonowy dla m. st. Warszawy w Warszawie, XII Wydział Gospodarczy KRS, Kapitał Zakładowy 53 550 000,00 zł, NIP 527-020-44-71, BDO: 000013290

Ce message électronique et tous les fichiers attachés qu'il contient sont confidentiels et destinés exclusivement à l'usage de la personne à laquelle ils sont adressés. Si vous avez reçu ce message par erreur, merci de le retourner à son émetteur. Les idées et opinions présentées dans ce message sont celles de son auteur, et ne représentent pas nécessairement celles de DANONE ou d'une quelconque de ses filiales. La publication, l'usage, la distribution, l'impression ou la copie non autorisée de ce message et des attachements qu'il contient sont strictement interdits.

This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual to whom it is addressed. If you have received this email in error please send it back to the person that sent it to you. Any views or opinions presented are solely those of its author and do not necessarily represent those of DANONE or any of its subsidiary companies. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its associated attachments is strictly prohibited.

v-kydela commented 3 years ago

Thank you for the clarification, @jarekgoslawski. Would you mind linking to the Stack Overflow post in question?

It's true that the Facebook connector converts hero cards to button templates, but you're not using the Facebook connector, you're using the Facebook adapter. The Facebook adapter does not convert Bot Framework cards like hero cards to Facebook templates, as I explained in my previous comment:

While the Azure Bot Service channel connectors do convert Bot Framework cards to their equivalents in the respective chat platforms, the Bot Framework channel adapters do not. This means the Facebook Messenger channel connector service will convert Bot Framework cards to Facebook Messenger templates, but the Facebook adapter does not support Bot Framework cards. If you try to send hero cards through the Facebook adapter, such as the hero cards generated by the QnA Maker library, you will get an error.

In order to solve your problem, you will need to manually convert the QnA Maker follow-up prompts into Facebook attachments or quick replies that the Facebook adapter will recognize, or just not try to send buttons to the user at all. You can have a look at the QnA Maker docs to see examples of the information QnA Maker sends to your bot for multi-turn conversations. You may be able to figure out how to build an adequate Facebook message just by looking at the Facebook adapter source code, but it's also a good idea to familiarize yourself with the Messenger platform docs.

jarekgoslawski commented 3 years ago

It was this post https://stackoverflow.com/questions/59861665/does-botbuilder-support-facebook-messengers-button-template

Anyway, I understand now that there is not any equivalent between Facebook channel and Facebook adapter (working in fact with Workplace). I was believing that Facebook adapter can deal with something more than just text. (like Facebook channel did).

Regarding info on Microsoft site that we can’t use any longer Facebook Channel (https://docs.microsoft.com/en-us/azure/bot-service/bot-service-channel-connect-facebook?view=azure-bot-service-4.0#connect-a-bot-to-facebook-workplace) … “On December 16, 2019, Workplace by Facebook changed its security model for custom integrations. Prior integrations built using Microsoft Bot Framework v4 need to be updated to use the Bot Framework Facebook adapters per the instructions below prior to February 28, 2020.(…)”

… and my claimed here “issues” with using Facebook adapter in Virtual assistant, it looks that we can use adapter in limited scope, and we need to code everything.

Are there any plans to build adapter for Workplace (or add to Facebook adapter) full support of Facebook/Workplace cards/templates?

Anyway I still count on your support in my question about “How can I see this full request body (with encoded Json) which adapters sends to Facebook/Workplace?

Best regards

[cid:image001.png@01D6DCFE.01591EB0] Jarek GOSLAWSKI - Messaging Specialist DANONE ITCC EMEA

From: Kyle Delaney notifications@github.com Sent: Thursday, December 24, 2020 12:48 AM To: microsoft/botbuilder-dotnet botbuilder-dotnet@noreply.github.com Cc: GOSLAWSKI Jaroslaw Jaroslaw.GOSLAWSKI@external.danone.com; Mention mention@noreply.github.com Subject: Re: [microsoft/botbuilder-dotnet] Microsoft.Bot.Builder.Adapters.Facebook return 400 (failure) on multiturn answer from QnA or adaptive card. (#5049)

Thank you for the clarification, @jarekgoslawskihttps://github.com/jarekgoslawski. Would you mind linking to the Stack Overflow post in question?

It's true that the Facebook connector converts hero cards to button templates, but you're not using the Facebook connector, you're using the Facebook adapter. The Facebook adapter does not convert Bot Framework cards like hero cards to Facebook templates, as I explained in my previous comment:

While the Azure Bot Service channel connectors do convert Bot Framework cards to their equivalents in the respective chat platforms, the Bot Framework channel adapters do not. This means the Facebook Messenger channel connector service will convert Bot Framework cards to Facebook Messenger templates, but the Facebook adapter does not support Bot Framework cards. If you try to send hero cards through the Facebook adapter, such as the hero cards generated by the QnA Maker library, you will get an error.

In order to solve your problem, you will need to manually convert the QnA Maker follow-up prompts into Facebook attachments or quick replies that the Facebook adapter will recognize, or just not try to send buttons to the user at all. You can have a look at the QnA Maker docshttps://developers.facebook.com/docs/messenger-platform/send-messages to see examples of the information QnA Maker sends to your bot for multi-turn conversations. You may be able to figure out how to build an adequate Facebook message just by looking at the Facebook adapter source code, but it's also a good idea to familiarize yourself with the Messenger platform docshttps://developers.facebook.com/docs/messenger-platform/send-messages.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/microsoft/botbuilder-dotnet/issues/5049#issuecomment-750585180, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AMUNEBZKQ3JTDHAHH3EPCLTSWJ6RJANCNFSM4U2ZTCYQ.

Danone Sp. z o.o., 01-103 Warszawa, ul. Redutowa 9/23, KRS 0000014227, Sąd Rejonowy dla m. st. Warszawy w Warszawie, XII Wydział Gospodarczy KRS, Kapitał Zakładowy 53 550 000,00 zł, NIP 527-020-44-71, BDO: 000013290

Ce message électronique et tous les fichiers attachés qu'il contient sont confidentiels et destinés exclusivement à l'usage de la personne à laquelle ils sont adressés. Si vous avez reçu ce message par erreur, merci de le retourner à son émetteur. Les idées et opinions présentées dans ce message sont celles de son auteur, et ne représentent pas nécessairement celles de DANONE ou d'une quelconque de ses filiales. La publication, l'usage, la distribution, l'impression ou la copie non autorisée de ce message et des attachements qu'il contient sont strictement interdits.

This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual to whom it is addressed. If you have received this email in error please send it back to the person that sent it to you. Any views or opinions presented are solely those of its author and do not necessarily represent those of DANONE or any of its subsidiary companies. Unauthorized publication, use, dissemination, forwarding, printing or copying of this email and its associated attachments is strictly prohibited.

v-kydela commented 3 years ago

Hi @jarekgoslawski

The Stack Overflow post you linked to is actually relevant to your question because it explains how to do what you're trying to do. There's also a blog post that goes into detail about incorporating Facebook templates into Bot Framework bots: https://blog.botframework.com/2017/03/28/custom-channel-data/

Anyway, I understand now that there is not any equivalent between Facebook channel and Facebook adapter (working in fact with Workplace). I was believing that Facebook adapter can deal with something more than just text. (like Facebook channel did).

I want to clear up some confusion. "Channel" is unfortunately an ambiguous term because it can refer to the ABS-specific channel connector service (the Facebook connector in this case) or it can refer to the chat platform that the connector connects to (Facebook itself). Under that second definition, Facebook would be your channel whether you're using the adapter or the connector, though it looks like when you say channel you mean connector.

In any case, the Facebook adapter can absolutely deal with more than just text, it just can't deal with Bot Framework cards. You can see in FacebookHelper.cs that you can include Facebook templates in either an activity's channel data or its attachments. I already mentioned how to do it with channel data, and if you want to see how to do it with attachments then you can try the Facebook adapter sample.

Are there any plans to build adapter for Workplace (or add to Facebook adapter) full support of Facebook/Workplace cards/templates?

If by full support you mean the ability to automatically convert Bot Framework cards to Facebook templates like the connector does, then I don't think that's on the roadmap. But if you're asking about plans to include classes in the SDK that conform to Facebook's JSON schemas, that's already been done. You can see several classes in the Facebook adapter library that will help you build Facebook templates, like AttachmentPayload.

Anyway I still count on your support in my question about “How can I see this full request body (with encoded Json) which adapters sends to Facebook/Workplace?

When you run the bot locally then you can always use something like Fiddler to intercept all requests that your machine initiates, or you may be able to find a way to view the requests in the Visual Studio debugger.

v-kydela commented 3 years ago

@jarekgoslawski - Are you still working on this?

igordevezas commented 3 years ago

@v-kydela I am using Microsoft Bot Composer combined with FacebookAdapter and when trying to send a payload:

# MyMessage
[Activity 
        ChannelData=${json(channelData())}
]

# channelData
-```{
    "attachment":{
      "type":"template",
      "payload":{
        "template_type":"button",
        "text":"What do you want to do next?",
        "buttons":[
          {
            "type":"web_url",
            "url":"https://www.messenger.com",
            "title":"Visit Messenger"
          }
        ]
      }
    }
}```

I get:

image

Any lights here, please?

Many thanks.

v-kydela commented 3 years ago

@igordevezas - Your issue sounds a bit more complicated because of Composer. I'm guessing you got that to work by using an ejected runtime. I'm sorry but I can't help you with that here. You should ask a question on Stack Overflow instead. They have a great guide that will help you get an answer: https://stackoverflow.com/help/how-to-ask

Linking to duplicate: https://github.com/microsoft/BotFramework-Composer/issues/2705#issuecomment-754164581

jarekgoslawski commented 3 years ago

@jarekgoslawski - Are you still working on this?

yep, I'm working on it to "compensate" what adapter itself can't do . thx for your support, for me you can close this case.

igordevezas commented 3 years ago

@v-kydela tks! just raised the question