search

codeforequity-at / botium-core

The Selenium for Chatbots - Bots Testing Bots
https://www.botium.ai
MIT License
229 stars 64 forks source link

Suggestion: add support for Google assistant responses #78

Closed iMoses closed 5 years ago

iMoses commented 6 years ago

I'm using the emulator on verbose mode and although botium gets a responses it doesn't say anything.

I assume the reason is that I'm using the actions-on-google package on a fulfillment webhook that returns a payload meant for Google assistant.

I suggest adding support to cases in which the dialogflow conversation data appears as part of the payload in a actions-on-google format.

Example response:

find me tables
  botium-DialogflowContainer dialogflow response: { responseId: '1d244403-d526-49ef-9aee-c8ef49fdc9e7',
  botium-DialogflowContainer   queryResult: 
  botium-DialogflowContainer    { fulfillmentMessages: [ [Object] ],
  botium-DialogflowContainer      outputContexts: [ [Object], [Object], [Object] ],
  botium-DialogflowContainer      queryText: 'find me tables',
  botium-DialogflowContainer      speechRecognitionConfidence: 0,
  botium-DialogflowContainer      action: 'search',
  botium-DialogflowContainer      parameters: { fields: [Object] },
  botium-DialogflowContainer      allRequiredParamsPresent: true,
  botium-DialogflowContainer      fulfillmentText: '',
  botium-DialogflowContainer      webhookSource: '',
  botium-DialogflowContainer      webhookPayload: { fields: [Object] },
  botium-DialogflowContainer      intent: 
  botium-DialogflowContainer       { inputContextNames: [],
  botium-DialogflowContainer         events: [],
  botium-DialogflowContainer         trainingPhrases: [],
  botium-DialogflowContainer         outputContexts: [],
  botium-DialogflowContainer         parameters: [],
  botium-DialogflowContainer         messages: [],
  botium-DialogflowContainer         defaultResponsePlatforms: [],
  botium-DialogflowContainer         followupIntentInfo: [],
  botium-DialogflowContainer         name: 'projects/project-id/agent/intents/...',
  botium-DialogflowContainer         displayName: 'search',
  botium-DialogflowContainer         priority: 0,
  botium-DialogflowContainer         isFallback: false,
  botium-DialogflowContainer         mlEnabled: false,
  botium-DialogflowContainer         webhookState: 'WEBHOOK_STATE_UNSPECIFIED',
  botium-DialogflowContainer         action: '',
  botium-DialogflowContainer         resetContexts: false,
  botium-DialogflowContainer         rootFollowupIntentName: '',
  botium-DialogflowContainer         parentFollowupIntentName: '' },
  botium-DialogflowContainer      intentDetectionConfidence: 1,
  botium-DialogflowContainer      diagnosticInfo: { fields: [Object] },
  botium-DialogflowContainer      languageCode: 'en-us' },
  botium-DialogflowContainer   webhookStatus: { details: [], code: 0, message: 'Webhook execution successful' } } +2m
codeforequity-at commented 6 years ago

Agreed, would be very useful..

iMoses commented 6 years ago

I've created a workaround on my private repository.

A middleware that fills the fulfillmentText with the textToSpeech of the first simpleResponse it finds. 

app.use(conv => {
    const { handler } = conv;
    conv.handler = async function() {
        const res = await handler.apply(this, arguments);
        const { simpleResponse } = res.body.payload.google.richResponse.items.find(item => item.simpleResponse) || {};
        if (simpleResponse) res.body.fulfillmentText = simpleResponse.textToSpeech;
        return res;
    };
});

Do you think this is a stable enough approach?

Would you like me to create a pull request for a solution such as this?

codeforequity-at commented 6 years ago

I currently have no idea where this middleware would fit with Botium - so I suggest you create a pull request and we work on it

frankgerhardt commented 6 years ago

Good support for DialogFlow and Google Assistant is the first thing I look for, after Facebook Messenger.

I'm a bit confused now to see some things working partially with DialogFlow but not seeing it mentioned on the website anywhere.

mahmeds7777 commented 6 years ago

Any update as to whether this has been implemented yet, or any functional workarounds? Appreciate it

codeforequity-at commented 6 years ago

has not yet been implemented. it is in the backlog and will be implemented for sure, as lots of users asking for it.

mahmeds7777 commented 6 years ago

thanks codeforequity-at, appreciate your feedback. any idea of how to implement a quick work-around for it for the time being? as we're looking to build our test suit this week on dialoguflow / google assistant

honkeat commented 6 years ago

Encounter this as well. Agreed this would be very useful. There are responses from the fulfillment but botium is not displaying anything. The rest is pretty easy to setup and use.

codeforequity-at commented 6 years ago

With the latest build of the Botium Dialogflow connector it is possible to filter the fulfillmentMessages for the platform you are targeting.

mahmeds7777 commented 5 years ago

@codeforequity-at How do I filter the fulfillmentMessages for Dialogflow/AoG, I'm using the emulator and it is currently returning the name of the Intent:

image

codeforequity-at commented 5 years ago

Is the capability "DIALOGFLOW_USE_INTENT" set ? in this case, Botium always outputs the detected intent instead of the fulfillment text.

mahmeds7777 commented 5 years ago

@codeforequity-at Yes, "DIALOGFLOW_USE_INTENT" is set to true in the root folder in botium.json. I tried setting the aforesaid property to false and/or removing it, but doing so results in no output. What am I missing? Appreciate your feedback

image

codeforequity-at commented 5 years ago

Please enable debug mode and post output and your botium.json file content here (pls remove secret keys etc first).

mahmeds7777 commented 5 years ago

@codeforequity-at Sure thing! Requested data:

Debug mode output:

botium-connector-dialogflow "query": { botium-connector-dialogflow "structValue": { botium-connector-dialogflow "fields": { botium-connector-dialogflow "keyword": { botium-connector-dialogflow "stringValue": "Banana", botium-connector-dialogflow "kind": "stringValue" botium-connector-dialogflow } botium-connector-dialogflow } botium-connector-dialogflow }, botium-connector-dialogflow "kind": "structValue" botium-connector-dialogflow }, botium-connector-dialogflow "any": { botium-connector-dialogflow "stringValue": "", botium-connector-dialogflow "kind": "stringValue" botium-connector-dialogflow }, botium-connector-dialogflow "command": { botium-connector-dialogflow "stringValue": "Add", botium-connector-dialogflow "kind": "stringValue" botium-connector-dialogflow } botium-connector-dialogflow } botium-connector-dialogflow }, botium-connector-dialogflow "allRequiredParamsPresent": true, botium-connector-dialogflow "fulfillmentText": "", botium-connector-dialogflow "webhookSource": "", botium-connector-dialogflow "webhookPayload": null, botium-connector-dialogflow "intent": { botium-connector-dialogflow "inputContextNames": [], botium-connector-dialogflow "events": [], botium-connector-dialogflow "trainingPhrases": [], botium-connector-dialogflow "outputContexts": [], botium-connector-dialogflow "parameters": [], botium-connector-dialogflow "messages": [], botium-connector-dialogflow "defaultResponsePlatforms": [], botium-connector-dialogflow "followupIntentInfo": [], botium-connector-dialogflow "name": "*REDACTED**", botium-connector-dialogflow "displayName": "ProductSearch", botium-connector-dialogflow "priority": 0, botium-connector-dialogflow "isFallback": false, botium-connector-dialogflow "webhookState": "WEBHOOK_STATE_UNSPECIFIED", botium-connector-dialogflow "action": "", botium-connector-dialogflow "resetContexts": false, botium-connector-dialogflow "rootFollowupIntentName": "", botium-connector-dialogflow "parentFollowupIntentName": "", botium-connector-dialogflow "mlDisabled": false botium-connector-dialogflow }, botium-connector-dialogflow "intentDetectionConfidence": 1, botium-connector-dialogflow "diagnosticInfo": { botium-connector-dialogflow "fields": { botium-connector-dialogflow "webhook_latency_ms": { botium-connector-dialogflow "numberValue": 91, botium-connector-dialogflow "kind": "numberValue" botium-connector-dialogflow } botium-connector-dialogflow } botium-connector-dialogflow }, botium-connector-dialogflow "languageCode": "en-us" botium-connector-dialogflow }, botium-connector-dialogflow "webhookStatus": { botium-connector-dialogflow "details": [], botium-connector-dialogflow "code": 5, botium-connector-dialogflow "message": "Webhook call failed. Error: 404 Not Found" botium-connector-dialogflow } botium-connector-dialogflow } +499ms

botium.json:

{ "botium": { "Capabilities": { "PROJECTNAME": "REDACTED", "CONTAINERMODE": "dialogflow", "DIALOGFLOW_PROJECT_ID": "REDACTED", "DIALOGFLOW_CLIENT_EMAIL": "REDACTED", "DIALOGFLOW_PRIVATE_KEY": "-----BEGIN PRIVATE KEY-----REDACTED\n-----END PRIVATE KEY-----\n" } } }

codeforequity-at commented 5 years ago

In this case you should find out what this webhook error 404 means

mahmeds7777 notifications@github.com schrieb am Di., 30. Okt. 2018, 19:12:

@codeforequity-at https://github.com/codeforequity-at Sure thing! Requested data:

Debug mode output:

botium-connector-dialogflow "query": { botium-connector-dialogflow "structValue": { botium-connector-dialogflow "fields": { botium-connector-dialogflow "keyword": { botium-connector-dialogflow "stringValue": "Banana", botium-connector-dialogflow "kind": "stringValue" botium-connector-dialogflow } botium-connector-dialogflow } botium-connector-dialogflow }, botium-connector-dialogflow "kind": "structValue" botium-connector-dialogflow }, botium-connector-dialogflow "any": { botium-connector-dialogflow "stringValue": "", botium-connector-dialogflow "kind": "stringValue" botium-connector-dialogflow }, botium-connector-dialogflow "command": { botium-connector-dialogflow "stringValue": "Add", botium-connector-dialogflow "kind": "stringValue" botium-connector-dialogflow } botium-connector-dialogflow } botium-connector-dialogflow }, botium-connector-dialogflow "allRequiredParamsPresent": true, botium-connector-dialogflow "fulfillmentText": "", botium-connector-dialogflow "webhookSource": "", botium-connector-dialogflow "webhookPayload": null, botium-connector-dialogflow "intent": { botium-connector-dialogflow "inputContextNames": [], botium-connector-dialogflow "events": [], botium-connector-dialogflow "trainingPhrases": [], botium-connector-dialogflow "outputContexts": [], botium-connector-dialogflow "parameters": [], botium-connector-dialogflow "messages": [], botium-connector-dialogflow "defaultResponsePlatforms": [], botium-connector-dialogflow "followupIntentInfo": [], botium-connector-dialogflow "name": "*REDACTED**", botium-connector-dialogflow "displayName": "ProductSearch", botium-connector-dialogflow "priority": 0, botium-connector-dialogflow "isFallback": false, botium-connector-dialogflow "webhookState": "WEBHOOK_STATE_UNSPECIFIED", botium-connector-dialogflow "action": "", botium-connector-dialogflow "resetContexts": false, botium-connector-dialogflow "rootFollowupIntentName": "", botium-connector-dialogflow "parentFollowupIntentName": "", botium-connector-dialogflow "mlDisabled": false botium-connector-dialogflow }, botium-connector-dialogflow "intentDetectionConfidence": 1, botium-connector-dialogflow "diagnosticInfo": { botium-connector-dialogflow "fields": { botium-connector-dialogflow "webhook_latency_ms": { botium-connector-dialogflow "numberValue": 91, botium-connector-dialogflow "kind": "numberValue" botium-connector-dialogflow } botium-connector-dialogflow } botium-connector-dialogflow }, botium-connector-dialogflow "languageCode": "en-us" botium-connector-dialogflow }, botium-connector-dialogflow "webhookStatus": { botium-connector-dialogflow "details": [], botium-connector-dialogflow "code": 5, botium-connector-dialogflow "message": "Webhook call failed. Error: 404 Not Found" botium-connector-dialogflow } botium-connector-dialogflow } +499ms

botium.json:

{ "botium": { "Capabilities": { "PROJECTNAME": "REDACTED", "CONTAINERMODE": "dialogflow", "DIALOGFLOW_PROJECT_ID": "REDACTED", "DIALOGFLOW_CLIENT_EMAIL": "REDACTED", "DIALOGFLOW_PRIVATE_KEY": "-----BEGIN PRIVATE KEY-----REDACTED\n-----END PRIVATE KEY-----\n" } } }

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/codeforequity-at/botium-core/issues/78#issuecomment-434408500, or mute the thread https://github.com/notifications/unsubscribe-auth/AZEdT7bbro4Lx3tmNN__ZvaIi8GXhQMQks5uqJaBgaJpZM4VBAb7 .

mahmeds7777 commented 5 years ago

@codeforequity-at Thank you. Was an issue with the fulfillment URL, the emulator is functioning as expected now. Any chance we can also test GUI elements like rich text responses via Dialogflow or AoG?

codeforequity-at commented 5 years ago

Yes, there is a chance to test GUI elements. With the latest builds Botium supports validation of Buttons, Pictures and Card attachments for most connectors (Dialogflow connector is among them). Currently I cannot provide documentation or tutorials for it, the pattern is similar to this convo file:

#bot
Hallo und willkommen
BUTTONS Button 1 | One more button
MEDIA myimage.png

So in addition to validating the text response from the bot, you can also check if the bot shows a button or a picture.

mahmeds7777 commented 5 years ago

Thank you @codeforequity-at I finally had a chance to give the workflow a test run, but ended up getting the following response from Botium: hello BOT SAYS (default): Cannot display response in Dialogflow simulator. Please test on the Google Assistant simulator instead.

What is this message, and do we display the proper response? Appreciate it.

codeforequity-at commented 5 years ago

Hi, please, enable debug mode and post the output, thanks.

mahmeds7777 commented 5 years ago

Hi @codeforequity-at Thank you for your patience, attached is the debug output:

botium-connector-dialogflow UserSays called +4s   botium-connector-dialogflow dialogflow request: {   botium-connector-dialogflow   "session": “REDACTED”,   botium-connector-dialogflow   "queryInput": {   botium-connector-dialogflow     "text": {   botium-connector-dialogflow       "text": "hello",   botium-connector-dialogflow       "languageCode": "en-US"   botium-connector-dialogflow     }   botium-connector-dialogflow   },   botium-connector-dialogflow   "queryParams": null   botium-connector-dialogflow } +1ms botium-connector-dialogflow                             "kind": "listValue"   botium-connector-dialogflow                           },   botium-connector-dialogflow                           "@type": {   botium-connector-dialogflow                             "stringValue": "type.googleapis.com/google.actions.v2.PermissionValueSpec",   botium-connector-dialogflow                             "kind": "stringValue"   botium-connector-dialogflow                           }   botium-connector-dialogflow                         }   botium-connector-dialogflow                       },   botium-connector-dialogflow                       "kind": "structValue"   botium-connector-dialogflow                     }   botium-connector-dialogflow                   }   botium-connector-dialogflow                 },   botium-connector-dialogflow                 "kind": "structValue"   botium-connector-dialogflow               },   botium-connector-dialogflow               "userStorage": {   botium-connector-dialogflow                 "stringValue": "{\"data\":{\"id\":\"OGU3OTY5NWItZTEyZC00NGU2LTllMjgtZWU1NTM4NWY5ZDNm\"}}",   botium-connector-dialogflow                 "kind": "stringValue"   botium-connector-dialogflow               }   botium-connector-dialogflow             }   botium-connector-dialogflow           },   botium-connector-dialogflow           "kind": "structValue"   botium-connector-dialogflow         }   botium-connector-dialogflow       }   botium-connector-dialogflow     },   botium-connector-dialogflow     "intent": {   botium-connector-dialogflow       "inputContextNames": [],   botium-connector-dialogflow       "events": [],   botium-connector-dialogflow       "trainingPhrases": [],   botium-connector-dialogflow       "outputContexts": [],   botium-connector-dialogflow       "parameters": [],   botium-connector-dialogflow       "messages": [],   botium-connector-dialogflow       "defaultResponsePlatforms": [],   botium-connector-dialogflow       "followupIntentInfo": [],   botium-connector-dialogflow       "name": "REDACTED",   botium-connector-dialogflow       "displayName": "Welcome",   botium-connector-dialogflow       "priority": 0,   botium-connector-dialogflow       "isFallback": false,   botium-connector-dialogflow       "webhookState": "WEBHOOK_STATE_UNSPECIFIED",   botium-connector-dialogflow       "action": "",   botium-connector-dialogflow       "resetContexts": false,   botium-connector-dialogflow       "rootFollowupIntentName": "",   botium-connector-dialogflow       "parentFollowupIntentName": "",   botium-connector-dialogflow       "mlDisabled": false   botium-connector-dialogflow     },   botium-connector-dialogflow     "intentDetectionConfidence": 1,   botium-connector-dialogflow     "diagnosticInfo": {   botium-connector-dialogflow       "fields": {   botium-connector-dialogflow         "webhook_latency_ms": {   botium-connector-dialogflow           "numberValue": 1367,   botium-connector-dialogflow           "kind": "numberValue"   botium-connector-dialogflow         }   botium-connector-dialogflow       }   botium-connector-dialogflow     },   botium-connector-dialogflow     "languageCode": "en-us"   botium-connector-dialogflow   },   botium-connector-dialogflow   "webhookStatus": {   botium-connector-dialogflow     "details": [],   botium-connector-dialogflow     "code": 0,   botium-connector-dialogflow     "message": "Webhook execution successful"   botium-connector-dialogflow   }   botium-connector-dialogflow } +3s BOT SAYS (default): Cannot display response in Dialogflow simulator. Please test on the Google Assistant simulator instead.

codeforequity-at commented 5 years ago

Not possible with dialogflow connector. Working on Google Assistant connector.

ujja commented 5 years ago

https://www.npmjs.com/package/botium-connector-google-assistant