Adaptive Card does not support speech input

saikiran521 commented 4 years ago

(Edited by @compulim to convert into a feature request)

Background

When an Adaptive Card is presented to the user, the user cannot submit the card using microphone button.

Saying "submit" to the microphone do not submit the content of the card, instead, it send a text of "submit" to the bot.

Design notes

TBC: Few different approaches to be filled in.

"What can I say?"

We should find out what the user could say on the screen (a.k.a. commands). And for those "commands", we don't send it to the bot as text, we do something else (e.g. submit button).

How does it works with Direct Line Speech?
- DLSpeech need to support "command filtering"
- For example
  - The screen has a submit button
  - The user say, "submit", it should not send it as text to the bot
  - The user say, "I don't know what to do", it should send it as text to the bot
Should we also support "what can we say?" without sending things to the bot?
- Should we chalkmark say-able things on the screen?
- How about suggested actions which are far off the screen?

"You can also say ..." bot logic

The bot can use LUIS and add an alternative way for submission. For example, in the case above, instead of finding it in the dropdown list, the end-user can say, "Los Angeles timezone". And the bot understand it is one of the dropdown.

The bot might want to update the selection in the Adaptive Card to reflect what the user submitted.

Original issue

Hi All,

I am new to speak functionality in adaptive cards. We have a scenario where a drop down has to be selected and click on submit. if user just says submit, action is being submitted but the value from the drop down is going as null.

Below is the example screenshot of the adaptive card Capture

Adaptive card JSON is :

{
    "type": "AdaptiveCard",
    "version": "1.0",
    "speak":"<s><select_timezone></s>",
    "body": [
        {
            "type": "Container",
            "items": [
                {
                    "type": "TextBlock",
                    "text": "<select_timezone>",
                    "id": "HeaderTimeZone",
                    "size": "Medium",
                    "color": "Dark",
                    "weight": "default"
                },
                {
                    "type": "Input.ChoiceSet",
                    "placeholder": "Placeholder text",
                    "id": "<DDLTimeZone>",
                    "value": "",
                    "choices": []

                }
            ]
        }
    ],
    "actions": [
        {
            "type": "Action.Submit",
            "title": "<submit>",
            "Id": "SubmitTimeZone",
            "data": {
                "action": "TimeZone"
            }
        }
    ],
    "$schema": "http://adaptivecards.io/schemas/adaptive-card.json"
}

Below is the code example for binding data to adaptive card and return adaptive card

var adaptiveCardPath = File.ReadAllText(Directory.GetCurrentDirectory() + "\\PATH\\");
StringBuilder stringBuilderJson = new StringBuilder(adaptiveCardPath);
stringBuilderJson.Replace("<select_timezone>", _responseManager.GetResponse(HourGlassResponses.SelectTimeZone).Text);
stringBuilderJson.Replace("<submit>", _responseManager.GetResponse(HourGlassResponses.Submit).Text);
stringBuilderJson.Replace("<DDLTimeZone>", "DDLTimeZone");

JObject hoursObject = JObject.Parse(stringBuilderJson.ToString());
var bodyContainer = hoursObject["body"].Where(x => x["type"].Value<string>() == "Container").ToList().First()["items"];
var choiceSet = bodyContainer.Where(x => x["type"].Value<string>() == "Input.ChoiceSet");
JArray choices = choiceSet.First()["choices"] as JArray;

foreach (TimeZoneInfo time in TimeZoneInfo.GetSystemTimeZones())
{
    choices.Add(new JObject(new JProperty("title", time.DisplayName)
        , new JProperty("value", time.Id)));
}
choiceSet.First()["value"] = choices.First()["value"];

var adaptiveCardAttachment = new Attachment()
{
    ContentType = "application/vnd.microsoft.card.adaptive",
    Content = hoursObject,
};
await stepContext.Context.SendActivityAsync(MessageFactory.Attachment(adaptiveCardAttachment, inputHint: InputHints.ExpectingInput), cancellationToken);

we are using Microsoft custom speech service and the code is as below :

window.WebChat.renderWebChat({
directLine: window.WebChat.createDirectLine({ token: 'toren' }),
userID: id,
username: id,
locale: 'en-US',
styleOptions: styleOptions,
store: store,
selectVoice: (voices, activity) =>
global_voiceModel == "BenjaminRUS" ? voices.find(({ name }) => /BenjaminRUS/iu.test(name))
: voices.find(({ name }) => /JessaRUS/iu.test(name)),
webSpeechPonyfillFactory: createPonyfillFactory()
}, document.getElementById('webchat'));
document.querySelector('#webchat > *').focus();

function createPonyfillFactory() {
const speechServicesPonyfillFactory = window.WebChat.createCognitiveServicesSpeechServicesPonyfillFactory({
authorizationToken: user.voiceToken,
region: 'northeurope',
});

return (options) => {
const speech = speechServicesPonyfillFactory(options);

return {
  SpeechGrammarList: speech.SpeechGrammarList,
  SpeechRecognition: speech.SpeechRecognition,

  speechSynthesis: global_enableSpeech ? speech.speechSynthesis : null,
  SpeechSynthesisUtterance: global_enableSpeech ? speech.SpeechSynthesisUtterance : null
};
}
}

Thanks in Advance

(Edited by @compulim for code formatting)

hcyang commented 4 years ago

hi @compulim , please take a look. Thanks.

compulim commented 4 years ago

@saikiran521 Can you use the bug template and tell us about versions?

compulim commented 4 years ago

@saikiran521 I try to repro your case by uploading an Adaptive Card JSON, and I am seeing the value from the dropdown being sent correctly to the bot.

{
    "type": "AdaptiveCard",
    "version": "1.0",
    "speak": "<s><select_timezone></s>",
    "body": [
        {
            "type": "Container",
            "items": [
                {
                    "type": "TextBlock",
                    "text": "<select_timezone>",
                    "id": "HeaderTimeZone",
                    "size": "Medium",
                    "color": "Dark",
                    "weight": "default"
                },
                {
                    "type": "Input.ChoiceSet",
                    "placeholder": "Placeholder text",
                    "id": "<DDLTimeZone>",
                    "value": "",
                    "choices": [
                        {
                            "title": "Washington",
                            "value": "WA"
                        },
                        {
                            "title": "Los Angeles",
                            "value": "LA"
                        }
                    ]
                }
            ]
        }
    ],
    "actions": [
        {
            "type": "Action.Submit",
            "title": "<submit>",
            "Id": "SubmitTimeZone",
            "data": {
                "action": "TimeZone"
            }
        }
    ],
    "$schema": "http://adaptivecards.io/schemas/adaptive-card.json"
}

saikiran521 commented 4 years ago

@compulim - thank you for your response we will try and get back to you

benbrown commented 4 years ago

@saikiran521 Did you get a chance to try this? Any new info?

saikiran521 commented 4 years ago

@compulim below is the adaptive card JSON ..

{
  "type": "AdaptiveCard",
  "version": "1.0",
  "speak": "<s>Select the Time Zone</s>",
  "body": [
    {
      "type": "Container",
      "items": [
        {
          "type": "TextBlock",
          "text": "Select the Time Zone",
          "id": "HeaderTimeZone",
          "size": "Medium",
          "color": "Dark",
          "weight": "default"
        },
        {
          "type": "Input.ChoiceSet",
          "placeholder": "Placeholder text",
          "id": "DDLTimeZone",
          "value": "Dateline Standard Time",
          "choices": [
            {
              "title": "(UTC-12:00) International Date Line West",
              "value": "Dateline Standard Time"
            },
            {
              "title": "(UTC-11:00) Coordinated Universal Time-11",
              "value": "UTC-11"
            },
            {
              "title": "(UTC-10:00) Aleutian Islands",
              "value": "Aleutian Standard Time"
            },
            {
              "title": "(UTC-10:00) Hawaii",
              "value": "Hawaiian Standard Time"
            },
            {
              "title": "(UTC-09:30) Marquesas Islands",
              "value": "Marquesas Standard Time"
            },
            {
              "title": "(UTC-09:00) Alaska",
              "value": "Alaskan Standard Time"
            },
            {
              "title": "(UTC-09:00) Coordinated Universal Time-09",
              "value": "UTC-09"
            },
            {
              "title": "(UTC-08:00) Baja California",
              "value": "Pacific Standard Time (Mexico)"
            },
            {
              "title": "(UTC-08:00) Coordinated Universal Time-08",
              "value": "UTC-08"
            },
            {
              "title": "(UTC-08:00) Pacific Time (US & Canada)",
              "value": "Pacific Standard Time"
            },
            {
              "title": "(UTC-07:00) Arizona",
              "value": "US Mountain Standard Time"
            },
            {
              "title": "(UTC-07:00) Chihuahua, La Paz, Mazatlan",
              "value": "Mountain Standard Time (Mexico)"
            },
            {
              "title": "(UTC-07:00) Mountain Time (US & Canada)",
              "value": "Mountain Standard Time"
            },
            {
              "title": "(UTC-06:00) Central America",
              "value": "Central America Standard Time"
            },
            {
              "title": "(UTC-06:00) Central Time (US & Canada)",
              "value": "Central Standard Time"
            },
            {
              "title": "(UTC-06:00) Easter Island",
              "value": "Easter Island Standard Time"
            },
            {
              "title": "(UTC-06:00) Guadalajara, Mexico City, Monterrey",
              "value": "Central Standard Time (Mexico)"
            },
            {
              "title": "(UTC-06:00) Saskatchewan",
              "value": "Canada Central Standard Time"
            },
            {
              "title": "(UTC-05:00) Bogota, Lima, Quito, Rio Branco",
              "value": "SA Pacific Standard Time"
            },
            {
              "title": "(UTC-05:00) Chetumal",
              "value": "Eastern Standard Time (Mexico)"
            },
            {
              "title": "(UTC-05:00) Eastern Time (US & Canada)",
              "value": "Eastern Standard Time"
            },
            {
              "title": "(UTC-05:00) Haiti",
              "value": "Haiti Standard Time"
            },
            {
              "title": "(UTC-05:00) Havana",
              "value": "Cuba Standard Time"
            },
            {
              "title": "(UTC-05:00) Indiana (East)",
              "value": "US Eastern Standard Time"
            },
            {
              "title": "(UTC-05:00) Turks and Caicos",
              "value": "Turks And Caicos Standard Time"
            },
            {
              "title": "(UTC-04:00) Asuncion",
              "value": "Paraguay Standard Time"
            },
            {
              "title": "(UTC-04:00) Atlantic Time (Canada)",
              "value": "Atlantic Standard Time"
            },
            {
              "title": "(UTC-04:00) Caracas",
              "value": "Venezuela Standard Time"
            },
            {
              "title": "(UTC-04:00) Cuiaba",
              "value": "Central Brazilian Standard Time"
            },
            {
              "title": "(UTC-04:00) Georgetown, La Paz, Manaus, San Juan",
              "value": "SA Western Standard Time"
            },
            {
              "title": "(UTC-04:00) Santiago",
              "value": "Pacific SA Standard Time"
            },
            {
              "title": "(UTC-03:30) Newfoundland",
              "value": "Newfoundland Standard Time"
            },
            {
              "title": "(UTC-03:00) Araguaina",
              "value": "Tocantins Standard Time"
            },
            {
              "title": "(UTC-03:00) Brasilia",
              "value": "E. South America Standard Time"
            },
            {
              "title": "(UTC-03:00) Cayenne, Fortaleza",
              "value": "SA Eastern Standard Time"
            },
            {
              "title": "(UTC-03:00) City of Buenos Aires",
              "value": "Argentina Standard Time"
            },
            {
              "title": "(UTC-03:00) Greenland",
              "value": "Greenland Standard Time"
            },
            {
              "title": "(UTC-03:00) Montevideo",
              "value": "Montevideo Standard Time"
            },
            {
              "title": "(UTC-03:00) Punta Arenas",
              "value": "Magallanes Standard Time"
            },
            {
              "title": "(UTC-03:00) Saint Pierre and Miquelon",
              "value": "Saint Pierre Standard Time"
            },
            {
              "title": "(UTC-03:00) Salvador",
              "value": "Bahia Standard Time"
            },
            {
              "title": "(UTC-02:00) Coordinated Universal Time-02",
              "value": "UTC-02"
            },
            {
              "title": "(UTC-02:00) Mid-Atlantic - Old",
              "value": "Mid-Atlantic Standard Time"
            },
            {
              "title": "(UTC-01:00) Azores",
              "value": "Azores Standard Time"
            },
            {
              "title": "(UTC-01:00) Cabo Verde Is.",
              "value": "Cape Verde Standard Time"
            },
            {
              "title": "(UTC) Coordinated Universal Time",
              "value": "UTC"
            },
            {
              "title": "(UTC+00:00) Dublin, Edinburgh, Lisbon, London",
              "value": "GMT Standard Time"
            },
            {
              "title": "(UTC+00:00) Monrovia, Reykjavik",
              "value": "Greenwich Standard Time"
            },
            {
              "title": "(UTC+00:00) Sao Tome",
              "value": "Sao Tome Standard Time"
            },
            {
              "title": "(UTC+01:00) Casablanca",
              "value": "Morocco Standard Time"
            },
            {
              "title": "(UTC+01:00) Amsterdam, Berlin, Bern, Rome, Stockholm, Vienna",
              "value": "W. Europe Standard Time"
            },
            {
              "title": "(UTC+01:00) Belgrade, Bratislava, Budapest, Ljubljana, Prague",
              "value": "Central Europe Standard Time"
            },
            {
              "title": "(UTC+01:00) Brussels, Copenhagen, Madrid, Paris",
              "value": "Romance Standard Time"
            },
            {
              "title": "(UTC+01:00) Sarajevo, Skopje, Warsaw, Zagreb",
              "value": "Central European Standard Time"
            },
            {
              "title": "(UTC+01:00) West Central Africa",
              "value": "W. Central Africa Standard Time"
            },
            {
              "title": "(UTC+02:00) Amman",
              "value": "Jordan Standard Time"
            },
            {
              "title": "(UTC+02:00) Athens, Bucharest",
              "value": "GTB Standard Time"
            },
            {
              "title": "(UTC+02:00) Beirut",
              "value": "Middle East Standard Time"
            },
            {
              "title": "(UTC+02:00) Cairo",
              "value": "Egypt Standard Time"
            },
            {
              "title": "(UTC+02:00) Chisinau",
              "value": "E. Europe Standard Time"
            },
            {
              "title": "(UTC+02:00) Damascus",
              "value": "Syria Standard Time"
            },
            {
              "title": "(UTC+02:00) Gaza, Hebron",
              "value": "West Bank Standard Time"
            },
            {
              "title": "(UTC+02:00) Harare, Pretoria",
              "value": "South Africa Standard Time"
            },
            {
              "title": "(UTC+02:00) Helsinki, Kyiv, Riga, Sofia, Tallinn, Vilnius",
              "value": "FLE Standard Time"
            },
            {
              "title": "(UTC+02:00) Jerusalem",
              "value": "Israel Standard Time"
            },
            {
              "title": "(UTC+02:00) Kaliningrad",
              "value": "Kaliningrad Standard Time"
            },
            {
              "title": "(UTC+02:00) Khartoum",
              "value": "Sudan Standard Time"
            },
            {
              "title": "(UTC+02:00) Tripoli",
              "value": "Libya Standard Time"
            },
            {
              "title": "(UTC+02:00) Windhoek",
              "value": "Namibia Standard Time"
            },
            {
              "title": "(UTC+03:00) Baghdad",
              "value": "Arabic Standard Time"
            },
            {
              "title": "(UTC+03:00) Istanbul",
              "value": "Turkey Standard Time"
            },
            {
              "title": "(UTC+03:00) Kuwait, Riyadh",
              "value": "Arab Standard Time"
            },
            {
              "title": "(UTC+03:00) Minsk",
              "value": "Belarus Standard Time"
            },
            {
              "title": "(UTC+03:00) Moscow, St. Petersburg",
              "value": "Russian Standard Time"
            },
            {
              "title": "(UTC+03:00) Nairobi",
              "value": "E. Africa Standard Time"
            },
            {
              "title": "(UTC+03:30) Tehran",
              "value": "Iran Standard Time"
            },
            {
              "title": "(UTC+04:00) Abu Dhabi, Muscat",
              "value": "Arabian Standard Time"
            },
            {
              "title": "(UTC+04:00) Astrakhan, Ulyanovsk",
              "value": "Astrakhan Standard Time"
            },
            {
              "title": "(UTC+04:00) Baku",
              "value": "Azerbaijan Standard Time"
            },
            {
              "title": "(UTC+04:00) Izhevsk, Samara",
              "value": "Russia Time Zone 3"
            },
            {
              "title": "(UTC+04:00) Port Louis",
              "value": "Mauritius Standard Time"
            },
            {
              "title": "(UTC+04:00) Saratov",
              "value": "Saratov Standard Time"
            },
            {
              "title": "(UTC+04:00) Tbilisi",
              "value": "Georgian Standard Time"
            },
            {
              "title": "(UTC+04:00) Volgograd",
              "value": "Volgograd Standard Time"
            },
            {
              "title": "(UTC+04:00) Yerevan",
              "value": "Caucasus Standard Time"
            },
            {
              "title": "(UTC+04:30) Kabul",
              "value": "Afghanistan Standard Time"
            },
            {
              "title": "(UTC+05:00) Ashgabat, Tashkent",
              "value": "West Asia Standard Time"
            },
            {
              "title": "(UTC+05:00) Ekaterinburg",
              "value": "Ekaterinburg Standard Time"
            },
            {
              "title": "(UTC+05:00) Islamabad, Karachi",
              "value": "Pakistan Standard Time"
            },
            {
              "title": "(UTC+05:00) Qyzylorda",
              "value": "Qyzylorda Standard Time"
            },
            {
              "title": "(UTC+05:30) Chennai, Kolkata, Mumbai, New Delhi",
              "value": "India Standard Time"
            },
            {
              "title": "(UTC+05:30) Sri Jayawardenepura",
              "value": "Sri Lanka Standard Time"
            },
            {
              "title": "(UTC+05:45) Kathmandu",
              "value": "Nepal Standard Time"
            },
            {
              "title": "(UTC+06:00) Astana",
              "value": "Central Asia Standard Time"
            },
            {
              "title": "(UTC+06:00) Dhaka",
              "value": "Bangladesh Standard Time"
            },
            {
              "title": "(UTC+06:00) Omsk",
              "value": "Omsk Standard Time"
            },
            {
              "title": "(UTC+06:30) Yangon (Rangoon)",
              "value": "Myanmar Standard Time"
            },
            {
              "title": "(UTC+07:00) Bangkok, Hanoi, Jakarta",
              "value": "SE Asia Standard Time"
            },
            {
              "title": "(UTC+07:00) Barnaul, Gorno-Altaysk",
              "value": "Altai Standard Time"
            },
            {
              "title": "(UTC+07:00) Hovd",
              "value": "W. Mongolia Standard Time"
            },
            {
              "title": "(UTC+07:00) Krasnoyarsk",
              "value": "North Asia Standard Time"
            },
            {
              "title": "(UTC+07:00) Novosibirsk",
              "value": "N. Central Asia Standard Time"
            },
            {
              "title": "(UTC+07:00) Tomsk",
              "value": "Tomsk Standard Time"
            },
            {
              "title": "(UTC+08:00) Beijing, Chongqing, Hong Kong, Urumqi",
              "value": "China Standard Time"
            },
            {
              "title": "(UTC+08:00) Irkutsk",
              "value": "North Asia East Standard Time"
            },
            {
              "title": "(UTC+08:00) Kuala Lumpur, Singapore",
              "value": "Singapore Standard Time"
            },
            {
              "title": "(UTC+08:00) Perth",
              "value": "W. Australia Standard Time"
            },
            {
              "title": "(UTC+08:00) Taipei",
              "value": "Taipei Standard Time"
            },
            {
              "title": "(UTC+08:00) Ulaanbaatar",
              "value": "Ulaanbaatar Standard Time"
            },
            {
              "title": "(UTC+08:45) Eucla",
              "value": "Aus Central W. Standard Time"
            },
            {
              "title": "(UTC+09:00) Chita",
              "value": "Transbaikal Standard Time"
            },
            {
              "title": "(UTC+09:00) Osaka, Sapporo, Tokyo",
              "value": "Tokyo Standard Time"
            },
            {
              "title": "(UTC+09:00) Pyongyang",
              "value": "North Korea Standard Time"
            },
            {
              "title": "(UTC+09:00) Seoul",
              "value": "Korea Standard Time"
            },
            {
              "title": "(UTC+09:00) Yakutsk",
              "value": "Yakutsk Standard Time"
            },
            {
              "title": "(UTC+09:30) Adelaide",
              "value": "Cen. Australia Standard Time"
            },
            {
              "title": "(UTC+09:30) Darwin",
              "value": "AUS Central Standard Time"
            },
            {
              "title": "(UTC+10:00) Brisbane",
              "value": "E. Australia Standard Time"
            },
            {
              "title": "(UTC+10:00) Canberra, Melbourne, Sydney",
              "value": "AUS Eastern Standard Time"
            },
            {
              "title": "(UTC+10:00) Guam, Port Moresby",
              "value": "West Pacific Standard Time"
            },
            {
              "title": "(UTC+10:00) Hobart",
              "value": "Tasmania Standard Time"
            },
            {
              "title": "(UTC+10:00) Vladivostok",
              "value": "Vladivostok Standard Time"
            },
            {
              "title": "(UTC+10:30) Lord Howe Island",
              "value": "Lord Howe Standard Time"
            },
            {
              "title": "(UTC+11:00) Bougainville Island",
              "value": "Bougainville Standard Time"
            },
            {
              "title": "(UTC+11:00) Chokurdakh",
              "value": "Russia Time Zone 10"
            },
            {
              "title": "(UTC+11:00) Magadan",
              "value": "Magadan Standard Time"
            },
            {
              "title": "(UTC+11:00) Norfolk Island",
              "value": "Norfolk Standard Time"
            },
            {
              "title": "(UTC+11:00) Sakhalin",
              "value": "Sakhalin Standard Time"
            },
            {
              "title": "(UTC+11:00) Solomon Is., New Caledonia",
              "value": "Central Pacific Standard Time"
            },
            {
              "title": "(UTC+12:00) Anadyr, Petropavlovsk-Kamchatsky",
              "value": "Russia Time Zone 11"
            },
            {
              "title": "(UTC+12:00) Auckland, Wellington",
              "value": "New Zealand Standard Time"
            },
            {
              "title": "(UTC+12:00) Coordinated Universal Time+12",
              "value": "UTC+12"
            },
            {
              "title": "(UTC+12:00) Fiji",
              "value": "Fiji Standard Time"
            },
            {
              "title": "(UTC+12:00) Petropavlovsk-Kamchatsky - Old",
              "value": "Kamchatka Standard Time"
            },
            {
              "title": "(UTC+12:45) Chatham Islands",
              "value": "Chatham Islands Standard Time"
            },
            {
              "title": "(UTC+13:00) Coordinated Universal Time+13",
              "value": "UTC+13"
            },
            {
              "title": "(UTC+13:00) Nuku'alofa",
              "value": "Tonga Standard Time"
            },
            {
              "title": "(UTC+13:00) Samoa",
              "value": "Samoa Standard Time"
            },
            {
              "title": "(UTC+14:00) Kiritimati Island",
              "value": "Line Islands Standard Time"
            }
          ]
        }
      ]
    }
  ],
  "actions": [
    {
      "type": "Action.Submit",
      "title": "Submit",
      "Id": "SubmitTimeZone",
      "data": {
        "action": "TimeZone"
      }
    }
  ],
  "$schema": "http://adaptivecards.io/schemas/adaptive-card.json"
}

while I am trying to submit through voice .. I am getting activity.text as "submit" ., not getting the value of adaptive card

(Edited by @compulim for code formatting)

tracyboehrer commented 4 years ago

@saikiran521 Did you determine the cause of the problem? Do you still need our help?

saikiran521 commented 4 years ago

@tracyboehrer I am still facing the issue. Help for this scenario handling will be appreciated

compulim commented 4 years ago

Now I understand what you mean.

Unfortunately, currently Adaptive Cards doesn't support speech input. Thus, if the user say "submit", it don't tap the submit button for them. It just send "submit" as text to the bot.

Let me convert this bug into a feature request. (And please vote on this one, to make this higher priority during our planning)

microsoft / BotFramework-WebChat