High level design and task for Chat bot

[sojharo]

In this task, I would start working on high level design and tasks for the chat bot.

[jekram]

@sojharo Let's make sure we leverage some of the work that was done before

https://github.com/Cloudkibo/KiboEngage/issues/391

@zarmeen92

[sojharo]

I reviewed the past documentations prepared by zarmeen and me. I read vantross bot documentation that we created. I also reviewed the kylie.ai in order to refresh myself on how they are doing. I have started to create design on this however it is under construction. I am still looking at case 1.

[jekram]

Thanks for the update

[sojharo]

I have created the first draft of the document. Kindly review it. There are still following areas on which I need to do more research:

1. Microsoft Bot framework
2. Integration with other chat tools (how to do them)

https://docs.google.com/document/d/10VMyO96Wlmms4KFbYV3sjU3ubmOR6MR77naZMTrps8g/edit

Here are some of our previous documents as well that discuss our approach:

https://docs.google.com/document/d/1uxhkMDLUziQdBARcPM6O9BHHgBP_nuiMwIi-ZwPjg-o/edit

https://docs.google.com/a/cloudkibo.com/document/d/1In4DIA31CDGbwemLTBP62ZZNUnLxvb0pdgpPBfM7s0c/edit?usp=drive_web

https://docs.google.com/document/d/1vA0uNVUK4lVnGeBFS4eiePj4HEVmCsUHTuwfgtz7zps/edit

[jekram]

Thanks @sojharo

@dayemsiddiqui @zarmeen92 Please also review and provide beedback

[zarmeen92]

Read the document. If we will be using API.AI then we have to think regarding how we will enable companies logged into Chatbot to create and train their own Chat bots because API.AI has not yet exposed APIs for creating Chat bots via API call

[dayemsiddiqui]

After reading the document, it appears that API.AI is very limited, in future we might also face problems if we want to add new features to our bot because of the limited set of exposed api's also it would not allow the client to separately host the bot if that is a requirement.

[jekram]

@dayemsiddiqui @zarmeen92

Please be more specific of the API limitation. By creating a table between Microsoft & Api.ai

[jekram]

@sojharo What is the update here?

The current document is the very high level we ned to get into next level of details.

In regards to voice please answer the question below:

How is voice would work with APi.Ai or Microsoft FrameWork? Voice coming from Android or iOS? Voice coming from non-Mobile phone - landlines? Voice coming from devices like Amazon Alexa?

[zarmeen92]

This is the initial comparison between Microsoft and API.AI (list is not complete yet) https://docs.google.com/spreadsheets/d/1PKW0ALIFPDPZlTJEJmNCXNQbmb71O38bQK-o9ZA45Co/edit#gid=0

How is voice would work with APi.Ai or Microsoft FrameWork? API.AI works with only Text input and generates a text output. We can use Google Speech API to convert speech to text before call API.AI API. Microsoft framework requires using Microsoft Speech API to convert speech to text before using LUIS services inside bot.

Regarding Amazon Alexa, Microsoft bot framework does not provide integration with Alexa whereas API.AI does.

[sojharo]

I would work more on the document today. I would add more details in regards to MS Bot framework.

[sojharo]

Please be more specific of the API limitation. By creating a table between Microsoft & Api.ai

Here is one table created by zarmeen to explain how the API of api.ai is limiting. I would add Microsoft comparison column in this today.

https://docs.google.com/spreadsheets/d/1G3cjHYPlF-2YShMUBgD8-8ujRQUU8QdVeK-fwPFq2ts/edit#gid=0

[sojharo]

I looked into the details of MS Bot Framework. Here is what I have found.

They are providing the bot framework SDK for nodejs and other languages. We can use them to construct our bot. It can be used to connect communication channels i.e. slack, messenger etc. It can be used to add intelligence to the bot.

In API.AI, we need to go to dashboard to enable any integration with communication channels i.e. slack, messenger. In MS Bot framework SDK, we can just do it in code. It would be easier to automatically allow client to add more integrations without depending on us to enabling them from the API.AI dashboard.

The MS Bot SDK allows to send messages, rich messages, attachments, voice messages and many more. Also the SDK will automatically reformat the message according to channel type. For example, if skype doesn't allow cards like messenger do, then SDK would automatically convert it to clickable image for skype.

https://docs.microsoft.com/en-us/bot-framework/rest-api/bot-framework-rest-connector-concepts

As far as intelligence is concerned, these services are separately provided by Microsoft in the Cloud. We can enable them and add to our bot. For example, following services are provided by Microsoft to add the intelligence to our bot.

Following are about Language understanding and are very advanced.

• The Language Understanding Intelligent Service (LUIS) is able to process natural language using pre-built or custom-trained language models.
• The Text Analytics API detects sentiment, key phrases, topics, and language from text.
• The Bing Spell Check API provides powerful spell check capabilities, and is able to recognize the difference between names, brand names, and slang.
• The Linguistic Analysis API uses advanced linguistic analysis algorithms to process text, and perform operations such as breaking down the structure of the text, or performing part-of-speech tagging and parsing.
• The Web Language Model (WebLM) API can be used to automate a variety of natural language processing tasks, such as word frequency or next-word prediction, using advanced language modeling algorithms.

Following are about Extraction of Knowledge.

• The Entity Linking Intelligence Service annotates unstructured text with the relevant entities mentioned in the text
• The Recommendations API provides "frequently bought together" recommendations to a product
• The Knowledge Exploration Service provides natural language interpretation of user queries and returns annotated interpretations to enable rich search
• The Academic Knowledge API returns academic research papers, authors, journals, conferences, topics, and universities
• The QnA Maker is a free, easy-to-use, REST API and web-based service that trains AI to respond to users’ questions in a natural, conversational way.

Among above, QnA Maker is the one that we need to integrate in our MS Bot.

Following are the about Speech:

• The Bing Speech API provides speech-to-text and text-to-speech conversion capabilities.
• The Custom Recognition Intelligent Service (CRIS) allows you to create custom speech recognition models to tailor the speech-to-text conversion to an application's vocabulary or user's speaking style.
• The Speaker Recognition API enables speaker identification and verification through voice.

Among above, we would use the Bing Speech API in our v2 of the bot. API.AI doesn't provide speech API and it works with text only.

Following are search APIs that bot can use:

• Web Search
• Image Search
• Video Search
• News Search
• Auto Suggest API

Following are the image and video understanding APIs.

• The Computer Vision API extracts rich information about images (such as objects or people)
• The Emotion API analyzes human faces and recognizes their emotion
• The Face API detects human faces, compares them to similar faces, and can even organize people into groups according to visual similarity.
• The Video API analyzes and processes video to stabilize video output, detects motion, tracks faces, and can generate a motion thumbnail summary of the video.

Features of QnA Maker API:

• Create Knowledge Database with a set of questions and answers by using a single API. API.AI doesn't allow the list of questions and answers in a single API.
• Delete the entire knowledge database
• Download Knowledge database in tsv file format
• Generate answers with confidence score
• Publish Knowledge base
• Train Knowledge base The developer of the knowledge base service can use this API to submit user feedback for tuning question-answer matching. QnA Maker uses active learning to learn from the user utterances that come on a published Knowledge base service. In this process, QnA Maker records user feedback from different users and trains the knowledge base to respond accordingly, when there are sufficient number of users sending the same feedback. Every user feedback is logged and model training is triggered when there are 50 new feedback instances. Typically, the model updates are reflected when same question-answer pair from the knowledge base is sent as feedback for a given user query by at least 20 users. Most changes are immediately reflected in both the published and the saved knowledge bases. Some new question-answer pairs are only added to the saved knowledge base and they are moved to the published version in the next knowledge base publish operation by the developer. This gives the flexibility to the developer to keep or discard the newly added question-answer pairs.
• Update the knowledge database Add or delete QnA Pairs and / or URLs to an existing knowledge base.

[sojharo]

Putting the links here for reference that I was reading yesterday:

https://docs.microsoft.com/en-us/bot-framework/rest-api/bot-framework-rest-connector-concepts

https://dev.botframework.com/

https://docs.microsoft.com/en-us/bot-framework/#pivot=main&panel=intelligence

https://docs.microsoft.com/en-us/bot-framework/cognitive-services-bot-intelligence-overview#language-understanding

https://qnamaker.ai/Create

https://westus.dev.cognitive.microsoft.com/docs/services/58994a073d9e04097c7ba6fe/operations/58994a083d9e041ad42d9bad

[jekram]

@sojharo This is a good document. However, please put this in table format. It will make it easier to compare and read. Please create a word document.

Also, please look into Google and not just API.AI. I would think that Google would provide similar API and services.

Does both the product similar interface channels (Messenger, Slack..........)?

Microsoft has LUIS for NLP and if understands NLP is part of API.AI?

How does the service compare for voice (in and out)?

How does the two product compare in terms of prebuilt libraries?

[sojharo]

I looked into the Google and have found that they are only doing API.AI and Tensor Flow. They have recently jumped into the bot frameworks competition by acquiring the API.AI. So I couldn't find that they provide all the services which Microsoft is providing. Here are the articles:

http://tech.firstpost.com/news-analysis/google-research-opens-source-for-its-machine-learning-framework-tensorflow-287362.html

https://mspoweruser.com/google-acquires-api-ai-take-microsoft-bot-platfrom/

About the services, I think they have just started to create the services as I found this news from May 17, 2017 This is the bot analytics service:

https://techcrunch.com/2017/05/17/google-chatbase-chatbot-analytics/

https://chatbase.com/welcome

It integrates with the API.AI as well.

Does both the product similar interface channels (Messenger, Slack..........)?

No, Microsoft supports less interface channels than API.AI. I have added the table in the document for comparison.

Microsoft has LUIS for NLP and if understands NLP is part of API.AI?

Microsoft LUIS is basically a separate API (service) that can integrate with our chat bot or any application to make it intelligent to understand the commands or questions. While, in API.AI, the natural language processing is integral part of it and not a separate service.

How does the service compare for voice (in and out)?

Microsoft provides the Bing Speech API which can convert the Speech to Text and Text to Speech. It is also separate service and can be integrated with bot.

https://azure.microsoft.com/en-gb/services/cognitive-services/speech/

I did chat with Azure support person regarding MS Bot framework and intelligence (cognitive) services that they provide. I have sent the chat in the email to you. Here are the integration links he gave to me:

https://docs.microsoft.com/en-us/azure/cognitive-services/

https://docs.microsoft.com/en-us/bot-framework/cognitive-services-bot-intelligence-overview

The API.AI provides voice recognition through its open source SDKs. These are made for lot of platforms. Please see the image below.

How does the two product compare in terms of prebuilt libraries?

Microsoft bot framework provides SDKs for only .NET and Node.js. However, APi.AI provides the SDKs for almost all the platforms. Here is the list of platforms for which API.AI provides the SDKs and libraries:

I have also added the comparison table in the word document:

https://docs.google.com/document/d/10VMyO96Wlmms4KFbYV3sjU3ubmOR6MR77naZMTrps8g/edit#heading=h.nz844jnjm4re

[sojharo]

I worked more on this task and went on to create an train a bot. I gave the training data to it. While working on the bot, I realised that QnA Maker of Microsoft is very limited as compared to API.AI. It didn't created the context and history. It means it won't do follow-up chat i.e. help customer in filling a form. Also, there was no option of remembering the visitor's name. Api.AI gets the name of the visitor and greets him by his name. But QnA Maker is just FAQ bot and would get simple questions and answers. Here is the trained bot in action:

https://qnamaker.ai/Edit/Test?kbId=2c0278b5-3d45-4692-a28d-b17c79992668

This is how the training data is given to bot. It is very basic. Also, this is preview mode.

I didn't get much time to convert it into widget and upload on kiboengage page. However, in order to test it. You can go to following link and use your kibodeveloper@cloudkibo.com account of azure:

https://qnamaker.ai/Edit/Test?kbId=2c0278b5-3d45-4692-a28d-b17c79992668

[jekram]

@sojharo is qnamaker part of Microsoft Bot Framework?

[sojharo]

Yes, QnA Maker is one type of bot provided by Microsoft Bot framework which answers to the questions given. We need to train it with FAQs, links, documents or manually by adding questions and answers.

[sojharo]

I worked on this task yesterday and looked into the audio options in API. Audio wasn't available through the API. But we can use their different SDKs on client side which have audio support. I started reading on Speech API as well. After this, I started work on Android SDK. That SDK from API.AI had the audio speech option. This can use the built - in speech system of android to listen to our question and convert it to text and send to API.AI for response. I have started to use that library in a sample app. This is under construction as it is having some problems when I try to run it.

[jekram]

@sojharo what is the update here? and next steps?

We had enough discussion on this topic. Please complete the design and open task so we get this done.

[sojharo]

I worked on this and was able to successfully get speech input from android and then get response to that query from api.ai. After this, I started work on doing speech from our web widget by using their javascript sdk. This part is still in progress. I would complete both design and integration of audio in web widget today.

[sojharo]

On Saturday, I completed the design document and defined the tasks for this. Furthermore, I worked to integrate audio using the SDK that they had provided. However, it was breaking our current widget and error was coming from inside of the library. It wasn't working well with the Reactjs web pack methods. I spent some more time on it tried the HTML library as well. Still the problem wasn't solved.

Here is the design document with complete design and defined tasks:

https://docs.google.com/document/d/10VMyO96Wlmms4KFbYV3sjU3ubmOR6MR77naZMTrps8g/edit#

[jekram]

@sojharo

I do not understand the design document. This is a more of a discussion document.

I need to see a block diagram of flow. What components we are using from API.AI and what components we are developing.

Please show me the logical design that we would build and what we would leverage.

I also, do not understand the task. For each task, please provide a description, and where this component would reside?

When can we meet to review this document?

[jekram]

Is our Web Widget a ReactJs component? I thought it was from KiboSopport days and pure Java Script? Anyway, what is the next step to fix this?

[sojharo]

Yes, Sir, I defined the use cases and screens that we would be having. Also for which components we would have database tables. I am adding the diagrams in it.

I would add the description for the tasks in the document. I was thinking to put description on github when I open them here. I would add on both places.

Our current web widget are reactjs component designed by zarmeen. Previously they were angularjs components. As a next step, I would try to use the Google Speech API directly as the SDK is also using it behind the scenes.

As I am working today so we can meet either today or on Friday.

[jekram]

@sojharo Please update the design document before we meet of Friday morning at 9 AM. Also, put the task description in the document.

I also do not see the picture how this will work with KiboEngage.

[sojharo]

I will work on this today. I couldn't work on this on wednesday due to my work on #14 #15

[jekram]

WhatI is the update here? I want to review this and have a meeting with you on Saturday. Update the document and ping me.

[sojharo]

Here is the updated document:

https://docs.google.com/document/d/10VMyO96Wlmms4KFbYV3sjU3ubmOR6MR77naZMTrps8g/edit#

I couldn't complete it yesterday as I was having electricity issues and also rain water was filled in streets. I am sorry for inconvenience caused.

Please let me know your thoughts on document and if anything is unclear.

[sojharo]

I have worked more on the document as discussed. I have created two diagrams and restructured the document:

https://docs.google.com/document/d/10VMyO96Wlmms4KFbYV3sjU3ubmOR6MR77naZMTrps8g/edit#heading=h.8mz773qqh38a

[sojharo]

I discussed the document with zarmeen and dayem. After this, I have identified the tasks with use cases and other categories.