Closed rjgmail88 closed 6 years ago
SeanSobey
commented
6 years ago There are some libraries available, like botbuilder-unit and botbuilder-calling-test to assist with integration testing. Use something like nock to intercept external calls to eg LUIS.
From what I understand the new v4 emulator will allow recording and automated playback of transcripts which will make functional testing much easier.
rjgmail88
commented
6 years ago Thank you.
EricDahlvang
commented
6 years ago Closing, since @SeanSobey provided a great answer. (Thanks Sean!)
rjgmail88
commented
6 years ago I wanted to confirm one thing but, issue got closed immediately. As far what I can see here https://github.com/Microsoft/botbuilder-tools/tree/master/Chatdown
transcripts will be used to design a bot. and not necessarily go through BOT's business logic and Dialog flow/conversation flow.
Framing the scenarios supported by the bot.
Business decision makers to review, provide feedback.
Defining a "happy path" (as well as other paths) through conversational flows between a user and a bot.
.chat file format helps you create mockups of conversations between a user and a bot. Chatdown CLI tool converts
.chat files into conversation transcripts (.transcript files) that can be viewed in the Bot Framework Emulator
In this case one would not consider them for functional testing but for designing purpose.
SeanSobey
commented
6 years ago The new emulator appears to have a cli package that will eventually allow automated running of those transcripts. Since the emulator runs a local Direct Line server to connect with the bot this will be the most viable option to test a bot through it's API without deploying first. Other approaches until now either require lower level testing of the bot directly through the Console connector, or unofficial packages to try a local Direct Line connection.
The new emulator should allow running scripted scenarios against a bots REST API running locally, perfect for integration tests eg on a CI server.
To actually have automated testing of channels like Skype and Messager I think the best option is an automated browser via Selenium or Puppeteer to control the web version of those channels. This will however require a deployed bot to test against.
rjgmail88
commented
6 years ago Thanks. So looks like new emmulator will have some capabilities soon. Right? Final thing I would to ask you is about this library & get your thoughts on it.
SeanSobey
commented
6 years ago Seems decent enough, but as I mentioned currently it's quite tricky to get Direct Line connections working on localhost. So that library will only work against a published bot via direct line, not that ideal for all forms of testing.
On the current project I am working on, our testing strategy has been unit testing and a few integration tests per dialog connecting via ConsoleConnector using a custom test harness.
The plan going forward is to keep on unit testing, migrate the current integration tests onto botbuilder-unit testing all Dialogs and Libraries, with nock to mock API calls. Then once the emulator comes out with localhost testing capabilities we will create a end-to-end testing project to test the whole bot project using a few scripted scenarios and most likely mock API servers for eg LUIS using something like json-server. All those tests will run on a CI server as part of the build. We will also investigate functional testing for specific channels using webdriverjs or puppeteer. These tests will run post deploy on a staging environment as part of acceptance testing.
Hopefully that helps and gives you some options
rjgmail88
commented
6 years ago @SeanSobey, thank you. This is great info and helps me to make some decisions on my project.
Bot Info
Issue Description
I am using Mocha do to unit testing for BOT. But, I am not sure how to test bot conversational flow etc. I have multiple cannel so each time I change the code I need to make sure all the dialogs works fine for all the channel. I think something like which will test each response from BOT when user inputs something. I am using LUIS( 3 intents ) , QnA as well so need to make sure bot responses are right with correct intent recognition.