search

microsoft / botframework-sdk

Bot Framework provides the most comprehensive experience for building conversation applications.
MIT License
7.45k stars 2.44k forks source link

Regression Testing refinement #5314

Closed jwiley84 closed 5 years ago

jwiley84 commented 5 years ago

Up to now, my regression testing has consisted of:

1) Do the new packages load? 2) Does the bot respond in some way or does it say (Sorry, something went wrong!)?

With my testing scripts getting more and more streamlined, it has allowed me to quicken the pace of regression testing, and now we have the opportunity to begin adding more refined testing criteria to the regression tests.

Most of the sample bots between the two SDKs match feature for feature. This makes creating a "feature checklist" relatively easy to begin with. Using the samples as an outline, here are the features that get checked every time:

FEATURE Bots used Notes
Console Using console bot  
Simple dialog flow Echo bot, all the rest of the bots  
Simple state management Echobot with Counter bot  
Welcome Messages Welcome User bot  
Prompts Simple Prompts, MultiTurn Prompt, Prompt Validation bots  
Standard Cards (non-Adaptive) Using Cards bot Plus about 6 other bots with welcome cards
Adaptive Cards Adaptive Cards, Using Cards bots  
Card Actions Suggested Actions Bot  
Regex Checking Message Routing Bot  
Prompt Retries Prompt Validation Bot  
QnA Maker QnaMaker, Basic, QnA With AppInsights, Café-Bot bots  
LUIS NLP with LUIS, Basic, NLP with Dispatch, LUIS with App Insights, Café-Bot bots  
Dispatch NLP with Dispatch bot  
Attachments Handling Attachments bot  
Proactive Messages Proactive Messages Bot  
Middleware Multilingual, Console bots  
Translation Multilingual bot  
Authentication Bot-Auth, Bot-Auth-MSGraph bots This test requires either tunneling or deployment
Complex dialog flow Custom Dialogs, Message Routing bot  
Application Insights QnA With AppInsights, LUIS with App Insights  
State Storage Conversation History, Scaleout bot C# only
Facebook-specific events Facebook events bot  This test requires either tunneling or deployment
Transcription Logging Logger, Tranx Logger bots JS only
ASP-MVC styling ASP-MVC bot C# only
Cortana Skills DiceRoller Skill JS only, I don't have a Cortana connected bot, so this bot only has it's logic checked, not it's connectivity to Cortana. Going forward, I'll look for a way to make a Cortana bot so I can test this as well

This is what gets checked every time I run a regression test.

Going forward, I'm going to be adding a few more feature to test:

Feature Bot Notes
Teams Café bot or Enterprise bot Café bot has the most features, so deploying it to a resource group that has Teams configured will be easy enough
Enterprise bot Enterprise bot Testing the functions of the VSIX templates will be easy to integrate into my testing

So this is me, opening the floor for requests and input for testing.

jwiley84 commented 5 years ago

https://github.com/Microsoft/BotBuilder/issues/5318

Closing this as I created a template linked above.