search

microsoft / botbuilder-tools

Welcome to the Bot Framework Tools repository, which is the home for a set of tools for developers building bots with the Microsoft Bot Framework
https://github.com/Microsoft/botframework
MIT License
444 stars 262 forks source link

Batch Testing Publishes Models #1374

Closed henryjenkins closed 5 years ago

henryjenkins commented 5 years ago

Tool

Name: Dispatch Version: 1.5.8 OS: MacOS 10.14.6 (Mojave)

Describe the bug

When running a batch test, it has unintented consequences - it always publishes the Dispatch model and any connected services to their production slots.

To Reproduce

Steps to reproduce the behavior:

  1. Create 2 or more LUIS models
  2. Create a dispatch model using the LUIS models
  3. Update and publish new versions of the models to the staging slots (leaving the older versions in production)
  4. Prepare a test file for batch testing (TSV)
  5. Run dispatch test against the dispatch model

When running a test against a Dispatch model that is not in production, using connected services that are not in production (either the staging slot or any other version), then as a result of the test the Dispatch model and all connected services are always published to the production slot.

Expected behavior

Since this is a testing tool it is not expected that running a test would modify, in any way, our Dispatch model or any connected services. It would be expected that the test would use the provided configuration (.dispatch) file, and test against a given slot (production, staging, and perhaps even active versions). Unfortunately, the test always publishes everything to production and there is no method of choosing a slot to test against.

We are working around this by using the CLI tools to duplicate our LUIS models and add them to a new Dispatch model. This duplicated configuration is used for the duration of the test, and torn down afterwards.

[bug]

dmvtech commented 5 years ago

@henryjenkins I will attempt to see if I can reproduce. I see that dispatch v1.5.9 is out. Is it possible for you to see if you can reproduce with that version?

dmvtech commented 5 years ago

I'm not able to reproduce this yet. Having issues running the dispatch test command. I'm getting an error LabelIndexMap does not contain the key 'expected intent'

@tsuwandy Any thoughts on this one?

henryjenkins commented 5 years ago

Thanks, unfortunately somthings come up so I won't be able to look into it today but I'll try the new version as soon as I can.

tsuwandy commented 5 years ago

@henryjenkins, dispatch 1.6.0 is published to npm with this fixes. Let us know if you are still seeing issues.