Dispatch Batch Testing Inconsistency

[henryjenkins]

Tool

Name: Dispatch Version: 1.5.8 OS: MacOS 10.14.6 (Mojave)

Describe the bug

There's two available methods of batch testing. Via the UI in the LUIS portal, and via the Dispatch CLI tool.

When batch testing a Dispatch model on the LUIS portal, the expected intent is asserted against the actual intent from the Dispatch model.

When batch testing a Dispatch model using the Dispatch CLI tool, the expected intent is asserted against the actual intent from a connected service (a child LUIS model).

For example, I have a test case of two dummy LUIS models called test-0 and test-1, and test-0 contains an intent for MakeBooking. Batch testing the Dispatch model via the LUIS portal means the asserted intent is l_test-0. Batch testing the Dispatch model via the Dispatch CLI means the asserted intent is MakeBooking (from the intent within the connected service test-0.

It also means that the generated JSON file after running the batch test from the CLI is not usable in the LUIS portal, because it contains the intents for the connected service rather than the Dispatch model.

To Reproduce

Steps to reproduce the behavior:

1. Create 2 or more LUIS models
2. Create a Dispatch model from the LUIS models
3. Prepare a test file for both the CLI (TSV) and the LUIS portal (JSON)
4. Run each of the batch tests
5. Note the difference in the intents (Dispatch vs connected service)
6. Note the generated JSON after running the batch test from the CLI can't be used as-is because it uses the intents from the connected LUIS model rather than the Dispatch model

Expected behavior

Ideally, the batch testing behavior would be consistent whether used from the LUIS portal or the Dispatch CLI. Perhaps the CLI could have a flag to assert against the connected service or not.

[bug]

[henryjenkins]

Apologies, I just realised that the --doTopLevelPredictionOnly covers this.