Open rozele opened 4 years ago
There are a few benefits to this approach:
Acceptance Criteria
dotnet nlu test
command returns verbatim results from NLU providercompare
command CLI options or in a data envelope) should support generic utterances and LUIS batch formats. If a parser is specified, we should always support falling back on the default parsing behavior.test
command should still exist.compare
command.Examples If we use a CLI option:
dotnet nlu test -s luis -u tests.json -o results.json
dotnet nlu compare -s luis -e tests.json -a results.json
The tests.json
file may contain generic utterances format, whereas the results.json
file will contain raw LUIS responses.
If we use the envelope method:
dotnet nlu test -s luis -u tests.json -o results.json
dotnet nlu compare -e tests.json -a results.json
The tests.json
file may contain generic utterances format, whereas the results.json
file will contain raw LUIS responses embedded in an envelope, e.g.:
{
"format": "luis",
"utterances": [
{
/* raw LUIS response */
}
]
}
Today, we expect that the utterances JSON file is always an array of utterances with entities in one of two formats, an NLU.DevOps generic format:
Or LUIS batch format:
I suspect we can make this a bit simpler and afford an opportunity to leverage other tooling (that is less likely to get out of sync) if we allow dependency injection of the parser for utterances. One potential scenario I'd like to unblock is I'd like to write a simple script that takes a test utterance JSON file and sends the utterances off for prediction against LUIS / Lex / etc., storing the unmodified results directly from LUIS / Lex back in a JSON array.
I.e., could we easily enable something like this:
We could achieve this with a couple different options.
Option 1, we add some flags to the
compare
command for how to inject the parser:Option 2, we add an optional envelope to the utterances JSON file: