cognitive-catalyst / WA-Testing-Tool

Scripts that run against Watson Assistant for K fold validation on training set, testing on blind test, and draw precision curves for comparison.
Apache License 2.0
78 stars 61 forks source link

Ability to "weight" results in blind test scoring #235

Closed andrewrfreed closed 6 months ago

andrewrfreed commented 1 year ago

I want to be able to provide a "weight" column on my input file and have it used in scoring.

An 'unweighted' input (default) looks like this. Note that weighting is only achieved by duplicating utterances, this makes the tool run more slowly and takes longer to generate a ground truth.

image

A weighted input looks like this:

image

The blind-out would look as follows:

image

1) The original weight column is carried forward (green) - note this is done by #231 2) A 'weighted score' is added.

A few updates are required: 1) blind-out should compute a weighted score. 2) The console output should give the weighted results. (Currently reads Includes 2 correct intents in 3 tries for accuracy of 0.67., should instead read Includes 8 correct intents in 10 tries for accuracy of 0.80. 3) intent-metrics CSV and treemap should generate weighted scores

The simplest implementation is to place the "weighted score" into the existing score column.

If the input file does not include a weight column (default case) then a weight of 1 can be inferred for each row.