I want to be able to provide a "weight" column on my input file and have it used in scoring.
An 'unweighted' input (default) looks like this. Note that weighting is only achieved by duplicating utterances, this makes the tool run more slowly and takes longer to generate a ground truth.
A weighted input looks like this:
The blind-out would look as follows:
1) The original weight column is carried forward (green) - note this is done by #231
2) A 'weighted score' is added.
A few updates are required:
1) blind-out should compute a weighted score.
2) The console output should give the weighted results. (Currently reads Includes 2 correct intents in 3 tries for accuracy of 0.67., should instead read Includes 8 correct intents in 10 tries for accuracy of 0.80.
3) intent-metrics CSV and treemap should generate weighted scores
The simplest implementation is to place the "weighted score" into the existing score column.
If the input file does not include a weight column (default case) then a weight of 1 can be inferred for each row.
I want to be able to provide a "weight" column on my input file and have it used in scoring.
An 'unweighted' input (default) looks like this. Note that weighting is only achieved by duplicating utterances, this makes the tool run more slowly and takes longer to generate a ground truth.
A weighted input looks like this:
The
blind-out
would look as follows:1) The original
weight
column is carried forward (green) - note this is done by #231 2) A 'weighted score' is added.A few updates are required: 1)
blind-out
should compute a weighted score. 2) The console output should give the weighted results. (Currently readsIncludes 2 correct intents in 3 tries for accuracy of 0.67.
, should instead readIncludes 8 correct intents in 10 tries for accuracy of 0.80.
3)intent-metrics
CSV and treemap should generate weighted scoresThe simplest implementation is to place the "weighted score" into the existing
score
column.If the input file does not include a
weight
column (default case) then a weight of 1 can be inferred for each row.