hallapmark
commented
1 year ago Not exactly, but I am also not sure which level of abstraction we are talking about. I think it might help to distinguish between the labels and configurations used in the background in the abx2 code and the labels that are visible to the user of Benchmarks.
Publicly/Benchmark user visible labels (proposal, probably Ewan would be the stakeholder to say what the best names are) and corresponding abx2 configurations :
Context:
- on-triphone (abx1-style) evaluation ⇒ will use triphone item file and run abx2-code within-context
- within-context evaluation ⇒ will use phoneme item file and run abx2 within-context
- any-context evaluation (alternatively we have also called it without-context) ⇒ will use phoneme item file and run abx2 any-context
All of these will run speaker_mode=all, i.e. there are separate scores for within speaker and across speaker for all these options.
Explanation: I think the source of the sort of taxonomic difficulty is that from the abx code's standpoint, the on-triphone evaluation is also "within context", it is just that what is extracted are triphones along with the context, i.e. preceding and following phoneme, included (abc, where a and c are the fixed context). Whereas in the within-context on-phoneme evaluation, a and c are still the fixed context, but only b is extracted. This difference between the two "within" context options is caused only by a difference in the timestamps in the item file. Literally, the old abx1 code could run on-phoneme within context and on-triphone within context, if you make two calls with corresponding item files. Whereas for the any-context condition code changes were needed.
I do not know how best to add a further layer of categorization and give the Benchmark user the option to run multiple of these at the same time. One option would be:
-
on-triphone AKA abx1-style evaluation ⇒ run the on-triphone evaluation
-
on-phoneme AKA abx2-style evaluation ⇒ run the within-context (phoneme item file) evaluation AND the any-context (phoneme item file) evaluation.
Or maybe just:
context: all
-
on-triphone (abx1-style) evaluation (with the corresponding configuration)
-
within-context evaluation (...)
-
any-context evaluation (...)
ewan
nhamilakis
Following the updates, we did with Mark, i think the old abxLS module can be completely removed and replaced with the abxLSPhon one since it can do the same evaluation as the old one.
The type of evaluation is decided on the
contextparameter.Am i correct in my assumption ? i Would like your feedback @ewan @hallapmark Also, as Ewan said, maybe the benchmark name is not correct any more, should i change it to something else ?