deal with weights in Grammar#generate() - Githubissues

dialogos-project / dialogos

The DialogOS dialog system.

https://www.dialogos.app

GNU General Public License v3.0

21 stars 8 forks source link

deal with weights in Grammar#generate() #65

Open timobaumann opened 6 years ago

timobaumann commented 6 years ago

There appear to be issues with weights in the grammar:

the grammar parser does not like weights very much (in fact, the lexer already doesn't like them).

According to https://www.w3.org/TR/2000/NOTE-jsgf-20000605/#16132 , https://www.w3.org/TR/speech-grammar/#S2.4.1 . Although the definitions somewhat overlap, they disagree slightly:

SRGF: Legal formats are "n", "n.", ".n" and "n.n" where "n" is a sequence of one or many digits. No weight → 1.0, does not mention whitespace.
JSGF: 56, 0.056, 3.14e3, 8f , all alternatives must have weights, allows whitespace within // markers.

This is unfortunate as AlternativeRules do support weights, output of weights is implemented and Sphinx would support weights.

The relevant line in com.clt.script/script.lex is 197 but I'm afraid to change anything in there.

output of weights works well for SRGF but weights are lost in JSGF output. (At least the internal representation seems to work in some cases.) This is unfortunate as Sphinx is fed the JSGF output.
Grammar#generate() does not take weights into account.

timobaumann commented 6 years ago

I've fixed 2 in 4544fc0, leaving open for anyone interested to fix this in the future.