[Feature Request][snips-nlu] Math symbols are not taken into account by the NLU when using google ASR output in the NLU with an assistant trained for snips-asr

[Psychokiller1888]

My intent has 3 slots:

• left
• right
• function

The intent has 189 training exemples

Snips does not capture the symbols for the function:

[12:17:37] Watching on localhost:1883 (MQTT) [12:23:22] [Hotword] detected on site office, for model alice [12:23:22] [Dialogue] session with id '32d9afda-dd18-4ad2-a579-4ef18719b277' was started on site office [12:23:27] [Asr] captured text "what is 10 / 3" in 3.9s [12:23:28] [Nlu] detected intent Psychokiller1888:maths with probability 0.255 for input "what is 10 / 3" Slots -> left -> 10 right -> 3 [12:23:28] [Dialogue] New intent detected Psychokiller1888:maths with probability 0.255 Slots -> left -> 10 right -> 3 [12:23:28] [Dialogue] was ask to end session with id 32d9afda-dd18-4ad2-a579-4ef18719b277 by saying 'Sorry? I didn't understand the calculation' [12:23:28] [Tts] was asked to say "Sorry? I didn't understand the calculation" [12:23:31] [Dialogue] session with id '32d9afda-dd18-4ad2-a579-4ef18719b277' was ended on site office. The session ended as expected [12:23:34] [Hotword] detected on site office, for model alice [12:23:34] [Dialogue] session with id 'cc2d915e-4efc-41a3-8bf3-e85290992502' was started on site office [12:23:40] [Asr] captured text "what is 20 + 75" in 5.3s [12:23:42] [Nlu] detected intent Psychokiller1888:maths with probability 0.255 for input "what is 20 + 75" Slots -> left -> 20 right -> 75 [12:23:42] [Dialogue] New intent detected Psychokiller1888:maths with probability 0.255 Slots -> left -> 20 right -> 75 [12:23:42] [Dialogue] was ask to end session with id cc2d915e-4efc-41a3-8bf3-e85290992502 by saying 'Sorry? I didn't understand the calculation' [12:23:42] [Tts] was asked to say "Sorry? I didn't understand the calculation" [12:23:45] [Dialogue] session with id 'cc2d915e-4efc-41a3-8bf3-e85290992502' was ended on site office. The session ended as expected

It is using Google ASR

[Psychokiller1888]

For reference:

[cpoisson]

Hi @Psychokiller1888,

Is it still an ongoing issue?

[Psychokiller1888]

Yep :)

[cpoisson]

urm ok :) Thanks for your feedback, for some reason the image does not appear correctly.

[cpoisson]

It's interesting.

On the console, it seems to work, but when you look at it closely, the test input string is reformated beforehand.

"+" become "plus" "-" become "minus" "/" become "" "*" become "" "x" stay "x"

[Psychokiller1888]

Yeah, the point of the screenshot is to show the discrepency between what's typed and what's translated. On the console I do type "2 + 2" and it's translated to it's word counterpart. Same on my live install, Google hears "2 + 2" but the NLU is unable to take that "+" and reports the wrong intent

Image: https://puu.sh/CX2cU.png

Although the slot is configured with synonyms for that:

https://puu.sh/CXiJY.png

[adrienball]

@Psychokiller1888 Can you confirm that you have selected Google ASR on the web console? The reason why I'm asking this is that the verbalization that you see on the console ("+" => "plus") should only happen when Snips ASR was selected.

[Psychokiller1888]

I am using a mix of snips ASR and google asr whenever a word is unknown. So I need Snips ASR to generate the data, thus my assistant is configured as using Snips ASR, but my live assistant can use both. It can use Snips ASR if it looses connection to internet or ask it to stay offline, it can use Google ASR if it finds unknown words if I did not forbid online connectivity and it has connectivity. The discrepency comes from Snips vs Google, one returns symbols and the other turns them to words. One line should be decided, because this prevents many of my coded apps to work properly

[Psychokiller1888]

Not to forget that the slot has synonyms to should turn "plus" to "+" according to my screenshot ;). So even if there's a different return between Google and Snips, "plus" or "+" should always be "+" and not verbalized

[adrienball]

The issue is that the NLU pipeline has a specific configuration for each ASR engine (Snips, Google). These configurations are adapted to the ASR output. Hence, for Snips ASR, the NLU pipeline performs a verbalization preprocessing step which is not required for Google ASR as the latter generally outputs non-verbalized forms. Your use case, which is legit, is not covered properly at the moment as you are using a single NLU engine after both Google ASR or Snips ASR.

As a workaround, you can try to switch to Google ASR in the console and retrain your assistant. This will produce a new nlu_engine (which should work better in your pipeline) that can you can use instead of the one you currently have.

[Psychokiller1888]

Yes, but i won't be able to use it offline then

[Psychokiller1888]

I don't really get it. The ASR can return whatever it wants, the slot configuration should turn the value to the main value, not to the synonyms no?

[cpoisson]

I leave the label question for @adrienball 🙈

So basically, we are more talking about a missing feature here.

As a snips assistant developer I want to be able to use snips-asr and snips-asr-google at the same time So that I can switch on google ASR when my utterances cannot be catched by snips-asr

[Psychokiller1888]

Not really, as this is specific to the symbols. Switching between Snips and Google is a hack of mine, not a feature for Snips

I don't understand what this has to do with the ASR to be honest. Maybe I don't see the grand picture?

[adrienball]

@Psychokiller1888 I found out the root cause of your issue. The + symbol, as well as most of the symbols, is tokenized out internally in the NLU, meaning that "hello + world" is tokenized into ["hello", "world"]. We do not plan to update the tokenization at the moment, but as a workaround you can post-process the output of Google ASR and replace math symbols by their literal equivalent.

[Psychokiller1888]

@adrienball That means catching before the nlu and manually sending the post processed string to the nlu? Well, that's a giant crap, when you have nearly 100 intents and you have to stop them all to parse the string, replace occurences and manually sending to nlu for 1 intent in the 100. Because I can't rely on the NLU, as it returns the wrong intent :)