[Story] Speech Recognition

[ar13pit]

Stage 1

• [ ] Add downloading of released models in tue-env-targets
• [ ] Add closed grammar functionality to yapykaldi (Refer to zamia speech JSGF parser and adaptation). For this JSGF parser must be adapted to our grammar format and we will need to wrap a couple of C++ functions into python that would attach the closed grammar to the acoustic model.
• [x] Add class based API for ASR in yapykaldi. (#1014 ) (tue-robotics/yapykaldi#1)
• [x] Decide where parsing of the output of yapykaldi into semantics must be done. In yapykaldi or in the ros wrapper.
• [ ] Create a ros HMI server wrapper for ASR in yapykaldi (tue-robotics/yapykaldi_ros#1)

Stage 2 (Enhancements to Yapykaldi)

• [ ] Add Voice Activity detection to know when to stop (or start)
• [ ] Add garbage phonemes to closed grammar to make recognition robust
• [ ] Add wake word recognition to start recognition only on certain words (i.e. first word of the grammar targets)

Low priority but important

• [ ] Create an ubuntu PPA and release kaldi shared libs, include files, yapykaldi. This will simplify the installation process drastically.
• [ ] Move speech_recognition to docker when yapykaldi PPA is ready.

[LoyVanBeek]

I'll take up the

Create a ros HMI client wrapper for yapykaldi (yapykaldi_ros)

[ar13pit]

I'll take up the

Create a ros HMI client wrapper for yapykaldi (yapykaldi_ros)

Great. There is one feature that is needed there though, caching of the grammar string if the same grammar was sent in the previous HMI query as compiling the speech model all the time will be very expensive.

[LoyVanBeek]

Allright. I've spent most my time setting up the dependencies and I need to be on 18.04 I think though, so I'll continue next week.

[LarsJanssenTUe]

Allright. I've spent most my time setting up the dependencies and I need to be on 18.04 I think though, so I'll continue next week.

Needing to be on 18.04 would be undesired if not explicitly necessary. Why is this needed @ar13pit ?

[ar13pit]

No its not needed. I wrote that in the README as I explicitly tested it on 18.04 but it will work on 16.04 as well as along as CUDA is not being used and gcc7 is used.

[LoyVanBeek]

Hmm, kaldi complained about needing CMake 3.12 or somethingh while by 16.04 box has 3.5 and could not get a higher version installed yet.

[ar13pit]

Did you try running the command tue-get install python-yapykaldi ? It should install all the dependencies, build kaldi and yapykaldi.

However, if you don't want to do that go with pip2 install --user cmake. You won't have to spend time building it from source.

[LoyVanBeek]

@ar13pit is https://github.com/gooofy/zamia-speech/#model-adaptation what you meant with "Zamia speech JSGF"?

There is also https://pypi.org/project/pyjsgf/

[ar13pit]

Yup and that section is a huge pile of shit.

[LoyVanBeek]

@ar13pit https://github.com/tue-robotics/yapykaldi_ros/pull/1 and https://github.com/tue-robotics/yapykaldi/pull/1 can be reviewed

[LoyVanBeek]

Decide where parsing of the output of yapykaldi into semantics must be done. In yapykaldi or in the ros wrapper. I currently have this in the ROS wrapper and I think that makes the most sense too.

Same for the converting to JSGF format: I'd say that happens in the ROS wrapper and the JSGF grammar is passed to Asr.recognize or Asr.start.

[ar13pit]

I actually getting rid of JSGF completely as that is also an intermediate format. Instead I'll keep NLTK grammar object as that is pretty close to our grammar parser. So this object is passed onto Asr.recognize and a new ASR model is created.