search

nyumaya / nyumaya_audio_recognition

Classify audio with neural nets on embedded systems like the Raspberry Pi
https://nyumaya.com
Apache License 2.0
82 stars 14 forks source link

Alexa AVS testing #19

Closed jonsmirl closed 5 years ago

jonsmirl commented 5 years ago

Have you tried passing the Alexa AVS hot word tests? It would be great if an open source implementation could pass those test. As far as I know only closed source solutions have passed.

https://developer.amazon.com/docs/alexa-voice-service/acoustic-testing-guide.html

yodakohl commented 5 years ago

The Amazon testing guide seems to be tailored to physical hardware devices running the Alexa Voice Service using their Keyword Spotting Engine.

Developers use the Alexa Voice Service (AVS) to integrate Alexa directly into their products. Your team chooses hardware and designs the experience, AVS provides the automatic speech recognition, natural language understanding, and text-to-speech engines.

This Repository provides a software implementation of Keyword Spotting so this cannot be certified by Amazon.

jonsmirl commented 5 years ago

I don't work for Amazon, but maybe if you start talking to them they might help out on your project. They have quality tensorflow models for the various hot words in many languages but I can't figure out how to gain access to them. You could also ask to have your engine including the Alexa SDK and see what they say. https://github.com/alexa/avs-device-sdk Get it working as a plugin like snowbird/sensory and then file an issue asking for a merge. See what happens.

You will need to pick reference hardware for doing the testing. Once the engine is able to pass on the reference platform it is usually not hard to get it running on other hardware. That is what snowbird and sensory have done. One of the Amazon tests plays audio for 24hrs containing keyword utterances intermixed with noise. To pass that test you can only misidentify three times. That is a pretty high hurdle for accuracy.

yodakohl commented 5 years ago

I'm afraid I currently can't divert any resources to this. Anyone who needs a certification of any kind will have to do it on their own.