search

Picovoice / porcupine

On-device wake word detection powered by deep learning
https://picovoice.ai/
Apache License 2.0
3.77k stars 501 forks source link

Porcupine Issue: adding audio player will stop wakeWordCallback from getting called again #478

Closed geekyme closed 3 years ago

geekyme commented 3 years ago

Expected behaviour

wakeWordCallback gets called every single time the trigger word is mentioned

Actual behaviour

wakeWordCallback only gets called once. The moment an audio player is created, it stops getting called.

Steps to reproduce the behaviour

1) Clone https://github.com/geekyme/picovoice_test 2) flutter run 3) Say "jarvis", an audio will be played 4) Repeat Step 3 and it will no longer work.

Revert the commit for adding the audio - https://github.com/geekyme/picovoice_test/commit/cf1f5d5137e1620b6fc7c84d514bed461ae46216

Now perform step 3 and 4, the callback will be triggered every time you say "jarvis". Somehow, adding that audio player stops porcupine from working correctly.

laves commented 3 years ago

@geekyme does this happen on both iOS and Android?

geekyme commented 3 years ago

I can't test on android since I don't have an android device. So far just iOS.

Problem does not happen on the simulator.

laves commented 3 years ago

Okay. What iOS version and device model are you seeing it on?

geekyme commented 3 years ago

IOS 14.7.1, iPhone 12 Pro Max

geekyme commented 3 years ago

Any luck on this?

laves commented 3 years ago

@geekyme we've scheduled someone to take a look at this in the coming weeks. Thank you for your patience.

geekyme commented 3 years ago

The main piece of code that appears to be interfering with picovoice is: 

AssetsAudioPlayer.playAndForget(Audio("/assets/audio/wake_up_speech.mp3"))

Running this code anywhere will stop picovoice from working correctly.

geekyme commented 3 years ago

Tried a different audio library (just_audio), same results.

See this commit on a separate branch just_audio - https://github.com/geekyme/picovoice_test/commit/b0a455d25f0fed6160540da7a3909e8889f82271. If you run my code, saying "jarvis next" repeatedly works.

But after clicking the increment button, the audio will be played and "jarvis next" will not work anymore.

It seems just by playing any audio from code will mess up picovoice.

laves commented 3 years ago

@geekyme, we've spent some time looking at this and it seems this is not necessarily a problem with Picovoice or our flutter audio recorder plugin - more of a combination of bad assumptions made by Apple and these audio player plugins.

The problem is related to iOS's AVAudioSession. Each iOS app has a single audio session assigned to it that can be configured to do certain things. For our purposes, we enable recording and playback so that we can take in microphone audio, but still allow audio to be played. When you play a sound using these audio player plugins, they switch the audio session to playback only, which turns off audio recording. As a result, our Picovoice plugin stops receiving audio and cannot perform speech recognition. Apple assumes each app will have complete control over its own audio session, so this changing of settings is not considered an interruption event and cannot be handled like other audio interruptions. If you use any other flutter audio recording plugin, start recording, play a sound using one of these plugins, you'll see they stop recording audio as well.

Unfortunately, there is no great way around this from our perspective. We've added extra error handling to detect when we've lost audio, but we cannot do much more at this time. I'd recommend you'd report this problem to the developers of the plugins you are trying to use and recommend that they not shut off recording capabilities when they play audio (or at least make it configurable).