stateful aug30 model + VAD gating

[benejoseph]

neural net model

• works across most speakers on office test set actually recorded from sense
• numerous false positives on continuous speech which is negated by the "VAD gating"

VAD gating

• looks for sufficiently long period of non-speech (0.75 seconds) before allowing "okay sense" to trigger callback
• gives two seconds to actually say okay sense
• thus if something that triggers "okay sense" occurs in the middle of a sentence, it is ignored, but it still uploads the audio features 😄
• eliminates over 90% of the false positives I see from my continuous speech samples

[benejoseph]

@plasticchris

[plasticchris]

What if a user is having trouble, so they are repeating "ok sense" over and over? won't this ensure it never works for them?

[benejoseph]

why don't you try it?

[plasticchris]

Ok, tried it and verified it's possible to never trigger while repeating "ok sense" over and over

[benejoseph]

and what happens if you wait a second?

[benejoseph]

Also, repeating okay sense over and over doesn't work with the RNN anyway. We've always had to wait a short period if time.

[plasticchris]

Yes but this is what people do when it doesn't work at first

On Thu, Dec 8, 2016, 1:49 PM Ben notifications@github.com wrote:

also, repeating okay sense over and over doesn't work with the RNN anyway

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/hello/kitsune/pull/712#issuecomment-265865758, or mute the thread https://github.com/notifications/unsubscribe-auth/AHR0tKFN4g174_bWklrzsqnIwOWyE_cVks5rGHtQgaJpZM4LHTRA .

[benejoseph]

I think we should try this on our master units and get feedback. Perhaps your perception isn't shared by everyone else.

[plasticchris]

It's not a perception. It's what I've observed happens when the first attempt doesn't trigger. Try watching someone new to the device to see it.

On Thu, Dec 8, 2016, 2:34 PM Ben notifications@github.com wrote:

I think we should try this on our master units and get feedback. Perhaps your perception isn't shared by everyone else.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/hello/kitsune/pull/712#issuecomment-265876343, or mute the thread https://github.com/notifications/unsubscribe-auth/AHR0tHmdio5vHva25K-PfEXnwNdiiVwUks5rGIYEgaJpZM4LHTRA .

[pims]

From my anecdotal testing, it feels more responsive. We can adjust the no-speech delay if it's an issue.

Let's get some real feedback by pushing to master.

[plasticchris]

Ok, but people in master are already conditioned and/or in the training set. This is important, it will drastically impact the new user experience as people get increasingly frustrated and then give up.

[plasticchris]

It it possible to decouple the VAD gating from the new net?

[benejoseph]

programatically or at compile time?

[plasticchris]

Compile time is fine, but it looks like the threshold for ok sense got lowered significantly. It's not clear if the VAD gating is required to make that work.

[benejoseph]

Barring setting extreme thresholds (0.05 or 0.95), the main influence over false positives is the choice of neural net. This net has more false positives per hour than the previous net by an order of magnitude, but it works across many more speakers in the real world. The VAD gating is an attempt reduce false positives from continuous speech. So I'd like the two to go hand-in-hand. But to disable the VAD gating all you have to do is comment out the "if" statement surrounding the keyword on_end callback in keyword_net.c

[benejoseph]

don't merge, btw. the VAD gate seems to get stuck.

[plasticchris]

mmk

[benejoseph]

closing because we're addressing actual root causes! Yipee!