GiviMAD / rustpotter

An open source wakeword spotter forged in rust
Apache License 2.0
88 stars 3 forks source link

Poor accuracy? #2

Closed decahedron1 closed 1 year ago

decahedron1 commented 1 year ago

Hi, I stumbled across this library while looking for an open-source wakeword detection library. I tested it with ~8 samples and got 'ok' results. Fiddling with the thresholds and noise modes got me either a lot of false positives (~20%) or a lot of false negatives (~50%). I tried to give it 20 samples but only got more false positives/negatives. I tried to build a model with ~200 samples via the CLI but ran into panicked at 'index out of bounds: the len is 42 but the index is 42' in src/wakeword.rs:168:31.

Testing the model against training samples didn't show a very good result (<0.65) for any of the samples. Is there anything else I could try to improve the performance of my models?

GiviMAD commented 1 year ago

Sorry for the late response, I was a bit off during summer. I need to write a documentation for this library to explain how it works. The detection will not get more accurate by adding more samples, instead try use more clear samples and correctly cut (no silence at the beginning and end). As the library generates a spectrum of each sample than will be compared with the live audio spectrum of the same length. So no reason for adding too many samples. Yes I usually get about that, 0.60 to 0.70 with models generated from my voice, less for models with STT voices but I don't get too much false positives ones I get to the correct values, I think that can be caused by noise in the samples or to much silence. I'll investigate this one (I'm currently refactoring the project): panicked at 'index out of bounds: the len is 42 but the index is 42' in src/wakeword.rs:168:31. I wrote this library a little quick because I want to use it on my home, it is next to an ECHO device running day to day connected to openHAB and I didn't get more false positives in one than the other (I'm using a model generated with GoogleSTT from the demo page), so I'll close the issue for now. In case you didn't see it: You can use this page to test the library with models generated using GoogleSTT voices: https://givimad.github.io/rustpotter-worklet-demo/ And this one to generate your own ones: https://givimad.github.io/rustpotter-create-model-demo/