nyumaya / nyumaya_audio_recognition

Classify audio with neural nets on embedded systems like the Raspberry Pi
https://nyumaya.com
Apache License 2.0
82 stars 14 forks source link

False recognition when no signal is present #6

Closed yodakohl closed 5 years ago

yodakohl commented 6 years ago

Some models may be prone to false detections when no signal is present. This can also trigger an initial false detection on startup. Will be fixed in the next model release.

yodakohl commented 6 years ago

False detections for Marvin if a very loud noisy signal is present. (https://soundcloud.com/user-837898227/marvin-false-detect)

torntrousers commented 6 years ago

I've been getting a lot of false detections with the marvin_sheila_small model, even with the sensitivity set up high, especially with something like TV on in the room.

yodakohl commented 6 years ago

I'm working on expanding the dataset with more background data. Did you mean low sensitivity? High sensitivity means it will detect more false positives.

torntrousers commented 6 years ago

I meant high, 0.9, have I this the wrong way around and 0.1 is what I want to be most accurate?

yodakohl commented 6 years ago

Just checked that I haven't published the marvin-sheila-small results yet. Here they are. If you set the Sensitivity to 0.1 will lead to less false positives, but will also be harder to trigger.

Accuracy clean 
Sens: 0.1 Accuracy: 0.9085173501577287
Sens: 0.2 Accuracy: 0.9400630914826499
Sens: 0.3 Accuracy: 0.943217665615142
Sens: 0.4 Accuracy: 0.9779179810725552
Sens: 0.5 Accuracy: 0.9779179810725552
Sens: 0.6 Accuracy: 0.9810725552050473
Sens: 0.7 Accuracy: 0.9873817034700315
Sens: 0.8 Accuracy: 0.9873817034700315
Sens: 0.9 Accuracy: 0.9936908517350158
Sens: 0.95 Accuracy: 1.0
Sens: 0.99 Accuracy: 1.0

Accuracy noisy 
Sens: 0.1 Accuracy: 0.8138801261829653
Sens: 0.2 Accuracy: 0.8517350157728707
Sens: 0.3 Accuracy: 0.8580441640378549
Sens: 0.4 Accuracy: 0.943217665615142
Sens: 0.5 Accuracy: 0.9305993690851735
Sens: 0.6 Accuracy: 0.9526813880126183
Sens: 0.7 Accuracy: 0.9337539432176656
Sens: 0.8 Accuracy: 0.9337539432176656
Sens: 0.9 Accuracy: 0.9558359621451105
Sens: 0.95 Accuracy: 0.9810725552050473
Sens: 0.99 Accuracy: 0.9810725552050473

False predictions per hour
Sens: 0.1 False Per Hour: 0.7794772727973933
Sens: 0.2 False Per Hour: 1.1692159091960899
Sens: 0.3 False Per Hour: 1.3640852273954382
Sens: 0.4 False Per Hour: 7.2101647733758885
Sens: 0.5 False Per Hour: 7.989642046173281
Sens: 0.6 False Per Hour: 8.574250000771327
Sens: 0.7 False Per Hour: 9.743465909967416
Sens: 0.8 False Per Hour: 10.133204546366112
Sens: 0.9 False Per Hour: 22.799710229323754
Sens: 0.95 False Per Hour: 37.60977841247423
Sens: 0.99 False Per Hour: 51.835238641026656